Sensor Fusion for Semantic
Segmentation of Urban Scenes
Richard Zhang1, Stefan A. Candra1, Kai Vetter12, Avideh Zakhor1
1University of California, Berkeley
2Lawrence Berkeley National Laboratory
Presented at ICRA, May 2015

1Goal• Extract semantic information
by fusing multiple modalities• Image• LiDAR scan

• Challenges• Incorporate information frommultiple scales• Fuse information from multiple
modalities with nonoverlapping sensor coverage2Top Level Pipeline3

Previous Work
Sengupta et al.
ICRA 2013
He and Upcroft.
IROS 2013

Segmentation

Image Features

Point cloud features

Fusionnone

DescriptivenoneSingle segmentation
image only

Descriptive

none

majority votingof imageclassifications in 3D

4

Previous Work
Sengupta et al.
ICRA 2013
He and Upcroft.
IROS 2013
Cadena and Košecká.
ICRA 2014.

Segmentation

Image Features

Point cloud features

Fusionnone

Descriptive

none

Descriptive

none

majority votingof imageclassifications in 3D

SimpleSimpleEarly fusion

Single segmentation
image only
Single segmentation
image only5

Previous Work
Sengupta et al.
ICRA 2013
He and Upcroft.
IROS 2013
Cadena and Košecká.
ICRA 2014.
Ours

Segmentation

Image Features

Point cloud features

Fusionnone

Descriptive

none

Descriptivenonemajority votingof imageclassifications in 3D

SimpleSimpleEarly fusion

Descriptive

Descriptive

Late fusion

Single segmentation
image only
Single segmentation
image only
multiple segmentations
both domains

6

Segmentation• Treating each pixel/point
separately is computationallyintensive• Under-segmentation & Oversegmentation errors
 Multiple segmentations
performed to getsuperpixels/supervoxels

7

Segmentation• Treating each pixel/point
separately is computationallyintensive• Under-segmentation & Oversegmentation errors
 Multiple segmentations
performed to getsuperpixels/supervoxels

8Feature Extraction

Image superpixel features

Point cloud supervoxel features9

Early Fusion
Imagefeatures
Image
information
onlyImage-onlysegmentation

Point cloudfeatures
10

Early Fusion
Imagefeatures
Image
information
onlyImage-onlysegmentation

Point cloud
information
only

Point cloud-onlysegmentation
Point cloudfeatures
11

Early Fusion
Imagefeatures
Image
information
only

Image-only
segmentation

Overlapping
sensorcoverage

Fused
segmentation

Point cloud
information
only

Point cloud-onlysegmentation
Point cloudfeatures
12

Early Fusion
Image
featuresImage
information
onlyOverlapping
sensorcoverage

Image-only
segmentation

Large feature space
Less training data

Point cloud
information
only

Fused
segmentation

Point cloud-only
segmentation
Point cloudfeatures13

Late Fusion
Image-only
segmentation
Image
featuresImage-only
segmentation
Overlapping
sensorcoverage

Fused
segmentation

Point cloud-only
segmentation

Point cloudfeatures
Point cloud only
segmentation14

Late Fusion
Image-only
segmentation
Image
featuresImage-only
segmentation
Overlapping
sensorcoverage

MidlevelfeatureFused
segmentation

Point cloud-only
segmentation

Point cloudfeatures
Point cloud only
segmentation

15

FusionGround Truth

Point cloud
unimodal segmentation
PC only

Image unimodal segmentation
16

FusionGround Truth

Point cloud
unimodal segmentation
PC only(projected onto image)

Image unimodal segmentation
17

FusionGround Truth

Point cloud
unimodal segmentation
PC onlyLate-fused multi-modal segmentation(projected onto image)

Image unimodal segmentation
18

FusionGround Truth

Point cloud
unimodal segmentation
PC onlyLate-fused multi-modal segmentation(projected onto image)

Image unimodal segmentation
19

FusionGround Truth

Point cloud
unimodal segmentation
PC onlyLate-fused multi-modal segmentation(projected onto image)

Image unimodal segmentation
20

FusionGround Truth

Point cloud
unimodal segmentation
PC onlyLate-fused multi-modal segmentation

Image unimodal segmentation

Fused segmentation & Post-processing
Conditional Random Field (CRF)(projected onto image)21

Fusion

PC only

Image only
Performance on overlapping region

Fused22

Fusion

PC only

Image only
Performance on overlapping region

Fused23

Fusion

PC only

Image only
Performance on overlapping region

Fused
24

Fusion

PC only

Image only
Performance on overlapping region

Fused
25

Results• Quantitative Results• Train and test on KITTI1 dataset, augmented withadditional annotations• 252 images across 8 sequences• 140 images for training, 112 for testing

1Geiger, et al. KITTI. CVPR 1226Results• Quantitative Results• Train and test on KITTI1 dataset, augmented withadditional annotations• 252 images across 8 sequences• 140 images for training, 112 for testing

1Geiger, et al. KITTI. CVPR 1227

Results• Quantitative Results• Train and test on KITTI1 dataset, augmented withadditional annotations• 252 images across 8 sequences• 140 images for training, 112 for testing

1Geiger, et al. KITTI. CVPR 1228

Results• Quantitative Results• Train and test on KITTI1 dataset, augmented withadditional annotations• 252 images across 8 sequences• 140 images for training, 112 for testing

1Geiger, et al. KITTI. CVPR 1229

Thank you!

30