Learning to See in Novel Viewpoints and Domains
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems

June 16, 2018

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupConditional Affordance Learning
Video Input

Control
Comands

Affordances
NeuralNetwork
Directional Input

𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑎𝑛𝑔𝑙𝑒 = 0.01 𝑟𝑎𝑑𝐶𝑒𝑛𝑡𝑒𝑟𝑙𝑖𝑛𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒= 0.15 𝑚
𝑅𝑒𝑑 𝑙𝑖𝑔ℎ𝑡 = 𝐹𝑎𝑙𝑠𝑒…

Sauer, Savinov and Geiger: Conditional Affordance Learning for Driving in Urban Environments. CoRL, 2018.

Controller

Brake = 0.02

NoVA:Learning to See in Novel Viewpoints and Domains[Coors & Geiger, 2019]

Are all autonomous vehicles created equal?

Are all autonomous vehicles created equal?

I Senor setups vary signiﬁcantly depending on the type of autonomous vehicle
5Generalization is Challenging

Source Domain

FCN8s

DRN26

Target Domain

FCN8s

DRN26

I Semantic Segmentation networks trained on source domain
do not generalize well to novel viewpoints (target domain)6

Cycle-Consistent Adversarial Domain Adaptation

I CyCADA addresses domain adaptation but not viewpoint adaptation
Hoffman, Tzeng, Park, Zhu, Isola, Saenko, Efros and Trevor Darrell: CyCADA: Cycle-Consistent Adversarial Domain Adaptation. ICML, 2018.

7

Cycle-Consistent Adversarial Domain Adaptation

Source Domain

FCN8s

DRN26

Target Domain

FCN8s

DRN26

I CyCADA does not handle viewpoint changes
I Leads to signiﬁcant performance drop also for source domain
Hoffman, Tzeng, Park, Zhu, Isola, Saenko, Efros and Trevor Darrell: CyCADA: Cycle-Consistent Adversarial Domain Adaptation. ICML, 2018.

8

Novel Viewpoint and Domain Adapation

Source domain

Target domain (no labels)

I NoVA: Geometry-aware image and semantic label translationI Key Idea: Decompose problem (reconstruct, warp, inpaint, stylize)
9Model Overview

Model Overview
Source Domain
Depth
Network

Depth Map

Target Domain
Reﬁne
NetworkRender

Source Image

Warped Image

Inpainted Image
Task
Network

Source Segm.

RenderWarped Segm.

Target Image
Task
Network

Predicted Segm.

11

Depth Estimation
Source Domain
DepthNetwork

Depth Map

Target Domain
Reﬁne
Network

Render

Source Image

Warped Image

Inpainted Image

Target Image

Task
Network

Source Segm.

RenderWarped Segm.

Task
Network

Predicted Segm.

Predict depth from a single image:I Can be trained either supervised using depth ground truth
I Or self-supervised using a second frame for supervision [Godard, CVPR 2017]12

Rendering
Source Domain
Depth
NetworkDepth Map

Target Domain
ReﬁneNetwork

Render

Source Image

Warped Image

Inpainted Image
Task
Network

Source Segm.

RenderWarped Segm.

Target Image
Task
Network

Predicted Segm.

Warp depth map and segmentation into target view:
I Forward splatting KT T K−1
S
I Soft z-buffering (weight based on inverse depth)
13Inpainting and Reﬁnement
Source Domain
DepthNetwork

Depth MapTarget Domain
Reﬁne
Network

Render

Source Image

Warped Image

Inpainted Image
Task
Network

Source Segm.

RenderWarped Segm.

Target Image
Task
Network

Predicted Segm.

Reﬁne warped image:I Inpaint occluded areas and stylize in style of target domain
I Residual network
14

Task Network
Source Domain
Depth
Network

Depth Map

Target Domain
Reﬁne
Network

Render

Source Image

Warped Image

Inpainted Image
Task
Network

Source Segm.

RenderWarped Segm.

Target Image
Task
Network

Predicted Segm.

Semantic segmentation task network:
I FCN8s and DRN26, cross-entropy loss applied to valid pixels
I Adversarial losses encourage similarity of image and semantic spaces
15

Loss Function
Source Domain
DepthNetworkDepth Map

Target Domain
Reﬁne
NetworkRender

Source Image

Warped Image

Inpainted Image
Task
Network

Source Segm.

RenderWarped Segm.

Target Image
Task
Network

Predicted Segm.

Loss function: LN oV A = Ldepth + Lpho + Ltask + LGAN

16

Experimental Evaluation

Viewpoint Adaptation
CARLA Car → CARLA TruckViewpoint Adaptation
MethodmIoU

Source Only
SceneAdapt [Mauro et al., AVSS 2018]
CyCADA [Hoffman et al., ICML 2018]
CyCADA [Hoffman et al., ICML 2018] + trgt-labels
SPLAT [Tzeng et al., 2018]
SPLAT [Tzeng et al., 2018] + trgt-labels

26.54
26.63
10.57
16.31
13.63
18.81

NoVAmono−self
NoVAmono−sup
NoVAstereo−sup
NoVAGT

42.54
45.27
49.67
51.89

Target Oracle

52.72
19

Viewpoint Adaptation

Source Image

Warped Image

Reﬁned Image

Warped Labels

20Viewpoint Adaptation

Source Image

Warped Image

Reﬁned Image

Warped Labels

21

Viewpoint Adaptation

NoVAmono−self

NoVAmono−sup

NoVAstereo−sup

NoVAGTNoVA Performance for Different Depth Estimators:
I NoVA works with self-supervised or supervised depth predictions
22

Viewpoint Adaptation

Source Image

NoVAmono−self

CyCADA

SPLAT

23

Viewpoint Adaptation

SourceTranslated

Reconstructed

CyCADA on Sim2Sim:I CyCADA translates incorrectly but reconstructs correctly
I Learns to encode source image semantics in the noise of translated image

24

Viewpoint and Domain Adaptation
CARLA Truck → Cityscapes Car

Viewpoint and Domain Adaptation
MethodmIoU

Source Only
SceneAdapt [Mauro et al., AVSS 2018]
CyCADA [Hoffman et al., ICML 2018]
SPLAT [Tzeng et al., 2018]

18.84
11.54
19.26
21.01

NoVAmono−self
NoVAmono−supNoVAstereo−sup
NoVAGT

30.23
34.36
32.96
35.91

Target Oracle

51.30

26

Viewpoint and Domain Adaptation

Source Image

Warped Image

Reﬁned Image

Warped Labels

27

Viewpoint and Domain Adaptation

Source Image

NoVAmono−self

CyCADA

SPLAT

28

Ablation Study
Sim2SimmIoU

Source Only
+ Forward Warping+ Residual Reﬁnement

26.54
47.81
51.89

Sim2RealmIoU

Source Only
+ Forward Warping+ Residual Reﬁnement

18.84
26.95
35.91

29Semi-Supervised Viewpoint and Domain Adaptation

mIoU

50454035100 3001000

NT2975Semi-Supervised Adaptation on Sim2Real:
I Combination of NS = 30, 000 translated images with NT labeled target images
30Video31

Summary
This work:
I Semantic segmentation networks do not generalize to novel viewpoints
I Existing domain adaptation techniques cannot address the problem
I NoVA is a new model for viewpoint and domain adaptation
I Geometry-aware image and label translation
I Decompose problem: depth estimation, warping, inpainting, domain

Future work:
I Warpings and inpaintings sometimes still exhibit artifacts
I Current results are not temporally consistent32Thank you!
http://autonomousvision.github.io