Image Domain Transfer
Jan Kautz, VP of Learning and Perception ResearchImage Domain Transfer: enabling machines
to have human-like imagination abilities

FInput image

Domain transferred image
This image is generated byour method.

Example use cases

Low-res to high-res

Blurry to sharp

Image to painting

LDR to HDR

Synthetic to real

Thermal to color

Day to night

Summer to winterNoisy to clean

Two Approaches

F• Example-based
Input image§ Non-parametric model§ The transfer function F is defined by an example image.

• Learning-based

Domain transferred image

F ( |xexample )§ Parametric model§ The transfer function is learned via fitting a training dataset.

F(|Training Dataset)Example-based Image Domain Transfer

F ( |xexample )Example-based image domain transfer
Often referred to as Style Transfer

Style photo

Content photo

Stylized content photo

xoutput = F (xinput |xexample )Example-based image domain transfer• Artistic Style Transfer• Content: real photo;• Style: painting• Gatys et. al. , Johnson
et. al., Li et al., Huanget. al.

• Photo Style Transfer

Style (painting)ContentOutput

StyleContentOutput• Content: real photo;• Style: real photo• Luan et. al., Pitie et.
al., Reinhard et. al.

FastPhotoStyle• ”A Closed-form Solution to Photorealistic Image Stylization” by
Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, Jan Kautz,
ECCV 2018• Code: https://github.com/NVIDIA/FastPhotoStyle

Fast Photo Style
Content image

We model photo style transfer as a closeform function mapping given by
Stylization step
Style

ContentStyle image

Photo smoothing step

Stylization Step
Assumption: Covariance matrix of deep features encodes the style information.
VGG19 up to conv4_1THC HCT = EC ⇤C EC

HS HST = ES ⇤S ESTStylization StepDecoder

Architecture is similar
to Li et al., but uses
unpooling.

whiteningcoloringWhen semantic label maps are available
Content

Style

StyleContentOutput

Stylization step

Photo Smoothing

Photo smoothing step

Style

Content

Assumption: If we can compute a new image where the image pixel values resemble
those in the intermediate image but the similarities between neighboring pixels
resemble those in the content image, then we have a photorealistic stylization outputs.

Similarity between
neighboring pixels(Gaussian/matting affinity)
Close-form solution: R⇤ = (1↵)(I↵D12Intermediate
image pixel valuesWD12) 1Y

StyleContentOutput

StyleContentOutput

Comparison

Comparison

Quantitative results

Learning-based Image Domain Transfer

F(|Training Dataset)Generative Adversarial Networks (GANs)

zGeneratorp(G(z)) ! p(X)Discriminator

False

Discriminator

True

Supervised vs Unsupervised
Supervised

UnsupervisedUnimodal vs Multimodal
UnimodalF(Multimodal

F(p(Y |X) = (F (X)))=p(Y |X) = F (X, S))=

Categorization
SupervisedUnsupervised

Unimodalpix2pix, CRN, SRGAN

UNIT, Coupled GAN, DTN,
DiscoGAN, CycleGAN, DualGAN,
StarGANMultimodal

pix2pixHD, vid2vid,
BiCycleGAN

MUNIT

Categorization
Supervised

Unsupervised

Unimodalpix2pix, CRN, SRGAN

UNIT, Coupled GAN, DTN,
DiscoGAN, CycleGAN, DualGAN,
StarGANMultimodal

pix2pixHD, vid2vid,

MUNIT

BiCycleGANpix2pixHD: Supervised and multimodal
image domain transfer• ”High-Resolution Image Synthesis and Semantic Manipulation withConditional GANs” byTing-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan
Kautz, Bryan Catanzaro, CVPR 2018
• Code: https://github.com/NVIDIA/pix2pixHD

pix2pixHD

pix2pix
Generatorp(G(S), S) ! p(X, S)

Discriminator

False

DiscriminatorTruepix2pixHD
Generator

Low-dimensional feature map

Instance
PoolingFeatureEncoder

Discriminator

False

Once we finish training, we extract theinstance-pooled feature maps from all
the training images and fit a mixture ofGaussian model to features belong to
the same semantic class.
During inference, we sample from themixture of Gaussian distributions forachieving multimodal image translation.

Coarse-to-fine Generator

Multi-scale Discriminators

Robust Objective(GAN + discriminator feature matching loss)

D3

DiDi
real

...

Residualblocks

match

D2...

match

Residualblocks

D1

2

real

synthesized

realsynthesizedpix2pixHD multimodal resultspix2pixHD label changes

Comparison

Categorization
UnimodalMultimodal

Supervised

Unsupervised

pix2pix, CRN, SRGAN

UNIT, Coupled GAN, DTN,
DiscoGAN, CycleGAN, DualGAN,
StarGANvid2vid,

pix2pixHD,
BiCycleGAN

MUNIT

vid2vid: Video-to-Video Synthesis• ”Video-to-Video Synthesis” byTing-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, AndrewTao, Jan Kautz, Bryan Catanzaro, NIPS 2018
• Code: https://github.com/NVIDIA/vid2vidMotivation

Using pix2pixHD

Sequential Generator

Semtan

Pasia
cmm
ti

psInagrtedmeia t

Spatio-temporally Progressive TrainingeSpatially progressive
Residual blocks

esw
F lomaaF inp

Wparl Image

...

...

edW

Temporally progressive

Multi-scale Discriminators
Image DiscriminatorVideo Discriminator

Alternating training

D1T

D2
D3STSTSResultsResults

AI Rendered GameCategorization
UnimodalSupervised

Unsupervised

pix2pix, CRN, SRGAN, …

UNIT, Coupled GAN, DTN,DiscoGAN, CycleGAN, DualGAN,StarGAN

Multimodal

pix2pixHD, vid2vid,BiCycleGAN

MUNIT

UNIT: Unsupervised and unimodal image
domain transfer• ”Unsupervised Image-to-image Translation Networks” byMing-Yu Liu, Thomas Breuel, Jan Kautz, NIPS 2017
• Code: https://github.com/mingyuliutw/UNIT

Supervised vs Unsupervised
Supervised

Unsupervised

Generator 1p(G1 (X1 )|X1 ) ! p(X2 )Discriminator 1FalseDiscriminator 1

True

Discriminator 2

False

Discriminator 2

True

But p(G1 (X1 )|X1 ) 9 p(X2 |X1 )Generator 2p(G2 (X2 )|X2 ) ! p(X1 )But p(G2 (X2 )|X2 ) 9 p(X1 |X2 )UNIT assumption: Shared Latent Space

Domain 2 GAN
Discriminator
Coupling themapping
function via
weightsharing

Encoder1

Decoder2

Sharedlatent
space

Encoder2

Decoder
1

Discriminator 1

False

Discriminator 1

True

Discriminator 2

False

Discriminator 2

TrueEncoder1

Decoder
2Encoder2

Decoder1Day to Night TranslationInput

Translated

Input

Resolution
640x480

TranslatedSnowy to Summery Translation
Input

Translated

Input

Resolution
640x480

TranslatedSunny to Rainy Translation
Input

Translated

Input

Resolution
640x480

TranslatedCategorization
Supervised

Unsupervised

Unimodalpix2pix, CRN, SRGAN

UNIT, Coupled GAN, DTN,
DiscoGAN, CycleGAN, DualGAN,
StarGANMultimodal

pix2pixHD, BiCycleGAN

MUNIT

MUNIT: Unsupervised and multimodal
image domain transfer• ”Multimodal Unsupervised Image-to-image Translation” byXun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz, ECCV 2018
• Code: https://github.com/NVlabs/MUNITMUNIT assumption: Partially Shared Latent
SpaceContentEncoder1

Decoder2

StyleEncoder
1
StyleEncoder
2
ContentEncoder
2

Discriminator 1

False

Discriminator 1

True

Discriminator 2

False

Discriminator 2

True

Shared content
latent space
Decoder1

MUNIT network

MUNIT results

MUNIT Style Transfer
ContentEncoder
1

StyleEncoder2

Decoder2

Style Transfer Comparison

Conclusion• Example-based image domain transfer• Learning-based image domain transfer
Learning-based
Supervised

Unsupervised

UnimodalMultimodal

YOUR LIFE’S WORK STARTS HERE
JOIN NVIDIA
100 Best Companies to Work For— Fortune

Most Innovative Companies— Fast Company

World’s Most Admired Companies— Fortune

Employees’ Choice: Highest Rated CEOs
— Glassdoor

50 Smartest Companies— MIT Tech Review

INTERESTED? Email: aijobs@nvidia.com

77