Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
Huang et al., ICCV 2017

YANG Chengkai
Department of Electrical Engineering
and Information Systems, M1

Style Transfer

ContentStyle

Source: http://stevydya.blogspot.com/2015/04/the-university-of-tokyo.html,
WikiArtOutputConvolutional Neural Networks
Convolution

input

Source: https://www.analyticsvidhya.com/blog/2021/07/convolution-neural-network-better-understanding/featureConvolutional Neural Network
VGG Architecture

CNN extract image features

Source: https://www.analyticsvidhya.com/blog/2021/07/convolution-neural-network-better-understanding/CNN Captures StyleImage Style Transfer using Convolutional
Neural Networks, Gatys et al., CVPR 2016

Content representation: Activations of the filter i at a position j in a layer l
Style representation: Inner product between vectorised feature maps

Arbitrary Style Transfer

Image Style Transfer using Convolutional
Neural Networks, Gatys et al., CVPR 2016

Arbitrary: Works for any
pair of inputs

Slow: Require network
optimization to generate
an output

AdaIN Style Transfer

Huang et al.,ICCV 2017

VGG Encoder – Extract features
AdaIN – Perform style transfer on features
Decoder – Reconstruct image from features

Arbitrary: Works for any pair of inputs
Fast: No optimization required for testingBatch Norm vs Instance Norm
shifting and scaling theactivations
by mean and standard
deviation

𝐵𝑁 𝑥 = 𝛾𝑥−𝜇 𝑥
𝜎 𝑥+𝛽𝐼𝑁 𝑥 = 𝛾𝑥−𝜇 𝑥
𝜎 𝑥+𝛽Affined parameters
learned by gradientdescent

Source: https://medium.com/syncedreview/facebook-ai-proposes-groupnormalization-alternative-to-batch-normalization-fb0699bffae7

Adaptive Instance Normalization

Huang et al.,ICCV 2017𝑥𝑦𝑥 : content feature 𝑦 : style feature
Style Representation is feature statistics: Feature-wise mean and varianceStep 1 Normalize Content Image
Removes original style

Step 2 Align style image statistics
Affine parameters adapted from style image

Architecture

Huang et al.,ICCV 2017

VGG19 upto ReLU4-1

f(c):content feature

Decoder generates stylizedimages from feature map t

Stylized image

Decoderf(s):style feature

Align content statistics to style statistics
Output feature map t

Nearest UpsamplingTraining

Huang et al.,ICCV 2017
Content Loss•

ContentImage•

MS-COCO (80,000 images)

Euclidean distance between image
features of t and output image
t is AdaIN output -> better convergence

ContentImage

Style Loss
WikiArt (80,000 images)

Pre-process• Resize (512)• Ramdom Crop (256 x 256)

1_1

••

2_1

3_1

Match mean, variance between features of style image and output image
𝜙𝑖 denotes a layer in VGG. Used relu1_1, relu2_1, relu3_1, relu4_1 with equal weights

4_1

Implementation
Unofficial Implementation using Python and PyTorch, available on github• Built AdaIN architecture from scratch with PyTorch• Code for training (Because of time, I did not finish training. Currently, I use pretrained weights)• Code for testing• Basic Image Style Transfer• Style Interpolation• Video Style Transfer• Utility functions (grid image, … )I referred to:• Official Implementation in Torch7• Pretrained weights of encoder and decoder from naoto0804

Content Images

MSCOCOhttps://9gag.com/animals/amv2g
Oj?ref=fsidebar
https://inf.news/en/digital/cf25a5685
894c53d45c9b7c6a6040564.html

https://www.pinterest.com/pin/81831
8194786905146/

MSCOCOhttps://www.cbr.com/spider-man-noway-home-new-costume-goldemblem-marvel/
https://www.indiatimes.com/entertai
nment/tom-and-jerry-birthday-factsyou-did-not-know-561641.html
http://stevydya.blogspot.com/2015/04/the-university-of-tokyo.html,

Style Images fromWikiart, not seen by themodel during trainingTextureColorLevel of Style Transfer

Content Features

𝛼 = 0.0

𝛼 = 0.25

𝛼 = 0.5

Stylized Features

𝛼 = 0.75

𝛼 = 1.0

𝑆𝑡𝑦𝑙𝑒Style Interpolation

𝑤1𝑤2𝑤3𝑤4

Video Style Transform

https://github.com/naoto0804/pytorch-AdaIN/tree/master/input/videos

Speed Analysis
Image Style Transfer time: 0.09213 s/image ≈10 images per second▪ NVIDIA RTX 2060
▪ Image size 512 x 512px▪ Averaged over 128 iterations

WikiArt

Others

Quality Analysis

Lose more details
when style image notselected from WikiArt
Decoder is biased against
training dataset.

https://www.gruppodani.com/en/some-new-textures-designs-on-the-leather-that-speak-to-the-senses/
https://www.dreamstime.com/snake-skin-pattern-texture-repeating-seamless-fashionable-print-ready-textile-prints-image165760557

Applications
A Style-Based Generator Architecture for Generative
Adversarial Networks (StyleGAN), CVPR 2019

Pull Requests

Huang et al.,ICCV 2017

Spatial Control

Color Control

Anything Else！