Satoshi Iizuka*

Edgar Simo-Serra*
Waseda University(*equal contribution)

Hiroshi Ishikawa

Colorization of Black-and-white Pictures2

Our Goal: Fully-automatic colorization3

Colorization of Old Films

4

Related Work Scribble-based [Levin+ 2004; Yatziv+ 2004;
An+ 2009; Xu+ 2013; Endo+ 2016]
 Specify colors with scribbles
 Require manual inputs[Levin+ 2004]

 Reference image-based [Chia+ 2011;
Gupta+ 2012] Transfer colors of reference images
 Require very similar images
InputReference[Gupta+ 2012]

Output5

Related Work Automatic colorization with hand-crafted features [Cheng+ 2015]
 Uses existing multiple image features
 Computes chrominance via a shallow neural network Depends on the performance of semantic segmentation
 Only handles simple outdoor scenes

Imagefeatures
Input

Neural Network

ChromaOutput6

Contributions
 Novel end-to-end network that jointly learns global and local features

for automatic image colorization
 New fusion layer that elegantly merges the global and local features
 Exploit classification labels for learning

7

Layers of Our Model Fully-connected layer
 All neurons are connected between layers
 Convolutional layer Takes into account underlying spatial structure
No. of feature maps
𝑦Neuron…

…

Fully-connected layer𝑥Convolutional layer
8Our ModelLuminance

Colorization
Mid-Level Features
NetworkNetwork
Fusion Layer
ScalingUpsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network

 Two branches: local features and global features
 Composed of four networks
9Low-Level Features Network
Luminance

Colorization
Mid-Level Features
NetworkNetwork
Fusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network Extract low-level features such as edges and corners
 Lower resolution for efficient processing
10

Global Features Network
Luminance

Colorization
Mid-Level Features
NetworkNetwork
Fusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network

 Compute a global 256-dimensional vector representation of the image

11

Mid-Level Features Network
Luminance

Colorization
Mid-Level Features
NetworkNetwork
Fusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network Extract mid-level features such as texture

12Fusion Layer
Luminance

Colorization
Mid-Level Features
NetworkNetworkFusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network13

Fusion Layer
 Combine the global features with the mid-level features
 The resulting features are independent of any resolution
Mid-Level Features
Network

Fusion Layer

global𝐲fusion = 𝜎 𝐛 + 𝑊𝐲𝑢,𝑣mid𝐲𝑢,𝑣

Global Features Network

14Colorization Network
Luminance

Colorization
Mid-Level Features
NetworkNetwork
Fusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features Network

 Compute chrominance from the fused features
 Restore the image to the input resolution15Training of Colors
 Mean Squared Error (MSE) as loss function
 Optimization using ADADELTA [Zeiler 2012]
 Adaptively sets a learning rate
ForwardMSE
ModelBackward
Input

OutputGround truth

16

Joint Training
Luminance

Colorization
Mid-Level Features
NetworkNetworkFusion Layer
Scaling

Sharedweights

Upsampling

Chrominance

Low-Level Features
NetworkGlobal Features NetworkClassification
Network Training for classification jointly with the colorization
 Classification network connected to the global features

20.60% Formal Garden
16.13% Arch13.50% Abbey
7.07% Botanical Garden
6.53% Golf Course

Predicted labels17

Dataset MIT Places Scene Dataset [Zhou+ 2014]
 2.3 million training images with 205 scene labels
 256 × 256 pixels⋯AbbeyAirport terminal

Aquarium

Baseball field⋯
Dining room

Forest roadGas station

Gift shop

18

Computational Time Colorize within a few seconds

80ms

20

Colorization of MIT Places Dataset

21

Comparisons

Input[Cheng+ 2015]

Ours(w/o global features)

Ours(w/ global features)

22

Effectiveness of Global Features

Input

w/o global features

w/ global features
23

User Study 10 users participated We show 500 images of each type: total 1,500 images per user
 90% of our results are considered “natural”

NaturalUnnatural24

Colorization of Historical Photographs

Mount Moran, 1941

Scott's Run, 1937

Youngsters, 1912

Burns Basement, 191025

Style Transfer

Low-Level Features
Network26

Style Transfer

Low-Level Features
Network27

Style Transfer
 Adapting the colorization of one image to the style of another
Local

GlobalLocal

GlobalLocal

GlobalInputs

Output28

Limitations
 Difficult to output colorful images

Input

Ground truth

OutputGround truth

Output Cannot restore exact colors

Input

29

Conclusion
 Novel approach for image colorization by fusing global and local information
 Fusion layer
 Joint training of colorization and classification
 Style transfer

Farm Land, 1933

California NationalPark, 1936

Homes, 1936

Spinners, 1910

Doffer Boys, 190930

Thank you!
 Project Page

http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization
 Code on GitHub! https://github.com/satoshiiizuka/siggraph2016_colorization

Community Center, North Dome,
1936
1936

Norris Dam, 1933

Miner,1937
31