DeepRemaster: Temporal Source-Reference Attention
Networks for Comprehensive Video Enhancement
Satoshi Iizuka

sa2019.siggraph.org

Edgar Simo-Serra

Background• Vintage film is deteriorated• Noise, blur, and low contrast• Black and white or low quality colors• Digital remastering is a challenging task

“Oliver Twist”(1933)“A-Bomb Blast Effects”(1952)• Conducted manually by experts• Requires a significant amount ofboth time and money

Seven Samurai (1954)
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAOur Goal• Semi-automatically remastering of vintage films• This includes restoration, enhancement, and colorization

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIARelated Work• Image/video restoration• Gaussian noise [Dabov+ ’07, Maggioni+ ’12 ’14, Lefkimmiatis ’18]
• JPEG noise [Zhang+ ’17]
• Blur [Shi+ ’16]
Gaussian Noise• Image Colorization• Scribble-based [Levin+ 2004; Yatziv+ ’04;
An+ ’09; Xu+ ’13; Endo+ ’16; Zhang+ ’17]

• Reference-based [Chia+ ’11;
Gupta+ ’12; He+ ’18]

• Automatic [Iizuka+ ’16; Larsson+ ’16; Zhang+ ’16]

[Levin+ ’04]

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA[Zhang+ ’17]

Related Work• Reference-based video colorization• Recurrent neural networks [Liu+ ’18; Vondrick+ ’18]
• Processes a video by propagating color frame-by-frame• Cannot propagate between scene changes• Continues amplifying errors

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAOur Method• Model based on spatial and temporal convolutions• Automatic noise removal, super-resolution, and contrast adjustment• Semi-automatic colorization source-reference attention• Can colorize a video using an arbitrary number of reference images
References

Remastering
NetSA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAOur Network
Reference images

Temporal source-reference attention

InputPre-processing Network
Source-reference NetworkSA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

OutputOur Network
Reference images

Temporal source-reference attention

Input

Pre-processing Network
Source-reference Network
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

OutputPre-processing Network• Removes artifacts and noise from the input grayscale video• Formed exclusively by temporal convolutions

Temporal conv.

Input
Pre-processing Network
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAOur Network
Reference images

Temporal source-reference attention

Input

Pre-processing Network
Source-reference Network
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

OutputSource-reference Network• Takes the output of the pre-processing network along with an arbitrary numberof reference color images
Reference images• Colorizes the frames based on the reference imagesby using source-reference attention
Source-reference attention

Temporal conv.

Spatial conv.

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Temporal Source-reference Attention• Compute similarity between the source images and reference images• Actually computed on feature maps
Feature map

References

Input frame

source-reference attention
SA2019.SIGGRAPH.ORG

CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Advantages of Source-Reference Attention•••

Cannot form long-term dependencies
Temporal consistency is lost when a new reference is usedRequire precise scene segmentation

Recursion-based network

Can use all the color reference
information for colorization
Our temporal source-reference attention
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Optimization• Combination of two L1 losses• Fully supervised learning• Uses ADADELTA[Zeiler ’12] for optimization• Objective function:

Output ofsource-reference network

Output ofpre-processing network

Ground truth
luminance

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Ground truth
chrominanceTraining Data Generation• Example-based and algorithm-based deterioration• Example-based: scratch noise, fractal noise, dust noise, …
• Algorithm-based: Gaussian noise, blur, low contrast• 1200 videos from Youtube8M[Abu-El-Haji+ ’16] for training

Examples of noise data

Original
SA2019.SIGGRAPH.ORG

CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Deteriorated

Results

Comparisons

Input[Yu+ ’18]&[Zhang+ ’17a][Zhang+ ’17b]&[Vondrick+ ’18]

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAOurs

Quantitative Result
Remastering results

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAQuantitative Results
Colorization results

Restoration results

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIAComparisons[Yu+ ’18]&[Zhang+ ’17a]

Input[Zhang+ ’17b]&[Vondrick+ ’18]

Reference images(manually created)
Ours
SA2019.SIGGRAPH.ORG

CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Results
Reference images
Attention

Input

Output
SA2019.SIGGRAPH.ORG

CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Results
Reference images
Attention

Input

Output
SA2019.SIGGRAPH.ORG

CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Results“Isewan typhoon” (1959), the original film is provided by CBC Television Co.
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Restoration Results• Large noise removal

Input[Zhang et al. 2017b]
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

Ours

Limitations• Severely deteriorated film is difficult to remaster• Cannot fill large missing regions• Scene with intense motion

Input

OutputInput

SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA

OutputConclusion• Novel single framework to tackle entire remastering task

• Automatic noise removal, super-resolution, and contrast adjustment• Reference-based colorization via temporal source-reference attention• Significant improvement with respect to existing methods• Applicable to other reference-based image/video processing• GitHub:
https://github.com/satoshiiizuka/siggraphasia2019_remastering
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIA