The Unreasonable Effectiveness of Deep Features as a Perceptual MetricRichard Zhang12

Phillip Isola13

Alexei A. Efros1

1

Berkeley Artificial Intelligence ResearchEli Shechtman22

Adobe Research

Oliver Wang2

3OpenAIWhich image is more similar to the reference image?

PROBLEM STATEMENTQUALITATIVE EXAMPLES

Evaluate how well metrics correspond with human perceptual judgments.
1) Collect a large-scale perceptual similarity dataset2) Deep features across training objectives outperform widely-usedperceptual metrics (e.g., SSIM)
3) Train new metric (LPIPS) on perceptual judgmentsè Try it: richzhang.github.io/PerceptualSimilarity/

Patch 0DEEP NETWORKS AS A PERCEPTUAL METRIC

Which patch (left or right) is closer to the middle patch?

Reference

Patch 1Patch 0

Reference

Patch 1Patch 0

Reference

Patch 1!
"Humans!
"!
"#

Normalize
Normalize
Subtract!

"#

Subtract

Multiply
L2Multiplynorm
L2 norm
Spatial Average

Spatial Average& &Computing
Distance
ComputingDistance
Twoimages
è Distance

Avg

Avg

#$$
#$#$#
($($ %%ℎCrossCrossEntropy
Entropyℎ Loss Loss

Predicting
PerceptualJudgement
Predicting
PerceptualJudgementTrainingon2AFC

L2/PSNR, SSIM, FSIMJUST NOTICEABLE DIFFERENCES (JND)

Random Networks

TWO ALTERNATIVE FORCED CHOICE (2AFC)

Unsupervised Networks

Goal: Validate 2AFC with less “cognitively penetrable” test
Procedure: Ask human if 2 patches are identical or not
Evaluation: High correlation (ρ=.928) between 2AFC and JND

Self-Supervised Networks
Supervised Networks

Goal: Collect large-scale set of human perceptual judgments on distortions
Procedure: Sample a patch. Distort it twice. Ask human which is smaller.
Distortions for Train&Val:(1) Traditional distortions noise, photometric, blur, warps, compression(2) CNN-Based distortions Randomly generated denoising autoencoders
by varying hyperparameters
Distortions for Val only:(3) Real algorithms Outputs from superresolution, frame interpolation,
video deblurring, colorization algorithms
Original

HOW PERCEPTUAL ARE OFF-THE-SHELF NETWORKS?
Test on (1) Traditional + (2) CNN-based Distortions with off-the-shelf networks
82.6

Networks generalize “unreasonably” well across supervisory signals(supervised, self-supervised, unsupervised) and architectures

Human

75.5

74.8

Perturbed Patches76.478.0

76.8

Low-level

76.8

75.7

AlexNet (Random)
AlexNet (Unsupervised)
AlexNet (Self-supervised)

68.9(1) Traditional

L2

69.7

70.0

SSIM

FSIMc

70.6

Nets (Supervised:Imagenet classification)

Random K-Means Split-Brain BiGAN

PuzzleSqueeze AlexNet

Off-the-shelf (representation frozen, w=1)

Can we train with (1) traditional + (2) CNN-based distortions?
Critically, does it generalize to (3) real algorithms?
Perturbed Patches

80.6

79.8 79.9
78.0

Tuned (representation tuned, w learned)[1] Prashnani, Cai, Mostofi, Sen. PieAPP.
In CVPR, 2018. [Concurrent work]

79.4

78.7
76.8

Frozen (representation frozen, w learned)

82.6

81.4

75.7

74.7

Human

QUALITATIVE COMPARISONTRAINING ON PERCEPTUAL JUDGMENTS
Original

VGG

65.1
64.0

64.5

65.0 65.3

64.3(2) CNNbased

69.5

64.1 64.0
62.6
61.5

Squeeze

AlexNetVGG

PieAPP [1] Human(1) Traditional + (2) CNN-based Distortions

Squeeze

AlexNetVGG

PieAPP [1] Human(3) Real Algorithm Outputs

Off-the-shelf networks already perform well. Training a linear layer (LPIPS) on top yields small performance boost.
But fine-tuning through representation leads to overfitting on specified distortion distribution.

Though SSIM was not designed to handle geometric distortions, it
is commonly used, even when such distortions are a large factor.