Shift-tolerant Perceptual Similarity Metric

1 INTRODUCTION . Image similarity measurement is a common task for many computer vision and computer graphics applications . General similarity metrics like PSNR and RMSE , however , do not match the human visual perception well when assessing the similarity between two images . Therefore , many dedicated image similarity metrics , such as Structural Similarity ( SSIM ) and its variations ( Wang et al. , 2004 ; 2003 ; Zhang et al. , 2011 ; Wang & Simoncelli , 2005 ) , were developed in order to more closely reflect the human perception . However , manually crafting a perceptual similarity metric remains a challenging task as it involves the complex human cognitive judgement ( Medin et al. , 1993 ; Tversky , 1977 ; Wang et al. , 2004 ; Zhang et al. , 2018 ) . I0 Iref I1 Recently , learning-based image similarity metrics have been developed . These metrics learn from a large set of labelled data and predict the similarity between images that correlates well with human perception ( Bhardwaj et al. , 2020 ; Ding et al. , 2020 ; Kettunen et al. , 2019 ; Prashnani et al. , 2018 ; Zhang et al. , 2018 ; Czolbe et al. , 2020 ) . Among them , the Learned Perceptual Image Patch Similarity metric ( LPIPS ) by Zhang et al . ( 2018 ) is now widely adopted as a perceptual similarity metric and used in computer graphics and vision literature . This paper studies how image similarity metrics work on a pair of images that are not perfectly aligned . For instance , a tiny misalignment in the image pair such as a one-pixel translation between them , is imperceptible to the human eyes . But , will such a visually imperceptible misalignment compromise any existing similarity metrics ? For PSNR and RMSE , since they assume pixel-wise registration , naturally they are sensitive to as small as a one-pixel misalignment . As we will detail in this paper , our study found that the learned perceptual similarity metrics , such as LPIPS , are also sensitive to a small misalignment . Figure 1 shows such an example through a two-alternative forced choice test . In this test , viewers were asked “ which of the two distorted images , I0 or I1 , is more similar to the reference image Iref ? ” Then , we shifted I0 and I1 by one pixel and obtained their opinions again . None of 1We will make our code and data publicly available . the participants flipped their opinions from I0 to I1 or vice versa , which is intuitive as a one-pixel shift is imperceptible to viewers . But existing metrics , such as MS-SSIM and LPIPS , flipped their judgments after the one-pixel shift . Our problem is related to the recent work on making deep neural networks shift invariant ( Islam et al. , 2020 ; Kayhan & Gemert , 2020 ; Vasconcelos et al. , 2021 ; Zhang , 2019 ; Zou et al. , 2020 ; Lee et al. , 2020 ) . In a recent study , Azulay & Weiss ( 2019 ) found that an image classifier can change its top-1 prediction if the image is translated by only one pixel . Their results showed that after translating an image by one pixel , the classifier made a different top-1 prediction for 30 % of the 1000 validation images . Zhang ( 2019 ) introduced anti-aliasing filters into a deep neural network to make the feature extraction network shift-equivariant , which in term makes the whole network shift-invariant for the down streaming tasks . Compared to these works , our problem is different in that 1 ) a perceptual similarity metric takes two images as input instead of working on a single input image , and 2 ) only one of the two images is shifted , thus introducing imperceptible misalignment instead of shifting the two images simultaneously . This paper aims to develop a shift-tolerant perceptual similarity metric that correlates well with the human judgement on the similarity between images while being robust against imperceptible misalignment between them . We build our metric upon LPIPS , a deep neural network-based metric that is now widely adopted for its close correlation with the human perception . We investigate a variety of elements that can be incorporated into a deep neural network to make it resistant to an imperceptible misalignment . These elements include anti-aliasing filters , striding , pooling , padding , placement of anti aliasing , etc . Based on our findings on these elements , we develop a shift-tolerant perceptual similarity metric that not only is more consistent with human perception but also is significantly more resistant to imperceptible misalignment between a pair of images than existing metrics . In the remainder of this paper , we first report our study that verifies that viewers are not sensitive to small amount of shifts between two images when comparing them , in Section 3 . We then benchmark existing visual similarity metrics and show that these metrics are sensitive to imperceptible shifts between a pair of images in Section 4 . We then study several important elements that make a deep neural network-based similarity metric both tolerant to imperceptible shifts and consistent with the human perception of visual similarity in Section 5 . We finally report our experiments that thoroughly evaluate our new perceptual similarity metric by comparing it to state of the art metrics and through detailed ablation studies in Section 6 . 2 RELATED WORK . Visual similarity metrics are commonly used to compare two images or evaluate the performance of many image and video processing , editing and synthesis algorithms . While there are already many established metrics for these tasks , such as PSNR , MSE , SSIM and its variations ( Wang et al. , 2004 ; 2003 ; Wang & Simoncelli , 2005 ) , there is still a gap between their prediction and the human ’ s judgement . This section provides a brief overview of the recent advances in learned perceptual similarity metrics that aim to bridge the gap mentioned above . In their influential work , Zhang et al . ( 2018 ) reported that features from a deep neural network can be used to measure the similarity between two images that is more consistent with the human perception than other commonly used metrics . Accordingly , they developed LPIPS , a perceptual metric learned from a large collection of labelled data . Specifically , LPIPS uses a pre-trained network for image classification tasks or learns a neural network to compute the features for each of the two images or patches , and also learns to aggregate the feature distances into a similarity score . Since its debut , LPIPS has been widely used as a perceptual quality metric . On a related note , the computer vision and graphics community also calculate the difference between the deep features of two images as a loss function to train deep neural networks for image enhancement and synthesis . Such a loss function , often called perceptual loss , enables the neural networks to learn to generate perceptually pleasing images ( Dosovitskiy & Brox , 2016 ; Johnson et al. , 2016 ; Ledig et al. , 2016 ; Niklaus et al. , 2017 ; Sajjadi et al. , 2016 ; Zhu et al. , 2016 ) . Kettunen et al . ( 2019 ) developed the E-LPIPS metric that adopts the LPIPS network and uses randomly transformed samples to calculate expected LPIPS distance over them . They showed that ELPIPS is robust against the Expectation Over Transformation attack ( Athalye et al. , 2018 ) . Different from LPIPS , Prashnani et al . ( 2018 ) use the differences between features to generate patch-wise errors and corresponding weights , via two different fully-connected networks . Their final similarity score is a weighted average of the patch-wise distances . Czolbe et al . ( 2020 ) developed a similarity metric based on Watson ’ s perceptual model ( Watson , 1993 ) , by replacing discrete cosine transform with discrete fourier transform ( DFT ) . They posit that their metric is robust against small translations and is sensitive to large translations . Czolbe et al . ( 2020 ) used Watson-DFT as a differentiable loss function for image generation via variational autoencoders ( Kingma & Welling , 2013 ) . In earlier work , Wang & Simoncelli ( 2005 ) improved SSIM ( Wang et al. , 2004 ) by replacing the spatial correlation measures with phase correlations in wavelet subbands which made the metric less sensitive to geometric transformations . Ma et al . ( 2018 ) developed a geometric transformation invariant method ( GTI-CNN ) . Our work is closely related to theirs , as GTI-CNN is a similarity metric that is invariant to the misalignment between a pair of images . In their method , Ma et al . ( 2018 ) train a fully convolutional neural network to extract deep features from each image and calculate the mean squared error between them as their final similarity . They showed that training the fully convolutional neural network directly on aligned samples leads to a metric that is sensitive to the misalignment , which is consistent with what we found in our study . They reported that augmenting the training samples with small misalignment can make the learned metric significantly more resistant to the misalignment . Compared to this method , our work focuses on designing a deep neural network architecture that is robust to misalignment without any data augmentation . Bhardwaj et al . ( 2020 ) followed the understanding of the physiology of the human visual system and developed a fully convolutional neural network that generates a multi-scale probabilistic representation of an input image and then calculates the symmetric Kullback–Leibler divergences between such representations of two images to measure their similarity . They found that such a similarity metric is robust against small shifts between a pair of images . While benchmarking existing metrics , our study also finds that their metric is most robust against small shifts among all the metrics we tested . We posit that the robustness of their method partially comes from training their metric on neighboring video frames that might already have small shifts among them , thus effectively serving as data augmentation , as done by Ma et al . ( 2018 ) . We consider these as orthogonal efforts in developing a robust similarity metric . Also , as shown in our study , our metric is more consistent with the human judgement and more robust against imperceptible misalignment than these methods , even though our metric is trained on aligned samples directly without any data augmentation . Our work is most related to deep image structure and texture similarity ( DISTS ) metric by Ding et al . ( 2020 ) . They used global feature aggregation to make DISTS robust against mild geometric transformations . They also replaced the max pooling layers with l2 pooling layers ( Hénaff & Simoncelli , 2016 ) in their VGG backbone network for anti-aliasing and found that blurring the input with l2 pooling makes their network more robust against small shifts . Gu et al . ( 2020 ) found that existing metrics like LPIPS do not perform well with images generated by GAN-based restoration algorithms . They attributed it to the small misalignment between the GAN results and the ground truth . Therefore , they used l2 pooling ( Ding et al. , 2020 ; Hénaff & Simoncelli , 2016 ) and BlurPool ( Zhang , 2019 ) to improve LPIPS . They found that both can improve LPIPS while BlurPool performs better . Compared to these two recent papers , our paper systematically investigates a broad range of neural network elements besides BlurPool . By integrating these elements together , we develop a perceptual similarity metric that is both robust against small shifts and is consistent with the human visual similarity judgement . Our method outperforms existing metrics , and a variety of recently developed learned metrics . Integrating multiple network elements together makes our metric better than individual ones , including BlurPool .

Summary: The authors propose to make perceptual similarity metrics (PSM) invariant to small-shifts (few pixels translation) and still consistent with human judgement. To this end, they use an approach based on network architectures to evaluate which elements (anti-aliasing, pooling, striding, padding, skip connection) can achieve shift-invariance.  The paper have multiple contributions: 1. A study on the human perception of small shift estimating their sensitivity 2. A study of the sensitivity of PSMs to small shift 3. A systematic study of neural network architecture elements in relation to shift-invariance 4. A updated/shift-invariant version of the LPIPS metric 5. An ablation study to evaluate which elements contribute or to shift-invariance