Shift-tolerant Perceptual Similarity Metric

1 INTRODUCTION . Image similarity measurement is a common task for many computer vision and computer graphics applications . General similarity metrics like PSNR and RMSE , however , do not match the human visual perception well when assessing the similarity between two images . Therefore , many dedicated image similarity metrics , such as Structural Similarity ( SSIM ) and its variations ( Wang et al. , 2004 ; 2003 ; Zhang et al. , 2011 ; Wang & Simoncelli , 2005 ) , were developed in order to more closely reflect the human perception . However , manually crafting a perceptual similarity metric remains a challenging task as it involves the complex human cognitive judgement ( Medin et al. , 1993 ; Tversky , 1977 ; Wang et al. , 2004 ; Zhang et al. , 2018 ) . I0 Iref I1 Recently , learning-based image similarity metrics have been developed . These metrics learn from a large set of labelled data and predict the similarity between images that correlates well with human perception ( Bhardwaj et al. , 2020 ; Ding et al. , 2020 ; Kettunen et al. , 2019 ; Prashnani et al. , 2018 ; Zhang et al. , 2018 ; Czolbe et al. , 2020 ) . Among them , the Learned Perceptual Image Patch Similarity metric ( LPIPS ) by Zhang et al . ( 2018 ) is now widely adopted as a perceptual similarity metric and used in computer graphics and vision literature . This paper studies how image similarity metrics work on a pair of images that are not perfectly aligned . For instance , a tiny misalignment in the image pair such as a one-pixel translation between them , is imperceptible to the human eyes . But , will such a visually imperceptible misalignment compromise any existing similarity metrics ? For PSNR and RMSE , since they assume pixel-wise registration , naturally they are sensitive to as small as a one-pixel misalignment . As we will detail in this paper , our study found that the learned perceptual similarity metrics , such as LPIPS , are also sensitive to a small misalignment . Figure 1 shows such an example through a two-alternative forced choice test . In this test , viewers were asked “ which of the two distorted images , I0 or I1 , is more similar to the reference image Iref ? ” Then , we shifted I0 and I1 by one pixel and obtained their opinions again . None of 1We will make our code and data publicly available . the participants flipped their opinions from I0 to I1 or vice versa , which is intuitive as a one-pixel shift is imperceptible to viewers . But existing metrics , such as MS-SSIM and LPIPS , flipped their judgments after the one-pixel shift . Our problem is related to the recent work on making deep neural networks shift invariant ( Islam et al. , 2020 ; Kayhan & Gemert , 2020 ; Vasconcelos et al. , 2021 ; Zhang , 2019 ; Zou et al. , 2020 ; Lee et al. , 2020 ) . In a recent study , Azulay & Weiss ( 2019 ) found that an image classifier can change its top-1 prediction if the image is translated by only one pixel . Their results showed that after translating an image by one pixel , the classifier made a different top-1 prediction for 30 % of the 1000 validation images . Zhang ( 2019 ) introduced anti-aliasing filters into a deep neural network to make the feature extraction network shift-equivariant , which in term makes the whole network shift-invariant for the down streaming tasks . Compared to these works , our problem is different in that 1 ) a perceptual similarity metric takes two images as input instead of working on a single input image , and 2 ) only one of the two images is shifted , thus introducing imperceptible misalignment instead of shifting the two images simultaneously . This paper aims to develop a shift-tolerant perceptual similarity metric that correlates well with the human judgement on the similarity between images while being robust against imperceptible misalignment between them . We build our metric upon LPIPS , a deep neural network-based metric that is now widely adopted for its close correlation with the human perception . We investigate a variety of elements that can be incorporated into a deep neural network to make it resistant to an imperceptible misalignment . These elements include anti-aliasing filters , striding , pooling , padding , placement of anti aliasing , etc . Based on our findings on these elements , we develop a shift-tolerant perceptual similarity metric that not only is more consistent with human perception but also is significantly more resistant to imperceptible misalignment between a pair of images than existing metrics . In the remainder of this paper , we first report our study that verifies that viewers are not sensitive to small amount of shifts between two images when comparing them , in Section 3 . We then benchmark existing visual similarity metrics and show that these metrics are sensitive to imperceptible shifts between a pair of images in Section 4 . We then study several important elements that make a deep neural network-based similarity metric both tolerant to imperceptible shifts and consistent with the human perception of visual similarity in Section 5 . We finally report our experiments that thoroughly evaluate our new perceptual similarity metric by comparing it to state of the art metrics and through detailed ablation studies in Section 6 . 2 RELATED WORK . Visual similarity metrics are commonly used to compare two images or evaluate the performance of many image and video processing , editing and synthesis algorithms . While there are already many established metrics for these tasks , such as PSNR , MSE , SSIM and its variations ( Wang et al. , 2004 ; 2003 ; Wang & Simoncelli , 2005 ) , there is still a gap between their prediction and the human ’ s judgement . This section provides a brief overview of the recent advances in learned perceptual similarity metrics that aim to bridge the gap mentioned above . In their influential work , Zhang et al . ( 2018 ) reported that features from a deep neural network can be used to measure the similarity between two images that is more consistent with the human perception than other commonly used metrics . Accordingly , they developed LPIPS , a perceptual metric learned from a large collection of labelled data . Specifically , LPIPS uses a pre-trained network for image classification tasks or learns a neural network to compute the features for each of the two images or patches , and also learns to aggregate the feature distances into a similarity score . Since its debut , LPIPS has been widely used as a perceptual quality metric . On a related note , the computer vision and graphics community also calculate the difference between the deep features of two images as a loss function to train deep neural networks for image enhancement and synthesis . Such a loss function , often called perceptual loss , enables the neural networks to learn to generate perceptually pleasing images ( Dosovitskiy & Brox , 2016 ; Johnson et al. , 2016 ; Ledig et al. , 2016 ; Niklaus et al. , 2017 ; Sajjadi et al. , 2016 ; Zhu et al. , 2016 ) . Kettunen et al . ( 2019 ) developed the E-LPIPS metric that adopts the LPIPS network and uses randomly transformed samples to calculate expected LPIPS distance over them . They showed that ELPIPS is robust against the Expectation Over Transformation attack ( Athalye et al. , 2018 ) . Different from LPIPS , Prashnani et al . ( 2018 ) use the differences between features to generate patch-wise errors and corresponding weights , via two different fully-connected networks . Their final similarity score is a weighted average of the patch-wise distances . Czolbe et al . ( 2020 ) developed a similarity metric based on Watson ’ s perceptual model ( Watson , 1993 ) , by replacing discrete cosine transform with discrete fourier transform ( DFT ) . They posit that their metric is robust against small translations and is sensitive to large translations . Czolbe et al . ( 2020 ) used Watson-DFT as a differentiable loss function for image generation via variational autoencoders ( Kingma & Welling , 2013 ) . In earlier work , Wang & Simoncelli ( 2005 ) improved SSIM ( Wang et al. , 2004 ) by replacing the spatial correlation measures with phase correlations in wavelet subbands which made the metric less sensitive to geometric transformations . Ma et al . ( 2018 ) developed a geometric transformation invariant method ( GTI-CNN ) . Our work is closely related to theirs , as GTI-CNN is a similarity metric that is invariant to the misalignment between a pair of images . In their method , Ma et al . ( 2018 ) train a fully convolutional neural network to extract deep features from each image and calculate the mean squared error between them as their final similarity . They showed that training the fully convolutional neural network directly on aligned samples leads to a metric that is sensitive to the misalignment , which is consistent with what we found in our study . They reported that augmenting the training samples with small misalignment can make the learned metric significantly more resistant to the misalignment . Compared to this method , our work focuses on designing a deep neural network architecture that is robust to misalignment without any data augmentation . Bhardwaj et al . ( 2020 ) followed the understanding of the physiology of the human visual system and developed a fully convolutional neural network that generates a multi-scale probabilistic representation of an input image and then calculates the symmetric Kullback–Leibler divergences between such representations of two images to measure their similarity . They found that such a similarity metric is robust against small shifts between a pair of images . While benchmarking existing metrics , our study also finds that their metric is most robust against small shifts among all the metrics we tested . We posit that the robustness of their method partially comes from training their metric on neighboring video frames that might already have small shifts among them , thus effectively serving as data augmentation , as done by Ma et al . ( 2018 ) . We consider these as orthogonal efforts in developing a robust similarity metric . Also , as shown in our study , our metric is more consistent with the human judgement and more robust against imperceptible misalignment than these methods , even though our metric is trained on aligned samples directly without any data augmentation . Our work is most related to deep image structure and texture similarity ( DISTS ) metric by Ding et al . ( 2020 ) . They used global feature aggregation to make DISTS robust against mild geometric transformations . They also replaced the max pooling layers with l2 pooling layers ( Hénaff & Simoncelli , 2016 ) in their VGG backbone network for anti-aliasing and found that blurring the input with l2 pooling makes their network more robust against small shifts . Gu et al . ( 2020 ) found that existing metrics like LPIPS do not perform well with images generated by GAN-based restoration algorithms . They attributed it to the small misalignment between the GAN results and the ground truth . Therefore , they used l2 pooling ( Ding et al. , 2020 ; Hénaff & Simoncelli , 2016 ) and BlurPool ( Zhang , 2019 ) to improve LPIPS . They found that both can improve LPIPS while BlurPool performs better . Compared to these two recent papers , our paper systematically investigates a broad range of neural network elements besides BlurPool . By integrating these elements together , we develop a perceptual similarity metric that is both robust against small shifts and is consistent with the human visual similarity judgement . Our method outperforms existing metrics , and a variety of recently developed learned metrics . Integrating multiple network elements together makes our metric better than individual ones , including BlurPool .

Summary: The paper proposes a few modifications to the existing archiectures for perceptual image similarity that would be robust to the tiny shifts in the image. Specifically, the authors conduct an experiemnt that shows that humans mostly are not sensetive to small shifts of 1 or 2 pixels (Table 1), but most state of the art perceptual image similarity networks are sensetive to such shifts (Table 1 and Table 2). The authors discuss what parts of the network architecture might yield such sensetivity and offer a few modifications (Section 5). In Section 6, the authors perform extensive experiments to check the effectiveness of their suggestions. Finally the authors conclude that using anti-aliasing strided convolutions and pooling operators and reducing stride size are helpful to make a learned similarity metric shift-invariant. Their experiments show that by integrating these elements into a neural network, the learned metric is more robust against imperceptible shifts and more consistent with the human visual similarity judgment  (Table 1,2,3) .