Shift-tolerant Perceptual Similarity Metric
Authors: Abhijay Ghildyal
Feng Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Computer Vision
Perceptual Similarity Metric
Image Quality Assessment
Robustness
Convolutional Neural Networks
Anti-aliasing

Abstract: Existing perceptual similarity metrics assume an image and its reference are well aligned. As a result, these metrics are often sensitive to a small alignment error that is imperceptible to the human eyes. This paper studies the effect of small misalignment, specifically a small shift between the input and reference image, on existing metrics, and accordingly develops a shift-tolerant similarity metric. This paper builds upon LPIPS, a widely used learned perceptual similarity metric and explores architectural design considerations to make it robust against imperceptible misalignment. Specifically, we study a wide spectrum of neural network elements, such as anti-aliasing filtering, pooling, striding, padding, and skip connection, and discuss their roles in making a robust metric. Based on our studies, we develop a new deep neural network-based perceptual similarity metric. Our experiments show that our metric is tolerant to imperceptible shifts while being consistent with the human similarity judgment.

Under review as a conference paper at ICLR 2022
SHIFT-TOLERANT PERCEPTUAL SIMILARITY METRIC
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing perceptual similarity metries assume an image and its reference are well aligned. As result, these metrics are often sensitive to a small alignment enror that is imperceptible to the human eyes. This paper studies the effect of small misalignment, specifically a small shift between the input and reference image, on existing metrics, and accordingly develops a shift-tolcrant similarity metric. This paper builds upon LPIPS, widely used learned perceptual similarity metric and explores architectural design considerations to make robust against imper- ceptible misalignment. Specifically, we study wide spectrum of neural network elements, such as anti-aliasing filtering, pooling, striding, padding, and skip con- nection, and discuss their roles in making a robust metric. Based our studies, we develop new deep neural network-baser perceptual similarity metriq" Our experiments show that our metric is tolerant to imperceptible shifts while being consistent with the human similarity judgment.
INTRODUCTION
Image similarity measurement is common task for many computer vision and computer graphics applications. General similarity metrics like PSNR and RMSE, however, do not match the human visual perception well when assessing the similarity between two images. Therefore, many dedi- cated image similarity metries, such as Structural Similarity (SSIM) and its variations (Wang al. 2004 2003 Zhang et al. 2011 Wang & Simoncelli] 2005), were developed in order to more closely reflect the human perception. However, manually crafting perceptual similarity metric remains
challenging task as it involves the complex human cognitive judgement (Medin et al. 1993 Tversky
1977 Wang et al., 2004 Zhang et al., 2018).
10
Recently, learning-baac based image similarity metrics have been developed. These metrics learn from a large set of labelled data and predict the similarity between images that correlates well with human perception (Bhardwaj ct 2020, Ding al. 2020 Kettunen et al., 2019
I-pis-shif
Meine
Prashnani et al., 2018 Zhang et al. 2018: Czolbe et al. 2020). Among them, the Learned Perceptual Image
Humans
MS-SSIM
Patch Similarity metric (LPIPS) by Zhang et al. (2018) now widely adopted perceptual similarity metric
1.2
LPIPS (Alex)
Figure 1: Whether two images Lo and I are shifted by 1-pixel or not, viewers al- ways consider Iı as more similar to Ires than Io. However, existing metrics often switch their predictions after the impercep-
and used in computer graphics and vision literature.
This paper studies how image similarity metrics work on pair of images that are not perfectly aligned. For instance, tiny misalignment in the image pair such as one-pixel translation between them, is imperceptible tible 1-pixel shift.
to the human eyes. But, will such visually imperceptible misalignment compromise any existing similarity metrics? For PSNR and RMSE, since they assume pixel-wise registration, naturally they are sensitive to as small as a one-pixel misalignment. As we will detail in this paper, our study found that the learned perceptual similarity metrics, such as LPIPS, are also sensitive to small misalignment. Figure T shows such an example through a two-alternative forced choice test. In this test, viewers were asked "which of the two distorted images, In I,, is more similar to the reference image Tres? ?" Then, we shifted To and J by one pixel and obtained their opinions again. None of
We will make our code and data publicly available.

---

Under review as a conference paper at ICLR 2022
the participants flipped their opinions from Io to I1 or vice versa, which is intuitive one one-pixel shift is imperceptible to viewers. But existing metrics, such as MS-SSIM and LPIPS, flipped their judgments after the one-pixel shift.
Our problem is related to the recent work on making deep neural networks shift invariant (Islam
2020 Kayhan & Gemert 2020 Vasconcelos al. 2021 Zhang 2019 Zou et al. 2020 Lee et al.
et al. 2020). In a recent study, Azulay & Weiss (2019) found that an image classifier can change its top-1 prediction if the image is translated by only one pixel. Their results showed that after translating an image by one pixel, the classifier made a different top-1 prediction for 30% of the 1000 validation images. Zhhnng (2019) introduced anti-aliasing filters into a deep neural network to make the feature extraction network shift-equivariant, which in term makes the whole network ihift-invariant for the down streaming tasks. Compared to these works, our problem is different in that 1) perceptual similarity metric takes two images input instead of working on a single input image, and 2) only one of the two images is shifted, thus introducing imperceptible misalignment instead of shifting the two images simultaneously.
This paper aims to develop a shift-tolerant perceptual similarity metric that correlates well with the human judgement on the similarity between images while being robust against imperceptible mis- alignment between them. We build our metric upon LPIPS, a deep neural network-baa based metric that is now widely adopted for its close correlation with the human perception. We investigate a variety of elements that can be incorporated into a deep neural network to make it resistant to an impercepti- ble misalignment. These elements include anti-aliasing filters, striding. pooling, padding, placement of anti aliasing, etc. Based on our findings on these elements, we develop a shift-tolerant perceptual similarity metric that not only is more consistent with human perception but also is significantly more resistant to imperceptible misalignment between a pair of images than existing metrics.
In the remainder of this paper, we first report our study that verifies that viewers are not sensitive to small amount of shifts between two images when comparing them, in Section 3 We then benchmark existing visual similarity metrics and show that these metries are sensitive to imperceptible shifts between a pair of images in Section E We then study several important elements that make deep neural network- -aaed similarity metric both tolerant to imperceptible shifts and consistent with the human perception of visual similarity in Section[5] We finally report our experiments that thoroughly evaluate our new perceptual similarity metric by comparing it to state of the art metrics and through detailed ablation studies in Section
2 RELATED WORK
Visual similarity metrics are commonly used to compare two images or evaluate the performance of many image and video processing, editing and synthesis algorithms. While there are already many established metrics for these tasks, such as PSNR, MSE, SSIM and its variations (Wang et al,) 2004 [003] Wang & Simoncelli 2005), there is still gap between their prediction and the human's judgement. This section provides a brief overview of the recent advances in learned perceptual similarity metrics that aim bridge the gap mentioned above.
In their influential work, Zhang et al. (2018) reported that features from a deep neural network can be used measure the similarity between two images that is more consistent with the human perception than other commonly used metrics. Accordingly, they developed LPIPS, a perceptual metric learned from large collection of labelled data. Specifically, LPIPS uses pre-trained network for image classification tasks or learns a neural network to compute the features for each of the two images or patches, and also learns to aggregate the feature distances into a similarity score. Since its debut, LPIPS has been widely used as perceptual quality metric. On a related note, the computer vision and graphics community also calculate the difference between the deep features of two images as a loss function to train deep neural networks for image enhancement and synthesis. Such loss function, often called perceptual loss, enables the neural networks to learn to generate perceptually pleasing images (Dosovitskiy & Brox 2016; Johnson et al. 2016 Ledig et al. 2016, Niklaus al. 2017 Sajjadi et al. 2016 Zhu et al. 2016).
Kettunen et al. 2019) developed the E-LPIPS metric that adopts the LPIPS network and uses ran- domly transformed samples to calculate expected LPIPS distance over them. They showed that E- LPIPS is robust against the Expectation Over Transformation attack (Athalye al. 2018). Different
2

---

Under review as a conference paper at ICLR 2022
from LPIPS, Prashnani et al. (2018) use the differences between features to generate patch-wise errors and corresponding weights, via two different fully-connected networks. Their final similarity score is weighted average of the patch- wise distances. Czolbe al. (2220) developed similarity metric based on Watson's 's perceptual model (Watson 1993), by replacing discrete cosine transform with discrete fourier transform (DFT). They posit that their metrie is robust against small transla- tions and is sensitive to large translations. Czolbe et al. (2020) used Watson-DFT as a differentiable loss function for image generation via variational autoencoders (Kingma & Welling 2013).
In earlier work, Wang & Simoncelli (2055) improved SSIM (Wang et al. 2004 by replacing the spatial correlation measures with phase correlations in wavelet subbands which made the metric less sensitive to geometric transformations. Ma et al. (2018) developed a geometric transformation invariant method (GTI-CNN). Our work is closely related to theirs, as GTI-CNN a similarity met- ric that is invariant to the misalignment between a pair of images. In their method, Ma et al. (0188) train a fully convolutional neural network to extract deep features from each image and calculate the mean squared error between them as their final similarity. They showed that training the fully convolutional neural network directly on aligned samples leads to a metric that is sensitive to the misalignment, which is consistent with what we found in our study. They reported that augment- ing the training samples with small misalignment can make the learned metric significantly more resistant to the misalignment. Compared to this method, our work foeuses on designing deep neu- ral network architecture that is robust to misalignment without any data augmentation. [Bhardwaj al. (2020) followed the understanding of the physiology of the human visual system and devel- oped fully convolutional neural network that generates a multi-scale probabilistic representation of an input image and then calculates the symmetric Kullback- divergences between such representations of two images to measure their similarity. They found that such similarity metric robust against small shifts between a pair of images. While benchmarking existing metrics, our study also finds that their metric is most robust against small shifts among all the metrics we tested. We posit that the robustness of their method partially comes from training their metric on neighbor- ing video frames that might already have small shifts among them, thus effectively serving data augmentation, as done by Ma et al. (2018). We consider these as orthogonal efforts in developing a robust similarity metric. Also, shown in our study, our metric is judgement and more robust against imperceptible misalignment than these methods, even though our metric is trained on aligned samples directly without any data augmentation.
consistent with the human
Our work is most related to deep image structure and texture similarity (DISTS) metric by Ding let al. (2020). They used global feature aggregation make DISTS robust against mild geometric transformations. They also replaced the max pooling layers with la pooling layers (Hénaff & Si- moncelli 2016) in their VGG backbone network for anti-aliasing and found that blurring the input with le pooling makes their network more robust against small shifts. Gu et al. (020)) found that existing metries like LPIPS do not perform well with images generated by Gasebased restoration algorithms. They attributed it to the small misalignment between the GAN results and the ground truth. Therefore, they used l2 pooling (Ding et al., 2020, Hénaff & Simoncelli] 2016) and Blur- Pool (Zhang 2019) to improve LPIPS. They found that both can improve LPIPS while BlurPool performs better. Compared to these two recent papers, our paper systematically investigates a broad range of neural network elements besides BlurPool. By integrating these elements together, de- velop a perceptual similarity metric that is both robust against small shifts and is consistent with the human visual similarity judgement. Our method outperforms existing metries, and variety of re- cently developed learned metrics. Integrating multiple network elements together makes our metric better than individual ones, including BlurPool.
3 HUMAN PERCEPTION OF SMALL SHIFTS
As commonly expected, shifting one image by few pixels will not alter human similarity judgement on a pair of images (Bhardwaj et al., 2020 Xiao et al. 2018). We conducted a user study to verify this common belief. Our hypothesis is that it is difficul for people detect small shift in images.
In our study, we randomly picked 50 images from the MS-COCO test dataset (Lin et al. 2014). For each participant, we randomly divided these 50 images into 10 groups, each with 5 images. For each image in Group n with n E [o, 10), we cropped a 256 x 256 patch as a reference image and shifted the cropping window by n pixels to produce its shifted version. In this way, we generated
3

---

Under review as a conference paper at ICLR 2022
50 pairs of images for each participant. In the end, since we always cropped the reference from the same location, we had an p-pixll shifted version for each of the 50 images, and thus in total, we have 500 pairs of images in our study. During the study, each participant was presented with a pair of images one at a time. The order of the 50 pairs of images is also randomized for each participant. In each trial, a pair of images were placed side by side. The position of the reference image, e.g., right or left, is randomized to avoid any bias. We asked our participants to judge whether a pair of images are the same or not. We refer the reader to Appendix A.1 for more details.
the amount of shift is small, participants find it
We report the user responses in Figure 2 When
150-
130
140
93.3%
80.7%)
Yes (They same)
No (One of them appears shifted)
difficult to detect the shift. For pairs of images
120
110
with the 1-pixel shift and 2-pixel shift, they were considered the same in 80.7% and 56.0% of the responses, respectively. As expected, the shifts become easier to detect the size of the shifts increases. But even for pairs of the 5-pixel shift, they were still not identified in 26.7% of the sponses. As shown in our study, even after being informed about the possible shifts, participants still had difficulty in detecting small shifts. This verifies our hypothesis that difficult for peo- ple to detect small shift in images. In addition,
100-
90-
56.0%
70-
34.7% 34.7%
40-
5u-
26.7%
23.3%
s0
18.0%
20-
10.0%
10-
No
Horizontal Shift (pixels)
Figure 2: Human perception of small shifts. Im- age pairs with 1- and 2-pixel shift are deemed the same in 80.7% and 56.0% of the responses, resp. we use this data and test the consistency of various metries with the sensitivity of human perception to pixel shifts in Appendix [A. The results of this test provide further evidence that our metric is more consistent in regards to this.
4 EFFECT OF SMALL SHIFTS ON SIMILARITY METRICS
To understand how existing similarity metrics handle small shifts between a pair of images, we benchmarked representative metrics, including off-the-shelf metrics, such as 1.2 and SSIM, and recent deep learning-basee metrics. We derived new dataset from the Berkeley-Adohe Perceptual Patch Similarity Dataset (BAPPS) (Zhang et al.] 2018) in our study. The original BAPPS dataset consists of 36,344 examples, each with reference image 4,, and two distorted images 4 and Iz- These samples cover a wide range of common Table 1: Accuracy (2AFC) & shift-tolerance distortions, including traditional, CNN-based and from real- regl-thtss such as superresolution,
(Frf) of various metrics on the BAPPS valida- tion dataset. 2AFC is computed on the BAPPS dataset resized 64 x 64 while Fr scores are ob- tained from its shifted version of size 64 x 61.
frame interpolation, deblurring, and colorization. Please refer to Zhang et al.] (2018) for more de- tails. For each sample in the BAPPS dataset, we shifted the distorted images horizontally by k pix- els where & E 2,3. To avoid any boundary artifacts from shifting, we cropped each shifted image I, as follows.
Network
2AFC pixel
pixel pixel
1.2
62.91
12.27
23.07
28.83
SSIM (Wnn al. 2004
63.08
13.08
25.50
32.74
CW-SSIM Wang & Smmoncell
2005
60.55
11.33
18.28
23.22
i, = I. O: h, lo (w + 33]
(1)
E-LPIPS (Klllnnen et 2019
8.72
10.67
12.34
GTI-CNN Ma 2
63.74
9.37
12.32
16.25
where w and h are the original image size. In this way, all the images in our test were of size (w-3) x without regard to the amount of shift, which eliminates the effect of image sizes when we test how the amount of shift affects the per- formance of metrics. The reference images were also cropped to the same size as the distorted images but no shifts were applied. cropped all the images in each original sample to the size of (w - 3) x h in order to make the shifted sample and the original sample the same size to avoid the effect of the image size on a similarity metric in our late experiments. No shift was introduced to the original samples. A 3- 3-pixel shift in our setting is equivalent to shifting 1.2% of the pixels for the images of size 256 x 256 pixels.
DISTS Ding al 2000
68.89
5.57
8.20
10.07
PIM-1 (13hardwa
2020
69.45
1.63
3,06
4.39
PIM-5 (Bhardwa) et ali
2020
69.47
2.28
3.56
5.19
LPIPS (Ale) Ziking ct
2018
69.83
6.79
8,90
9.70
IPIPS (Alex) retrained from seratch"
70.04
9.25
9.34
11.55
LPIPS (Alex) ours"!
69.83
3.48
4.75
6.84
Trained on image patches of size 64 using author's 's (f) setup.
addition, we also
When evaluating similarity metric, we applied it to both the original sample in the BAPPS dataset as well as its corresponding shifted once. Specifically, for each sample, we obtained two pairs of similarity scores, (#1, 82) and $1, 82. 82) are the similarity scores between 7 and its reference
4

---

Under review as a conference paper at ICLR 2022
image I,., and Iz and I,, respectively. ($1, are the corresponding pair of similarity scores for the shifted sample. Each pair of scores indicates which of the two distorted images is more similar to the reference image according to the metric Table 2:
Accuracy (2AFC) & shift-tolerance (rry) across various metrics on the BAPPS vali- dation dataset. 2AFC is computed on the BAPPS dataset of original size 256 256 while / is ob- tained from its shifted version of size 256 x 253.
used in the test. We count the number of samples
for which the similarity rank flips when sam-
ple was shifted and compute the rank-flip rate as follows.
1 1(s < <5))) + ($1 <%)) T+ N where Try the rank-flip rate and N is the num- ber of samples in the test set. We use ff to eval- uate how robust a metric is against the small shift between pair of images.
1=1
Network
2AFC
pixel 2 pixel pixel
L2
62.92
3.59
7.55
10.82
SSIM (Wang el al. 2004
61.41
3.16
7.20
13.73
CW-SSIM Wang Nimoncelli
2005
61.48
3.91
6.88
9.47
MS-SSIM (Wang et 2005
62.54
2.22
5.83
10.66
PIEAPP Spars (Pashhian
2018
64.20
3.19
3.81
For all the learned metries involved in this study, we used the trained models shared by their au- thors unless otherwise noted. While the image size of the BAPPS dataset is 256 x 256, some models shared by their authors were trained on 64 64 resized images. Therefore we conducted studies on these two sizes separately to provide fair and informative comparisons.
PIEAPP Dems (Prahhiant el
2018
64.15
2.97
1.37
3.33
PIM-1 (Bhardwaj of 2020
67.45
0.79
1.70
2.52
PIM-5 (Bihrrwwa
2020
67.38
1.01
1.88
2.96
GTI CNN [Ma el al ||211
63.87
3.95
4.91
7.88
DISTS (Ding
(202
68.83
2.85
2.89
4.03
E-LPIPS (Ketiunen of al
2019
68.22
5.84
5.86
5.77
LPIPS (Ales) (Ziingg at
2018
68.59
3.41
3.84
LPIPS (Alex) retrained from scratch"
70.54
2.58
3.59
3.53
LPIPS (Alex) ours"!
70.39
0.66
1.24
1.79
I.PIPS (Alex) retrained fmom scratch"
70.65
2.87
3.92
3.74
LPIPS (Alex) ours" *:
70.48
0.57
1.06
1.50
We report the results in Tables and 2 are obtained by averaging over examples in each distortion category in the BAPPS dataset and then averaging over all the categories. The two-alternative forced choice (2AFC) scores were obtained from the original BAPPS dataset that indicates how metric's prediction correlates with the human opinion (Zhang et al. 2018). The rank-flip rate (rxz) is calculated from the shifted dataset. It shows how robust metric is to the shift between a distorted image and its reference image. As reported in Table the learned metrics match the human perception better than the non-learned ones such as L2, SSIM, and MS-SSIM. However, even these learned metrics are sensitive to small shifts except for the recent metric, PIM (Bhardwaj et al. 2020). Compared to these existing metrics except PIM, our metrics are more consistent with human perception as per 2AFC scores and more robust against small shifts. Overall, our method is comparable to PIM. Our method outperforms PIM on images of size 256 x 256 (Table 2 but does not work as well as it on smaller images (Table []. As discussed in Section PIM is trained from neighboring video frames which often contain small shifts, which makes it robust against the imperceptible shifts. Our work is orthogonal to PIM in that we investigate neural network elements to build robust similarity metric. Therefore, we purposely trained our metrics on the BAPPS dataset without any data augmentation.
All scores Trained on patches
using author's ( our (1) setup.
5 ELEMENTS OF SHIFT-TOLERANT METRICS
Some recent papers reported that training a deep neural network using samples with shifted images through either data augmentation or neighboring video frames can make a learned similarity met- ric robust against small shifts between a pair of im- ages (Ma al. 2018, Bhardwaj et al., 2020). This paper aims to solve this problem from a different perspective; we investigate how one can design a deep neural network that can be resistant to small shifts. We select the LPIPS network architecture as our baseline framework as it correlates with the hu- man visual similarity judgment well (Zhang et al.] 2018). To make this paper self-complete, we briefly describe the LPIPS framework. As illustrated in Figure LPIPS first uses a backbone network. such as AlexNet (Krizhevsky et al. 2012) and VGG (Si-
El Can.
Stentarty
Figure 3: LPIPS framework. The same feature extraction network (AlexNet) is used to extract feature embeddings from Idst and Iref- The difference between these embeddings is calcu- lated at different levels and is combined to- gether as the similarity between last and Tref-
monyan & Zisserman 2015), extract multi-level
5

---

Under review as a conference paper at ICLR 2022
feature embeddings from a distorted image Ldst or its reference image f We denote the resulting feature embeddings as Fdet and Fres, respectively. It then calculates the difference between Fdst and Fres at all the levels and linearly combines the embedding difference at different levels into final similarity / difference score, denoted as di Fast; Fres)- The combination coefficients and the feature extraction network are learned fine-tuned.
Below we discuss how various neural network elements affect a similarity metric and how they can be improved to handle imperceptible shifts between a pair of images. Our focus is to develop feature extraction network to generate feature embeddings from a pair of images that 1) are invariant to imperceptible shifts and 2) lead a metric that correlates well with the human judgements.
Reducing Stride. Striding is widely used in a deep neural network to reduce the input size. For instance, AlexNet has a strided convolution (stride=4) in its first convolutional layer (conv-1) and many max pooling operators with stride=2 in the rest of the network. However, it is commonly known that striding with size >1 leads to the sampling rate falling well below the Nyquist rate, which causes aliasing artifacts. In their experiments with image classification tasks.[Azulay & Weiss) 2019) showed that AlexNet without any subsampling is significantly less sensitive to translations and also maintains its accuracy. Similarly, we also investigate the reduction of the stride size in the convolutional layers in the LPIPS framework to make it more resistant to imperceptible shifts at no expense of its consistency with the human visual similarity perception.
Anti-aliasing. Convolution is the most common operator for deep convolutional neural network. A pure convolutional operator is shift-equivariant instead of being shift-invariant (Nair & Hinton] 2010). Shift equivariance makes a learned sim- ilarity metric sensitive to small shifts as small shifts between two images Lase and fref will be transferred to the shifts between their feature embeddings Fast and Fref- which will in term drastically increase the distance between the fea- ture embeddings d( Fast, Fres) as shown in Fig- (b). Downsampling in a neural network im- proves its shift invariance. Typically, downsam- pling can be achieved by a strided convolutional operator strided pooling operator with stride n (n > 1). However, as discussed earlier in Re- ducing Stride, striding introduces aliasing. While reducing stride size lessens aliasing, it prevents the network from reducing the feature size.
Cone Laper
Comy Layer
Coos Laper
0.03
ShiDld
(b)
0.01
0.00
Figure 4: Feature embedding difference maps at different levels. (a) an input image and its one-pixel shifted version. (b) difference maps between embeddings extracted by the original AlexNet. (c) difference maps between embed- dings extracted by AlexNet augmented with anti- aliased strided convolution and pooling layers.
To keep the benefit of downsampling while reducing stride, [Zhan (2019) invented BlurPool opera- tor. Take max pooling with stride n as an example. Such a max pooling operator can be decomposed into two steps: max pooling with stride 1, followed by a downsampling operator with stride n. To reduce the aliasing artifacts, [Zhang (2019) followed the pre- filtering idea for anti-aliasing and re- placed this max pooling operator with a sequence of three operators: a max pooling with stride 1, Gaussian filter, and downsampling operator with stride n. The last two operators are combined into as a single operator, called BlurPool. Similarly, a convolution operator with stride n can be replaced with its anti-aliased version a convolution operator with stride b and BlurPool with stride na/b. Zhang 2019)) found that replacing the original convolutional and pooling layers in a feature extraction neural network with their BlurPool versions helps generate feature embeddings that make the downstreaming tasks more shift invariant. BlurPool uses a fixed Gaussian filter for blurring and may lose some spatial features that important attributes defining the quality of an image. Zou et al. (2020) developed an adaptive anti-aliasing filter by learning a low-pass filter that is more caaee In this paper, we replace the strided convolution layers or pooling layers in the LPIPS framework with BlurPool or adaptive anti-aliasing filters to make it invariant to imperceptible shifts among images in pair. Figure 4 (c) shows that while anti-aliased convolution and pooling layers they significantly reduce the difference
cannot make the feature network completely between the feature embeddings from pair of shifted images.
Location of Anti-aliasing, In a deep neural network, such as AlexNet used in LPIPS, a convolution layer usually followed by an activation function like ReI.U. According to Zhang (001)) , the activa-
6

---

Under review as a conference paper at ICLR 2022
tion function is inserted between the stride-reduced convolutional layer and BlurPool, illustrated in Figure 5 (a). Vasconcelos et al. (202I) created variants of the anti-aliased strided convolution by placing the anti-aliasing filter different locations, specifically, before after the convolution op- eration. They found that some variants can lead to stronger learned inductive priors. But, will they provide significant improvements in shift tolerance? We build upon their findings and design varia- tions of the anti-aliased strided convolutions. Specifically, we modify AlexNet conv-l as illustrated in Figure 5 and explain the variants below.
Original. As shown in Figure 5 (a), we follow the original design of BlurPool and put it after ReLU (Zhang, (019). For anti- aliissii the stride size of conv-/ is reduced from 4 to 2 and the Blur- Pool layer has a stride of 2 so that the total stride of 4 preserved in this anti-aliased version. We take the output of ReLU as the feature embedding calculate the similarity.
ttal1 Cors, Sm2
The11 Dom, SH0
Tixt Cary, S+2
RULL
RuLU
BlurPool, Sm2
BluPool, 3
Rel
Onggnn
(b) Feeture After Blur
tc Blar Beffree Activation
Figure 5: Alternative positions of BlurPool.
Feature after blur. In the above design, the feature embedding is used before BlurPool. This ef- fectively reduces the anti- aati-aliasing effect on the feature embeddings although the reduced stride size in comvII still offers some level of anti-aliasing. Therefore, investigated variation of the anti- aliased convolution by taking the output of BlurPool as the feature embedding to be used for simi- larity caleulation, as illustrated in Figure 5 (b).
Blur before activation. Vasconcelos et al. (202)) suggested that blurring after the non-linearity, as
done in Figure 5 (a) and (b), prevents high frequency from getting passed on subsequent layers. Following their findings, we adopted their design by placing BlurPool before ReLU to keep the high- feeefrryy information from ReLU, as shown in Figure (c).
Border Handling. Islam et al. (2200) reported that feature embeddings extracted by a convolu- tional neural network encode absolute position information. This has an important implication for a learning-based similarity metric that feature embeddings from a convolutional neural network are and are not shift-invariant. They found that zero padding can relieve this bound- ary problem for computer vision tasks that are sensitive to spatial information. [Kayhan & Gemert (2020) further proposed the concept of full convolution (F-Conv), in which every element of the filter needs to be applied to every pixel in the input image. They implemented F-Conv as regular convolutional operator with zero padding of 2k where 2k + is the filter kernel size, as illustrated in Figure (Appendix A..)). Note. F-Conv will make the output of un-strided convolution operator (Aepenndi
2k larger than the input. They reported that F-Conv is least sensitive to the absolute position of the objects for image classification tasks. Inspired by these works, replace the regular convolution operators with F-Cony in the LPIPS framework and increase the padding size in BlurPool operators to achieve better shift-invariance.
Pooling. Max pooling is well known for be- ing more shift invariant than average pooling. We investigate whether its anti-aliased version, MaxBlurPool (described earlier in Anti-aliasing) is also more shift invariant than ArgBlurPool, the anti-aliased version of average pooling when used in the LPIPS framework. Average pool- ing in its original form already supports anti- aliasing. We follow Zhang (2019) and imple- ment Blrrluooo with a stride of n as Gaussian filtering followed by downsampling with factor of n.
Conyy 303
Gore 3d3
Concaci
ItLL
ReLl
BaxPPol 9x2
Oarx
Corw +
BurRoot 8-2
BluPool
BluPPol, 8-2
RALU
Rec.
Relli
(x)
[b]
Figure 6: Anti-aliased skipped connection. (a) VGG-like network with AvgBlurPool, (b) with skip connection, and (c) with anti-aliased skip.
Strided-skip Connections. Skip connection is widely used in deep neural network speedup training and obtain a high quality neural network model. We investigate whether skip connection helps improve shift invariance of a learned similarity metric. As discussed in Vasconcelos et al. 202T). strided skip connection introduces aliasing for the same reason discussed earlier in Anti- aliasing. We therefore explore anti-aliased strided skip connections, as shown in Figure
7

---

Under review as a conference paper at ICLR 2022
6 EXPERIMENTS
We built upon the LPIPS framework and incorporated the elements discussed in Section 5 to inves- tigate how these elements help develop a similarity metric that is consistent with the human visual similarity judgement and is robust against imperceptible shifts. We first compare our metrics to both off-the-shelf metrics, such as SSIM and MS-SSIM, and the recent learned similarity metries. We then conduct ablation studies to evaluate how these elements work.
Comparisons to Existing Metrics. In Sec-
Table 3: Experiments the CLIC dataset.
tion [ we derived a shifted dataset from the BAPPS dataset and compared our metrics
Number of rank dips
Network
Accuracy(58)
pix
2
pix
to representative existing metrics (Bhardwaj
L.2
58.16
833
2102
2214
al.
2020, Ding et al. 2020, Ma et al.
2018
SSIM Wang ct.
2004
60.00
349
931
1109
Kettunen et al. 2019; Prashnani et al.
2018
PIEAFT Prashnami
91
158
Wang et al. 2004, 2003; Zhhng et al. 2018).
E-LPIPS Keltunch el
2019
74.44
212
251
317
DISTS Ding el
2020
75.63
28
36
50
In our experiments, we adopt the 2AFC
metric
PIM-1 (Bhardwaj 2020
73.79
13
22
33
to evaluate how consistent a metric is with hu-
(TPP)( (Zhang at 2018
73.68
90
108
121
man judgment, and the rank- flippinginnng rate, frf=
(PPPS(Ales) retruined from scratch" 1
76.53
59
51
62
LPIPS(Alex) o
76.97
21
to evaluate how robust is against small shifts.
As shown in Table D and 2 our metrics are
Trained on image patches of size 64 using author's 's (() setup.
both more consistent with human visual similarity judgment and more robust against imperceptible
shifts than most of them, except recent metric PIM (Bhardwaj al. 2020), which our method is comparable. PIM achieves shift robustness by training on neighboring video frames that often have small shifts. We work on an orthogonal solution by investigating neural network elements to make the learned metric robust and therefore only train our metrics on the examples without any shift through data augmentation. We further evaluated the metrics on the perceptual validation dataset from the Challenge on Learned Image Compression (CLIC) 2021 The results in Table [ are con- sistent with previous results, i.e., our method outperforms all the other methods in terms of accuracy and is more shift-robust than others except PIM, which is similarly robust to ours.
Ablation Studies. We now examine how individual network elements affect our metries. these studies, we trained all our metrics using the original BAPPS training set on their original size of 256 x 256. We purposely did not train on the shifted version to focus on neural network element designs. To train and $2 are the predicted similarity scores of two images I1 and Ig to their corresponding reference image, and h is the human score. We trained (2018) except we used a lower dropout rate of 0.01. We tested all our metrics on the shifted testing dataset obtain the rank-ffipping rate. To obtain the 2AFC scores, we ran metries the full- size images (with no shift) of the original BAPPS dataset so that we could verify whether our metrics sacrifice consistency with human visual similarity judgment be robust against imperceptible shifts.
metrics, we used the loss function: M S (s, ft), where s = 81/(s1 + 52), si
metrics using the same settings as ZZhang et al.
We first examine elements discussed in Section 5 in- dividually. We use AlexNet as the backbone fea- ture extraction network with the LPIPS framework as it provides the best result among other backbone networks (Zhang et al. 2018). As reported in Ta- ble 4 anti-aliasing via BlurPool can greatly improve LPIPS's robustness against imperceptible shifts. Re- ducing stride size in its strided convolutional layer (comv-)) also helps making it significantly more ro- bust at little expense of the 2AFC score. Combin- ing PlurPool with reducing stride size makes the net- work even more robust against imperceptible shifts and more consistent with human judgment based on the 2AFC score. Then a larger reflection padding size also helps as it reduces the position information encoded in the feature embeddings from the image 7Zu 2020)
Table 4: Effect of (1) anti-aliasing (AA) via BlurPool, (2) F-Conv, (3) reduced stride, & (4) adaptive-AA on learned metrics.
AA (BlurPool)
Reflection-PP Pad
F-Conv
Stride 2AFC
TvJ
2
in com-1
pixel pixel 3 pixel
70.65
2.87
3.92
3.74
70.53
1.85
2.22
2.58
70.67
1,46
1.82
2.25
7
70.57
2.78
3.92
2
70.52
1.77
2.15
2.48
2
70.54
1.84
2.28
2.34
70.42
0.66
1.13
1.83
70.44
0.63
1.14
1.68
2
70.57
2.63
3.36
3.16
2
70.63
2.80
3.57
3.39
70.52
2.95
4.13
3.93
boundaries, as discussed in Section [ However, F-Conv, also designed reduce the boundary is-
sue, does not help. While the learned BlurPool (Zou et al.) (2020) also helps, it is not as effective as the original version for our task of making a robust similarity metric.
8

---

Under review as a conference paper at ICLR 2022
We test on different backbone feature networks, in-
Table Anti-aliasing via BlurPool can signif- icantly improve shift-tolerance and often im- prove 2AFC scores consistently for different
cluding VGG-16 (Simonyan & Zisserman, 2015),
ResNet- 18 (He 2016), and SqueezeNet (lan-
dola et a 2016). While reducing the stride size is effective, not all networks have strided convolu- tion layer. Hence, we focus on BlurPool applied to pooling layers. As shown in Table [5] BlurPool sig- nificantly improves the robustness of other backbone networks as well. What is interesting is the effect of the padding size within these backbone networks.
backbone feature extraction networks.
AA (BlurPool)
Network
Reffection-Pad
2AFC
Yrf
pixel 2 pisel
pixel
VGG-16
70.03
3.01
3.76
3.44
70.05
0.66
1,08
1.44
70.07
0.66
1.12
1.82
While a larger padding size improves 2AFC scores, it does not make shift-invarian and ResNet-18. 18. For them, the anti-aliased convolu- tion layers have stride size of I, which leads to mi- nor boundary issues. We conjecture that this makes larger padding size unnecessary.
ResNet-18
69.86
2.67
3.35
3.77
better for VGG-16
69.95
0.82
2.19
70.14
1.07
1.81
2.38
Squeeze
69.61
7.41
7.58
10.35
69.24
2.03
3.06
3.93
69.44
2.10
3.42
Table 6: Effect of BlurPool locations within an anti- aliased strided convolution as illustrated in Figure
We also examine the effect of the location of BlurPool within AlexNet. As reported
in Table 6. the original version (Figure [ (a)) works best when the stride size is 2 in conv-l. With a smaller stride size, it does not work as well as Blur Before Ac- tivation (Figure (c)). This is in part con- sistent with what was found by Vasconce- los et al. (202T). In the original design, BlurPool is placed after the activation layer for anti-aliasing at the expense of the reduction of the figh-frequency information from the activation layer. With the need for anti-aliasing due to larger stride size, this trade-off works out. However, when stride size is 1, the need for anti-aliasing is reduced; therefore, it is more helpful place BlurPool before the activation layer avoid the loss of high-freyeency information. Thus, Blur Before Activation works better when the stride size is 1.
Anti-Alias Stride
BlurPool
2AFC
Fry
(BlurPool) in Come-J
Location
pixel pixel pixel
Original
70.67
1.46
1.82
2.25
FeatureAlterBluur
70.55
1.73
1.84
2.49
BlurBeforr Activvattoon
70.50
2.06
2,02
2.74
Original
70.42
0.66
1.13
1.83
FferrrrA
70.52
0.69
1.11
1.60
BlurBeforcActivation
70.48
0.57
1.06
1.50
Table 7 shows that PoxllurPool has better shift tolerance but lower 2AFC scores (accuracy) than AvgBlurBool. Moreover, using anti-aliased strided-skip connections leads to higher accuracy with a negligible drop in shift tolerance.
Table 7: Effects of different pooling layers and anti-aliased strided skip connections.
Prf
Network elements
ZAFC
pixel 2 pixel pixel
BlurPlo
69.88
1.24
1.86
2.18
Summary. Among the network elements we in- vestigated, anti-aliased strided convolution, anti- aliased pooling, and reduction of stride size are most effective develop a perceptual similarity metric that is robust against imperceptible shifts. These findings are consistent across variety of
MaxBlurPool
69.58
0.95
1.54
2.12
AvgBlurPool & Strided Skip
69.82
1.38
2.06
2.61
ArgBlurPool & Anti aliused Skip
70.05
1.26
1.89
2.32
MaxBlurPool Anti- Skip
69.86
1.07
1.66
2.35
backbone network architectures. A larger padding size helps reduce the position information due to the boundary issues encoded in the feature embeddings. Anti-aliased skip connection can help im- prove accuracy but with little effect on shift invariance. The position of BlurPool matters. It should be placed before the activation layer if its precedent convolution uses small stride size.
7 CONCLUSION
This paper reported our investigation on how to design deep neural network as a learned percep- tual image similarity metric that is both consistent with the human visual similarity judgement and robust against the imperceptible shift among a pair of images. We discussed various neural network elements, such as anti-aliased strided convolution, anti-aliased pooling, the placement of BlurPool, stride size, and skip connection and studied their effect on similarity metric. We found that using anti-aliasing strided convolutions and pooling operators and reducing stride size are very helpful to make learned similarity metric inft--ivariant. Our experiments show that by integrating these elements into a neural network. we are able to develop a learned metric that is more robust against imperceptible shifts and more consistent with the human visual similarity judgement.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Jun Yuan, and Orion the pad CNNs can develop blind spots. In International Conference on Learning Representa- tions, 2021.
Mind
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, volume 80, pp. 274 283, 2018.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184):1-25, 2019.
Sangnie Bhardwaj. lan Fischer, Johannes Ballé, and Troy Chinen. An unsupervised information- theoretic perceptual quality metric. In Advances in Neural Information Processing Systems, pp.
13-24, 2020.
CLIC. Workshop and challenge on learned image compression, 2021. URL http://wwr //www. compression.cc// cc/2021/
Steffen Czolbe, Oswin Krause, Ingemar Cox, and Christian lgel. A loss function for generative neu- ral networks based on watson's 's perceptual model. In Advances in Neural Information Processing Systems, pp. 2051-2061, 2020.
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2020.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, pp. 658-666, 2016.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. International Journal of Computer Vision, 88(2):303-338 2010.
Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. Pipal: A large- scale image quality assessment dataset for perceptual image restoration. In European Conference 2020.
on Computer Vision, volume 12356, pp.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Olivier Hénaff and Eero P Simoncelli. Geodesics of learned representations. In International Conference on Learning Representations, 2016.
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size. arXin/1602.07360. 2016.
Md Amirul Islam, Sen Jia, and Neil D. B. Bruce. How much position information do convolutional neural networks encode? In International Conference on Learning Representations, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, volume 9906, pp. 694-711, 2016.
Osman Semih Kayhan and Jan C. van Gemert. On translation invariance cnns: Convolutional layers can exploit absolute spatial location. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14262-14273, 2020.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-LPIPS: Robust perceptual image simi- larity via random transformation ensembles. arXivv/900600373, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2013.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1106-110 1006-1114, 1114, 2012.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. arXiv/1609.0480) 2016.
Jake Lee, Junfeng Yang, and Zhangyang Wang. What does shift invariance look like? visual- ization study. In European Conference on Computer Vision Workshops, pp. 196-210, 2020.
Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In Tenth International Conference on Quality of Multimedia Experience (QoMEX), pp- 1-3, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Kede Ma, Zhengfang Duanmu, and Zhou Wang. Geometric transformation invariant image quality assessment using convolutional neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6732-6736, 2018.
Douglas L Medin, Robert L Goldstone, and Dedre Gentner. Respects for similarity. Psychological review, 100(2):254, 1993.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, pp. 807-814, 2010.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convo- lution. IEEE International Conference on Computer Vision, pp. 261-270, 2017.
Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. Image database tid2013: Peculiarities, results and perspectives. Signal Processing Image communication, 30:57-77, 2015.
Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen. Pieapp: Perceptual image-error assessment through pairwise preference. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2018,
2018.
Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch. EnhanceNet: Single image super- resolution through automated texture synthesis. arXiv/1612.07919, 2016.
Hamid R Sheikh.
Image and video quality 2003.
assessment
research at live.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Amos Tversky. Features of similarity, Psychological review, 84(4):327, 1977.
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Rob Romijnders, Nicolas Le Roux, and Ross Goroshin. Impact of aliasing on generalization in deep convolutional networks. In IEEE International Conference Computer Vision, 2021.
Zhou Wang and Eero Simoncelli. Translation insensitive image similarity complex wavelet do- main. In IEEE International Conference Acoustics, Speech, and Signal Processing. volume 2, pp. ii- 573, 2005.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In Asilomar Conference on Signals, Systems & Computers, volume 2, pp. 1398-1402. IEEE, 2003.
11

---

Under review as a conference paper at ICLR 2022
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13(4))600- 612. 2004.
Andrew B Watson. DCT quantization matrices visually optimized for individual images. In Human viston, visual processing, and digital display IV, volume 1913, pp. 202-216. International Society for Optics and Photonics, 1993.
Chaowei Xiao, Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially trans- formed adversarial examples. In International Conference on Learning Representations, 2018.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8): 2011.
Richard Zhang. Making convolutional networks shift- shift-invariant again. In International Conference on Machine Learning, pp. 7324-7334, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipu- lation the natural image manifold. In European Conference Computer Vision, volume 9909, pp. 597-613, 2016.
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In British Machine Vision Conference, 2020.
A APPENDIX
A.1
USER STUDY
We recruited 32 participants for our study. These participants have a wide range of professional background, including computer science, business, medicine, arts and education. Most of them are between 20 to 35 years old.
To ensure the quality of this user study, removed the responses from two participants who failed to pass a validation test. Specifically. a participant identified a pair of images with 0-pixel shift pair of images with 9-pixel shift the same for more than two-thirds of the time in the study, we did not include the response from that participant. In total, 30 participants passed our validation test. We obtained responses to 1500 trials in total, with 150 responses for each of the n-pixel shifts.
different
It is important disclose that before the study, we informed our participants that there might be shifts between some pairs of images to raise their alertness to the potential difference created by the shift. While this might bias participants, we found it helpful obtain more informative under- standing of the human perception of small shifts; otherwise, participants tended to overly overlook the difference between a pair of images.
In total, we have 500 samples with pair of images where one of the images is shifted. Each user was presented with 5 samples for each 0-9 pix-shift randomly. We made sure that no user saw the same sample twice in our study. The number of responses to each sample varied and the mean number of responses per sample is and the standard deviation is 1.33.
Why we chose 50 images? We generated 500 pairs of images, with 0-9 pixel shifts from them. To maintain the quality of our study and avoid boring the users, we only presented 50 samples to each user. Interestingly, humans managed attribute this partially to the fact that the users were informed that there might or might not be a shift between pair of images. This indeed introduced biases into the study such that their sensitivity to the shifts is likely increased. We chose to do so as, in our pilot study, we found that users were very confused when we asked them if pair of images looked the same not. Many of them thought
detect the shift for 2 pixel shift in 50% of cases. We
12

---

Under review as a conference paper at ICLR 2022
if we were asking them to compare high-level features such objects in the two images or if there were some artifacts in one of the pair of images.
Furthermore, analyzed the variability of the responses. In analysis, if user noticed the shift between pair of images, we label the response as and otherwise. We then calculate the standard deviation of the user responses for each image. The average (avg.) of the standard deviation (std.) in the responses per sample is 0.2 with a standard deviation of 0.23.
Table 8: Variability in user responses.
Pixel-shif
Avg. of std. per sample
Std. std. per sample
0.09
0.19
0.23
0.24
0.23
0.3
0.24
0.23
0.24
6
0.21
0.24
8
0.18
0.23
0.13
Finally, we compute the average standard deviation for the whole group of samples with the same amount of shift and report the results in Table g With no or only l-pixel shift, users were con- sistently sure that the images in each pair pixels), users consistently indicated that the images were shifted. In contrast, we see more variability user responses when the shift is 2 to 5 pixels. Hence, for images with 2 to 5-pixel shift, users were doubtful whether images were shifted or not, and their responses had a high variation.
the same. Similarly, with very large shift (6
A.1.1 JUST NOTICFABLE DIFFERENCES (JND)
Table 9: Consistency of perceptual similarity metries with the sensitivity of human perception to pixel shifts.
Metric
JND MAP%
Wang et
0.722
LPIPS (Alek) (hhnng et dl. 2018
0.757
IPPIS (Alex) retrained from scratch"
0.740
T.PIPS (Alex) ours
0.771
LPIPS (VGG) (Zhang 2018
0.770
I.PIS (VGG) retrained from scratch
0.769
LPIPS (VGG) ours""
0.775
DISTS DDing al. 2020
0.766
PTM-1 (Whardway
4020
0.773
Trained on image patches of size 64 using author's 's (f) setup.
We conducted the following test to study how consistent our shift-tolerant perceptual similarity metric is with the human perception results reported in Figure 2 In our study reported in Figure 2. we had asked our participants if the two images, which may be shifted by a few pixels, were the same not. Using these responses,
perform just noticeable difference test. We make use of only those samples which have at least human responses. There were 301 such samples, and the mean number of samples per pixel-shift (0 9) is 30.1 with standard deviation of 1.6 (maximum 33 and minimum 28). Following Zhang et al. 2018), we rank the pairs by a perceptual similarity metric and compute the area under the precision/recall curve (mAP) (as used by [Everingham et al. (210)) and Zhang et al. (2018)). 8)). The results in Table 9 show that our shift-tolerant LPIPS metrics follow the sensitivity of human perception pixel shifts more accurately than their vanilla versions. The accuracy of PIM-1 and DISTS is comparable to ours.
13

---

Under review as a conference paper at ICLR 2022
A.2 FULL CONVOLUTION
2k
2k +1
padding size
0
0
kernel size
0
0
0
0 0
F-Conv
0
input
Illustration of Full Convolution (F-Conv), where every value of the filter needs to be applied to each value of input image. Hence, the input needs to be padded first with padding size 2k for a filter with size 2k
Figure 7:
Kayhan & Gemert 2020).
A.3 ADDITIONAL ABLATION STUDIES
We provide few additional ablation studies to help examine the network elements discussed in our main paper.
A.3.1 PADDING SIZE
Table 10: Effects of increasing reflection-pal size. using AlexNet as its backbone network.
This test is conducted for our learned metric
AA (BlurPool)
ZAFC
Tw/
w/ Refloction-Pad
pixel
pixel
pixel
None
70.53
2.11
2.51
2.58
70.53
1.85
2.22
2.58
70.67
1.46
182
2.25
To further examine the effect of the padding size, we added a new variation where no padding is used all the BlurBool layers. As shown in Table [O, padding in the BlurPool layers makes our learned metric more robust against small shifts.
A.3.2 BLUR KERNEL SIZE
Table 11: Effects of increasing the blur kernel size.
AA (BlurPoof) Reflection-Pad Pad
Stride
Blue Kernel Size
2AFC
Trf
Com-11
Coms-
pixel
2 pixel
pixel
1.85
70.60
1,57
1.67
2.11
70.67
1.46
[.82
2.25
5
70.54
1.61
1.55
2.16
70.42
0.66
1.13
1.83
70.57
0.73
1.16
1.71
BlurPool is used for anti-aliasing for strided convolution and strided pooling operators (Zhang 2019). Anti-aliasing is achieved by employing a Gaussian filter to remove the high-frequene, in intermediate result from previous step of non-striding pooling or convolution operators. A key parameter is the size of Gaussian filter kernel. In our paper, we used fixed kernel size of 3. We add new test to examine the effect of the kernel size in conv-/ on our learned metric. As shown in the first group of Table II large kernel helps anti-aliasing in general and thus makes our metric more shift-tolerant. However, when combined with a reduced stride size (=1), a large kernel can also be counter productive and make metric less shift-invarian Table II This because when the stride size 1, aliasing is not a big issue. Then a large kernel size removes useful high-frequency information with little gain from anti-aliasing.
shown in the bottom group in
14

---

Under review as a conference paper at ICLR 2022
A.3.3 PADDING TYPE
Table 12: Comparing the type of padding, Zero vs. Reflection, in the BlurPool layers.
AA (BlurPool)
Frf
Refi-Pad Size
Zero-Pad Size 2
F-Conv
2AFC
pixel shift
pixel shift
pixel shifl
70.67
1.46
1.82
2.25
70.55
2.05
2.28
2,70
70.52
1.77
2.15
2.48
In all our experiments, set the default padding type as reflection padding. We now examine if zero padding performs better or not by replacing reflection padding with zero padding the BlurPool layers. As shown in Table [2] we found that reflection padding general outperforms In a recent study. [Alsallakh et al. (2021) found reflection pad to be more effective in reducing line artifacts in feature maps. We believe it reduces aliasing that improves shift tolerance.
padding.
A.3.4 ANTI-ALIASING OPERATOR TYPE
Table 13: Comparing BlurPool against le pooling.
Anti-alias
Blur Kemel
Trf
2AFC
BlurPool
Pooling
all layors
pix
pix
3
70.53
1.85
70.69
2.92
3.90
3.60
70.51
1.68
1.59
2.17
70.55
1.87
1.89
2.38
We use BlurPool for anti-aliasing in our experiments. Here we compare effect anti-aliasing via BlurPool versus l2 Pooling. As observed in Table 13 when the blur kernel size is 3, the shift tolerance of the network having BlurPool is substantially better than the network with L2 Pooling layers, but at the expense of accuracy. This maybe due to weaker blurring by Pooling. As the blur kernel size is increased to 5, the strength of the blurring effect increases in both, and we observe the performance gap reducing, in terms of both accuracy and shift tolerance.
A.4 RESULTS ON OTHER IQA DATASETS
Table 14: Performance comparison on the TID-2013 (Ponomarenko et al.
2015) dataset.
Metric
SRCC
KRCC
PLCC
LPIPS (VGG) retrained from scratch"
0.86
0.66
0.87
LPIPS (VGG)
0.86
0.67
DISTS (Ding et al
2020
0.83
0.64
0.85
Kadid-1U 1Ük.
Table 15: Performance comparison on the LIVE dataset (Shiikh 2003).
Metric
SRCC
KRCC
PLOCC
(VGO) retrained scratch'
0.0.79
LPIPS (VGG) ours"
0.95
0.80
0.94
DISTS (Ding et al. 2020
0.95
0.81
0.91
Trained on Kadid. lOk.
For an apples-to-apples comparison with DISTS (Ding et al. 2020), we train the LPIPS metric on the Kadid-10k dataset (Lin et al.] 2019). We use the pre-trained model for DISTS that is trained on Kadid- 10k and uses a VGG backbone network. As shown Tables [4 and [5] our shift-tolerant LPIPS metric improves the baseline I.IPS metric and outperforms DISTS on both datasets.
15

---

Under review as a conference paper at ICLR 2022
Table 16: Performance comparison the shifted LIVE dataset (Sheikh 2003
SROC
Metric
Original
No shifl
1 pixel shill
2-pixel shill
LPIPS (VGG) retrained from scratch"
0.94
0.94
0.90
0.91
LPIPS (VGG) ours
0.95
0.95
0.94
DISTS
Ding
2020
0.95
0.95
0.95
0.95
Trained on Kadid TOK.
We conducted a test using the SRCC metric on the shifted version of the LIVE dataset. As reported in Table T our upgraded LPIPS metric significantly improves the robustness against small shifts over the baseline LPIPS metric. Our results are comparable to DISTS.
16


Reviews:
[{"review_id": "a0lbMHAtqmW", "reviewer": "Reviewer_2NrG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe elements of shift-tolerant metrics in the context of convolution neural networks are well analyzed and validated.\n\nWeaknesses:\n\n1. The reviewer appreciates the authors' effort in summarizing recent work on shift-tolerate IQA (and CNN in general). However, the reviewer believes that it is more important to give oldies but goldies enough credit by citing and describing their work in the current manuscript. Dated back to the era of wavelets, there were researchers, trying to come up with shift-invariant wavelets. In the context of IQA, researchers were thinking about shift-invariant IQA as early as 2005. The reviewer is reluctant to directly point out those references but highly encourages the authors to dig them out.\n\n2. Eq. (2) is not a good measure of shift-invariance in the context of IQA. It is more suitable to think of mild shift as a form of non-structural distortion. When applied to images with severe structural distortions (e.g. JPEG compression), the primary goal of a perceptual metric is to capture such visible structural distortions; and it does not matter whether it is robust to mild shift. Therefore, a more reasonable experimental setup is given in GTI-IQA and DISTS papers: assigning shifted images the same MOSs and computing SRCC/KRCC/PLCC on the whole dataset.\n\n3. Shift-invariance is a small piece story in IQA; the authors are encouraged to show the proposed metric is able to deliver SOTA quality prediction performance on standard IQA datasets such as LIVE, CSIQ, TID-2013, and KADID-10K. And if possible try more image restoration datasets like CLIC.\n\n4. 3 pixels are not sufficient to test the robustness of IQA models to mild shift. A more appropriate hyperparameter setting should be resolution-dependent, e.g., 3% to 5% of pixels are shifted.\n\n5. The authors may contrast their algorithms more to DISTS than LPIPS, and may even build their models on top of DISTS for quality prediction, texture performance, and perceptual optimization reasons. The key to the success of DISTS in fighting mild geometric transformations is global feature aggregation, which is understated by the authors.\n\n6. How is the performance of the proposed method to other mild geometric transformations, such as rotation and dilation?\n\n7. Is the feature mapping of the proposed metric injective (or bijective)?", "summary_of_the_paper": "A shift-tolerant perceptual similarity metric is proposed based on LPIPS.", "summary_of_the_review": "See above for the detailed comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XNqA7Po44Pa", "reviewer": "Reviewer_GQvy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The paper tackles an important and interesting subject of developing shift-invariant metrics. The authors go one step further and support the claim of the importance of such metrics by conducting an experiment with humans (Section 3 and Figure 1)\n2) The authors suggest interesting metrics to measure shift-invariance, which counts the number of samples for which the similarity rank flips when a sample was shifted and computes the rank-flip rate  - eq. 2 \n3) The authors show that their suggested approach improves the proposed metrics for LPIPS arhcietcture (FIgure 3) and achieves comparable results with SOTA PIM similarity network \n4) It is interesting to see influence of the suggested architecture alterntaions on different backbones presented in Table 5\n\nCons:\n1) The experiment with human perception is not clearly explained. First, it is hard to understand the number of different pairs that was eventually evaluated by humans or if there were several humans that were presented with the same pair  -- it would be very helpful to summarize those in a table. Also it is very suprising that humans managed were sensetive to 2 pixel shift in ~ 50% of cases. This surprising fact is only explained in the supplementary material. It might be helpful to revise the paper to allow explanation on specific numbers that were chosen (e.g., why only 50 images) and the included biases (e.g. explaining humans that there are shifts they should look for).  \n2) The novelty of the paper is limited by minor alternations to the exiting architecture. Those alternations, e.g. anti-aliasing, are not new, as stated by the authors. The authors point our several times \"PIM achieves shift robustness by training on neighboring video frames that often have small shifts. We work on an orthogonal solution by investigating neural network elements to make the learned metric robust and therefore only train our metrics on the examples without any shift through data augmentation.\" , this requires a more elaborative discussion on why we would like to choose this second route (which is also not optimal for a single pixel compared to PIM, as evedient from Tabls 1 and 3) There is no discussion of the effect of these additions on the runtime and comparison of the runtime with PIM.  \nTable 2 shows that when evaluating bigger images the suggest approach is better than PIM, but the authors do not offer explanation why their approach works better\n4) Many experiments yield somewhat not intuitive results, like the padding influence on performance in Table 5. The explanations offered are not very clear. Since the paper does not support the suggested approaches theoretically, it might benefit from more intuitive explanations of the experimental results. \n ", "summary_of_the_paper": "The paper proposes a few modifications to the existing archiectures for perceptual image similarity that would be robust to the tiny shifts in the image. Specifically, the authors conduct an experiemnt that shows that humans mostly are not sensetive to small shifts of 1 or 2 pixels (Table 1), but most state of the art perceptual image similarity networks are sensetive to such shifts (Table 1 and Table 2). The authors discuss what parts of the network architecture might yield such sensetivity and offer a few modifications (Section 5). In Section 6, the authors perform extensive experiments to check the effectiveness of their suggestions. Finally the authors conclude that using anti-aliasing strided convolutions and pooling operators and reducing stride size are helpful to make a learned similarity metric shift-invariant. Their experiments show that by integrating these elements into a neural network, the learned metric is more robust against imperceptible shifts and more consistent with the human visual similarity judgment  (Table 1,2,3) . ", "summary_of_the_review": "I think the paper has merit, but it will benefit from revision, which empahsizes more motivation on why to choose a suggested approach as opposed to the other leading approach PIM and also why certain empirical results make sense. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P91fcnMpJ8u", "reviewer": "Reviewer_LHuY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n* Well-written and easy to follow/understand\n* Improving architecture to achieve robustness is a better approach than data augmentation which relies on a longer training (energy consumption)\n* Fine analysis of neural network elements (their analysis is useful beyond PSMs)\n* Human data\n\n### Weaknesses\n\n* No comparison of their psychometric data and proposed PSM\n* (minor) figure and table fonts should be similar to main text font\n\n### Detailed comments\n\nI understand that this is not the authors' main goal, however, their psychometric data could be exploited more to enforce the consistency of PSM with human judgement. The first question is: Do humans all notice the shift in the same images or is there a huge variability ? \nThen, I am curious about how PSM could reproduce (if consistent across humans) the sensitivity to shift reported in Figure 2. \n", "summary_of_the_paper": "The authors propose to make perceptual similarity metrics (PSM) invariant to small-shifts (few pixels translation) and still consistent with human judgement. To this end, they use an approach based on network architectures to evaluate which elements (anti-aliasing, pooling, striding, padding, skip connection) can achieve shift-invariance.\n\nThe paper have multiple contributions:\n1. A study on the human perception of small shift estimating their sensitivity\n2. A study of the sensitivity of PSMs to small shift\n3. A systematic study of neural network architecture elements in relation to shift-invariance\n4. A updated/shift-invariant version of the LPIPS metric\n5. An ablation study to evaluate which elements contribute or to shift-invariance\n", "summary_of_the_review": "This a well-conducted and strong paper which achieve is goal and give interesting insight about neural network architecture elements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a perceptual similarity on top on the commonly used perceptual loss in the literature (LPIPS). The authors draw experiments highlighting that human perceptual similarity is invariant to small shifts, whereas standard metrics are not. The paper studies several factors (anti-aliasing, pooling, striding, padding, skip connection) in order to propose a measure on top of LPIPS achieving shift-invariance. 

This paper initially received mixed reviews. RLHuY was positive about the submission, pointing out the relevance of the real human data and the studied factors for measuring the impact on shift invariance. RGQvy was slightly positive, but also raised several concerns on justification of the claimed properties, human perception experiments, and positioning with respect to data augmentation (PIM). RLHuY, an expert on the field, recommended clear rejection, pointing out missing references (including DISTS), the limited scope of the paper (shift invariance and tiny shifts). After rebuttal, RLHuY and RLHuY stuck to their positions ; RGQvy were inclined to borderline reject  because of unconvincing answers on comparison to data augmentation techniques. 

The AC's own readings confirmed the concerns raised by RGQvy and RLHuY, and points the following limitations: 

- The submission includes limited contribution and expected results: the studied modifications on neural networks' architecture, although meaningful, directly follow ideas borrowed from the literature. They are not supported by stronger theoretical analysis, and several insights related to accuracy or robustness remain unclear. 
- Experimental results are contrasted, e.g. compared to data-augmentation: although these approaches are more demanding at train time, they do not induce any overhead at test time - in contrast to the proposed approach.

Therefore, the AC recommends rejection.
Decision: Reject