Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: A shift-tolerant perceptual similarity metric is proposed based on LPIPS.
main_review: Strengths:

The elements of shift-tolerant metrics in the context of convolution neural networks are well analyzed and validated.

Weaknesses:

1. The reviewer appreciates the authors' effort in summarizing recent work on shift-tolerate IQA (and CNN in general). However, the reviewer believes that it is more important to give oldies but goldies enough credit by citing and describing their work in the current manuscript. Dated back to the era of wavelets, there were researchers, trying to come up with shift-invariant wavelets. In the context of IQA, researchers were thinking about shift-invariant IQA as early as 2005. The reviewer is reluctant to directly point out those references but highly encourages the authors to dig them out.

2. Eq. (2) is not a good measure of shift-invariance in the context of IQA. It is more suitable to think of mild shift as a form of non-structural distortion. When applied to images with severe structural distortions (e.g. JPEG compression), the primary goal of a perceptual metric is to capture such visible structural distortions; and it does not matter whether it is robust to mild shift. Therefore, a more reasonable experimental setup is given in GTI-IQA and DISTS papers: assigning shifted images the same MOSs and computing SRCC/KRCC/PLCC on the whole dataset.

3. Shift-invariance is a small piece story in IQA; the authors are encouraged to show the proposed metric is able to deliver SOTA quality prediction performance on standard IQA datasets such as LIVE, CSIQ, TID-2013, and KADID-10K. And if possible try more image restoration datasets like CLIC.

4. 3 pixels are not sufficient to test the robustness of IQA models to mild shift. A more appropriate hyperparameter setting should be resolution-dependent, e.g., 3% to 5% of pixels are shifted.

5. The authors may contrast their algorithms more to DISTS than LPIPS, and may even build their models on top of DISTS for quality prediction, texture performance, and perceptual optimization reasons. The key to the success of DISTS in fighting mild geometric transformations is global feature aggregation, which is understated by the authors.

6. How is the performance of the proposed method to other mild geometric transformations, such as rotation and dilation?

7. Is the feature mapping of the proposed metric injective (or bijective)?
summary_of_the_review: See above for the detailed comments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to make perceptual similarity metrics (PSM) invariant to small-shifts (few pixels translation) and still consistent with human judgement. To this end, they use an approach based on network architectures to evaluate which elements (anti-aliasing, pooling, striding, padding, skip connection) can achieve shift-invariance.

The paper have multiple contributions:
1. A study on the human perception of small shift estimating their sensitivity
2. A study of the sensitivity of PSMs to small shift
3. A systematic study of neural network architecture elements in relation to shift-invariance
4. A updated/shift-invariant version of the LPIPS metric
5. An ablation study to evaluate which elements contribute or to shift-invariance

main_review: ### Strengths

* Well-written and easy to follow/understand
* Improving architecture to achieve robustness is a better approach than data augmentation which relies on a longer training (energy consumption)
* Fine analysis of neural network elements (their analysis is useful beyond PSMs)
* Human data

### Weaknesses

* No comparison of their psychometric data and proposed PSM
* (minor) figure and table fonts should be similar to main text font

### Detailed comments

I understand that this is not the authors' main goal, however, their psychometric data could be exploited more to enforce the consistency of PSM with human judgement. The first question is: Do humans all notice the shift in the same images or is there a huge variability ? 
Then, I am curious about how PSM could reproduce (if consistent across humans) the sensitivity to shift reported in Figure 2. 

summary_of_the_review: This a well-conducted and strong paper which achieve is goal and give interesting insight about neural network architecture elements.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a few modifications to the existing archiectures for perceptual image similarity that would be robust to the tiny shifts in the image. Specifically, the authors conduct an experiemnt that shows that humans mostly are not sensetive to small shifts of 1 or 2 pixels (Table 1), but most state of the art perceptual image similarity networks are sensetive to such shifts (Table 1 and Table 2). The authors discuss what parts of the network architecture might yield such sensetivity and offer a few modifications (Section 5). In Section 6, the authors perform extensive experiments to check the effectiveness of their suggestions. Finally the authors conclude that using anti-aliasing strided convolutions and pooling operators and reducing stride size are helpful to make a learned similarity metric shift-invariant. Their experiments show that by integrating these elements into a neural network, the learned metric is more robust against imperceptible shifts and more consistent with the human visual similarity judgment  (Table 1,2,3) . 
main_review: Pros:
1) The paper tackles an important and interesting subject of developing shift-invariant metrics. The authors go one step further and support the claim of the importance of such metrics by conducting an experiment with humans (Section 3 and Figure 1)
2) The authors suggest interesting metrics to measure shift-invariance, which counts the number of samples for which the similarity rank flips when a sample was shifted and computes the rank-flip rate  - eq. 2 
3) The authors show that their suggested approach improves the proposed metrics for LPIPS arhcietcture (FIgure 3) and achieves comparable results with SOTA PIM similarity network 
4) It is interesting to see influence of the suggested architecture alterntaions on different backbones presented in Table 5

Cons:
1) The experiment with human perception is not clearly explained. First, it is hard to understand the number of different pairs that was eventually evaluated by humans or if there were several humans that were presented with the same pair  -- it would be very helpful to summarize those in a table. Also it is very suprising that humans managed were sensetive to 2 pixel shift in ~ 50% of cases. This surprising fact is only explained in the supplementary material. It might be helpful to revise the paper to allow explanation on specific numbers that were chosen (e.g., why only 50 images) and the included biases (e.g. explaining humans that there are shifts they should look for).  
2) The novelty of the paper is limited by minor alternations to the exiting architecture. Those alternations, e.g. anti-aliasing, are not new, as stated by the authors. The authors point our several times "PIM achieves shift robustness by training on neighboring video frames that often have small shifts. We work on an orthogonal solution by investigating neural network elements to make the learned metric robust and therefore only train our metrics on the examples without any shift through data augmentation." , this requires a more elaborative discussion on why we would like to choose this second route (which is also not optimal for a single pixel compared to PIM, as evedient from Tabls 1 and 3) There is no discussion of the effect of these additions on the runtime and comparison of the runtime with PIM.  
Table 2 shows that when evaluating bigger images the suggest approach is better than PIM, but the authors do not offer explanation why their approach works better
4) Many experiments yield somewhat not intuitive results, like the padding influence on performance in Table 5. The explanations offered are not very clear. Since the paper does not support the suggested approaches theoretically, it might benefit from more intuitive explanations of the experimental results. 
 
summary_of_the_review: I think the paper has merit, but it will benefit from revision, which empahsizes more motivation on why to choose a suggested approach as opposed to the other leading approach PIM and also why certain empirical results make sense.
review_title: null

Meta-review: This paper introduces a perceptual similarity on top on the commonly used perceptual loss in the literature (LPIPS). The authors draw experiments highlighting that human perceptual similarity is invariant to small shifts, whereas standard metrics are not. The paper studies several factors (anti-aliasing, pooling, striding, padding, skip connection) in order to propose a measure on top of LPIPS achieving shift-invariance. 

This paper initially received mixed reviews. RLHuY was positive about the submission, pointing out the relevance of the real human data and the studied factors for measuring the impact on shift invariance. RGQvy was slightly positive, but also raised several concerns on justification of the claimed properties, human perception experiments, and positioning with respect to data augmentation (PIM). RLHuY, an expert on the field, recommended clear rejection, pointing out missing references (including DISTS), the limited scope of the paper (shift invariance and tiny shifts). After rebuttal, RLHuY and RLHuY stuck to their positions ; RGQvy were inclined to borderline reject  because of unconvincing answers on comparison to data augmentation techniques. 

The AC's own readings confirmed the concerns raised by RGQvy and RLHuY, and points the following limitations: 

- The submission includes limited contribution and expected results: the studied modifications on neural networks' architecture, although meaningful, directly follow ideas borrowed from the literature. They are not supported by stronger theoretical analysis, and several insights related to accuracy or robustness remain unclear. 
- Experimental results are contrasted, e.g. compared to data-augmentation: although these approaches are more demanding at train time, they do not induce any overhead at test time - in contrast to the proposed approach.

Therefore, the AC recommends rejection.
Decision: Reject