Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce a measure, which they deem the deep-self-dissimilarity (DSD), which measures the dissimilarity between deep-features from the same image presented at different scales.  They show that this can be used as a powerful visual fingerprint, and can be adapted to both a full-reference and no-reference image quality metric.  
main_review: This is a clever and novel result, as far as I can tell, no other authors have suggested a measure of scale-wise dissimilarity of deep features as a descriptive measure of images, or one that has practical uses. The key idea, that despite images being self-similar in different scales in the pixel space, they are in fact quite dissimilar in different scales in deep feature response space, is also very novel and interesting on its own.

The performance of the full-reference metric on the pie-app dataset (outperforming LPIPS) is itself very interesting and impressive.

The visual results from the images are also impressive and suggest the measure is aligned with perception about images, though it would be fascinating to understand better how and why it aligns with perception.

A few comments:

I disagree with the description of it as a no-reference metric.  It is more accurately described as a reduced-reference quality metric.  Though it is true that the full scale reference image is not available to it, extracted statistics from a version of the image are available, which is more accurately reduced-reference. For the test case of super-resolution, it just turns out the degradation also coincides with the reduction to the reference.  This would not hold necessarily for other tasks requiring a no-reference metric.

Additionally, if you optimized a super resolution network just for the DSD metric, the output images will necessarily be related to the input images given no other constraints unlike other no-reference metrics which are comparisons between the output images and a distribution of natural images only, that will be optimized by output images that have no relation to the inputs.  This is because the DSD metric is actually a reduced-reference that also depends on the input, which is a reduced version of the reference.

This doesn't diminish the usefulness or novelty of the measure, especially in the case of super-resolution, but it should be clarified.


I'm not sure that I understand the experiment being run in figure 3 or what purpose it serves for the larger argument of the paper.


limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new image descriptor, called DSD (Deep Self-Dissimilarities). The authors observe that the very same image may happen to be classified in different ways by deep neural networks when viewed at different scales. For example (Figure 1) an image is classified by the same network as "Jean" at full resolution and as "Constrictor" at a lower scale. Therefore, to characterize an image, they consider the activations at a given layer of the net for the original image and its resized version, and compute some statistics that describe how they change. More precisely, the descriptor is computed as the difference between the Gram matrices of activations at full and reduced scale. Several experiments are carried out to illustrate the potential of the proposed descriptor.

main_review: The paper moves from some interesting considerations. In fact, the very same network makes different decisions for the same image depending on the image scale, and these variations may be descriptive of the image by themselves. The proposed descriptor seems to be able to capture such variations. However, using deep features to characterize images is hardly new, as the authors themselves acknowledge. So prior work on this topic should be analyzed adequately (just a few lines in the Related Work now) and similarities and differences with other proposal should be clearly pointed out. This would help putting the proposal in a correct perspective with respect to the state of the art. Along this line, it would be interesting to investigate in more depth what these observed dissimilarities depend upon, that is, which image features appear to be invariant/variant across scales.

Experiments on the possible uses of DSD provide mixed support for this proposal. Those concerning its use as full-reference and no-reference quality measures show an impressive performance and maybe should be expanded. The performance observed for image super-resolution and motion deblurring instead is less convincing and not above the state of the art. This is confirmed by the selected visual examples, Figure 3, 7, 9 and 10 where, contrary to what suggested by the comments, hardly any improvement can be noticed with respect to the references.

In summary, DSD is an interesting descriptor, with good potential, but should be better positioned with respect to the relevant literature and more deeply analyzed and justified.

Minor
- Eq.(2): I believe some constant may be missing to make the two terms, computed on image of different size, comparable
- Figure 2: what does rx_y stands for?
- typos: debluring

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

UPDATES

First of all, I want to thank the authors for answering my questions.
I appreciate the considerations about which features are invariant across image scales.
The analysis of the images with the highest and lowest DSD values is very interesting and can be of benefit for the paper. I also better analyzed the visual results for super-resolution and motion deblurring and I can see some improvements,
however I believe that a pointer in the text would definitely help.
Finally, for what concerns the prior work, I don't think the paper needs additional references or comparisons, but just an expansion of the related work analysis made in Section 2. This would allow a non-expert reader to better position the work with respect to prior art.

Overall, the authors addressed my major concerns and for this reason I improve my final rating.
limitations_and_societal_impact: The authors did not describe limitations and potential negative societal impacts of their work. For what concern limitations it would be interesting to better understand why for the image super-resolution and motion deblurring tasks, the proposed descriptor is not very effective.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes the use of "deep self-dissimilarities" (DSD) - meaning differences in feature activations at deeper layers of a network when computed on an image versus on a downsampled version of that image.  DSD is applied to several tasks - serving as a measure for predicting human preference of image quality/distortion, used as a loss-term for single image super-resolution, and as a loss term for motion deblurring.  In all cases, the proposed method performs well relative to existing state-of-the-art.
main_review: This paper begins by making the interesting observation that feature activations in later stages of a deep network do not exhibit consistency/similarity when computed over different scales/downsamplings of an image.  This is in contrast to raw image patches or early feature activations of the network.  Therefore, this paper proposes using this difference in activations, "deep self-dissimilarity" (DSD), as a visual fingerprint, and apply it to several tasks.

First, DSD is used as a full reference distortion measure, as well as a no-reference quality measure, and in both cases is shown to correlate better to human preferences than existing methods.  The latter requires making an assumption that, given a degraded version y of an image x, DSD(X) can be well-predicted by DSD(y|alpha) where y|alpha is a version of y downsampled by a factor of alpha.  As a note, it would be interesting to have be given some numbers/sense of how well this assumption holds (eg the test accuracy of the regression network).

Second, DSD is used as an additional loss term for single image super-resolution SR and motion deblurring.  In both cases, DSD performs well relative to existing state-of-the-art, and, unlike many methods for SR, without requiring GAN training.  

Overall, I felt that the paper was nice, in terms of making an interesting observation regarding deep self-dissimilarity, and based on this observation, proposing a novel measure that is empirically validated to give good performance on several tasks.

However, as a primary negative, from the paper, I don't have a strong sense or intuition for why the method works well.  What is the measure really keying in on, and what are possible failure cases/error modes?  Some explanation is given in section 3.3, but the overall argument seems to be quite high-level - that DSD focuses on smaller details compared with something like style dissimilarity.  A natural question that I don't think was addressed by the paper is how much the method depends on the specifics of the network from which the DSD is being computed?  For instance, how would the results change if that underlying network was trained with much more aggressive data augmentation or some architectural change to make it more scale-invariant?  Or, in another direction, Figure 2 gives mean dissimilarity, but how much variance is there?  Are there certain classes of images for which DSD is more/less informative?
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors observe that Gram matrices of image patches and of features extracted from convolutional networks behave differently. Namely that the latter are not invariant to image scale changes. Taking advantage of this invariance, they propose an image descriptor computed as a difference of Gram matrices of convnet features at two different image scales. The authors show the obtained descriptors can be used to compute an image quality measure in a full-reference setup that is in good alignment with human preference. An alternative no-reference image quality measure is also formulated and show cased in experiments for single-image super-resolution and motion debluring with promising results.
main_review: The proposed method is simple yet could be interesting, however, understanding the paper is harder than necessary, some choices seem to be ad-hoc and reporting of the results seems to be selective. See the suggestions below.

UPDATE: Thanks to the authors for addressing the concerns raised. After adapting the text as discussed, I would consider the paper to be acceptable.
limitations_and_societal_impact: Possible improvements:
- Fig. 2 - the caption should be improved, currently it does not give enough details to understand what is plotted, even though this information is in the main text. Consider labeling the x-axis of the plots clearer or mentioning in the caption (a) shows VGG layers and (b) patch sizes. Consider labeling the green bars as "Same image" instead of "Dissimilarity". And also consider replacing the y-label with "L_1 distance".
- Fig. 2 should include error bars.
- The definition (2) omits the layer from which the features are used. This makes it harder than necessary to think about the statements like the one on lines 91-92.
- In Table 1 I would expect to see results computed on the test set, however, the results for LPIPS are different from what was reported before. Make it clear which data is used to compute the results in Table 1.
- Connected to LPIPS results in Table 1, it is not clear which network was used for it, i.e., is this LPIPS-VGG,  LPIPS-Alex, LPIPS-Squeeze?
- Why does the Table 1 miss PieAPP results?
- The main text should be made more self-contained, definitely add details on the training/experiments setup, e.g., which resnet is meant on line 154, what the training setup is to allow for training a resnet with just 400 images, etc.
- Check that DeepDeblur and DeblurGAN-v2 are not mixed up in Fig. 8 (b).

Questions:
- Why is Fig. 2 showing the current selection of vgg layers, missing any layer deeper than 3-1?
- In Table 2, DSD4 performs good but DSD2 significantly worse, compared to this Table 1 shows values only for DSD2. What was the reasoning to report one or the other scale?
- Along the same lines as the point above, how is it selected which of the VGG layers is used to compute DSD?
- Were any other backbones than VGG tested?
review_title: null

Meta-review: This paper addresses a new image descriptor, referred to as deep self dissimilarity (DSD) which measures the dissimilarity between deep-features from the same image presented at different scales. Reviewers agree that the paper presents a novel idea which suggests a  measure of scale-wise dissimilarity of deep features as a descriptive measure of images. The paper is well written and the key idea is appreciated. The rebuttal addressed most of concerns raised by reviewers, leading that two of reviewers raised the score during the discussion period. I believe that the paper is deserved to be presented in the conference. 


Decision: Accept (Spotlight)