Deep Self-Dissimilarities as Powerful Visual Fingerprints
Authors: Idan Kligvasser
Tamar Rott Shaham
Yuval Bahat
Tomer Michaeli
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: descriptors
deep-features
IQA
image-restoration
super-resolution
motion-deblurring

Abstract: Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

Deep Self-Disimilarities as Powerful Visual Fingerprints
Idan Kligvasser
Tamar Rott Shaham
Technion-Israad Institute of Technology kligyasserdcampus. technio on. ac.
Technion- Institute of Technology stamarot@campus. technion. ac.
Yuval Bahat
Tomer Michaeli
Technion-Isracl Institute of Technology yuval. bahat@campus technion, ac.
Technion- Institute of Technology tomer m@ee. technion ac. il
Abstract
Features extracted from deep layers classification networks are widely used as image descriptors. Here, we exploit unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statis- ties across image scales, it turns that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used powerful visual fingerprint. Particularly, illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD loss function in training of image restoration networks, leads to results that are at least as thoto-realistic those obtained by GAN based methods, while not requiring adversarial training.
Introduction
Features extracted from deep layers of classification networks are widely used as powerful image descriptors. These features known capture high level semantics [44 as well low-level textural cues 14 and are thus exploited in numerous applications, including image enhancement [24] [8 [9], and editing
10 47). synthesis [52
[15 2 225 both in the form of per-element similarity
measures (e.g. the perceptual loss [1 [8] and LPIPS (51) and for comparing internal image
distributions (e.g. the style loss
contextual loss and projected distribution loss (0).
In this paper, we propose to exploit surprisingly dominant property of deep features: the dissimilarity between their internal distributions at different image scales. As opposed to small patches in pixel space, which are known to exhibit similar characteristics across scales [53] 16). here we show that deep features tend vary significantly for different scales of the same image. glimpse into this phenomenon is provided in Fig. for the VGG-19 network [46). As can be seen, the network often outputs completely different classification results for the same image different resolutions. We
show that this behavior also characteristic of earlier stages within the network, and can therefore serve a powerful image fingerprint.
naive approach account for this phenomenon would to aggregate deep feature descriptors from multiple image scales. For example, to measure discrepancy between two images, one could accumulate deep feature distribution distances, measured separately at different scales, as schemati- cally illustrated in Fig. 5 (middle) for the case of two scales. However, here propose different approach, which as we show, far more powerful as a visual fingerprint. Specifically, we present the deep dissiiiiiillartt (DSD) image descriptor, which captures differences between internal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

WI
"Jean"
"Constrictor"
"Christmas stocking"
"Mitten"
"Garfish"
"Neck brace"
Figure I: Class variations between scales, The semantics captured by an image classifier (VGG-19 in this case 46)), strongly depends the image resolution. This phenomenon also occurs carlier classification layers; the deep features tend to vary significantly when changing the input's 's resolution. We show how these dissimilarities quality measures, as well as loss functions for image restoration.
serve construct full-reference and no-reference image
feature distributions at different scales of an image. As we show, this descriptor can serve both as a full reference image fidelity metric (i.e. by comparing the DSD's 's of two images) and no-reference measure (quantifying the "naturalness" of an image according to its DSD). In both cases, metrics highly correlated with human perception, quality assessment datasets. Furthermore, we demonstrate the effectiveness DSD as a loss function for image restoration tasks. Our approach leads to high perceptual quality reconstructions that are at least comparable to GAN-based methods, while completely avoiding adversarial training.
verify on uscr-annotated image
2
Related work
Deep features for similarity measures. Using deep features of a pre-trained classification network for measuring fidelity between images, was first introduced in the context of network inversion and This idea was quickly adopted perceptual similarity tasks such as
visualization 29
45 50
[27] and more. Nocal (distribution based) variants of this per-element perceptual loss were also proposed. These include the style loss (15), the contextual loss [32). and the projected distribution loss [IO]. These approaches compare between the internal deep-feature distributions of two inputs, a single resolution. Here, we show that there is more powerful way to use deep features: comparing their between-scale diiiiiiisimllaiiies..
texture synthesis 14, style transfer [15] 18], super resolution
[88] 24). image inpainting
Internal recurrence and self-similarity Small image patches tend to recur within natural images, both in the same scale [8 9
6. [1], and across different scales [16]. Cross-scale recurrence has been used as a prior for various tasks, including super resolution [16 13] denoising (54 (37), dehazing [4], blind deblurring 34 3], and blind super resolution (33] In this work show that opposed to image patches, deep features do not exhibit strong similarities across scales, implying that deep self self-similarities cannot serve as a prior e.g. for restoration tasks. Nonetheless, we show that the deep self dissimilarity pattern of an image is meaningful, and can therefore serve as a powerful fingerprint.
3 The Self-Dissimilarity Between Deep Features at Different Scales
3.1 Deep self-dissimilarity
We characterize deep feature distributions through their second-order statistics. More concretely, we make use of Gram matrices of the channel activations (which are proportional to the empirical covariance the channels),
(G(()) WH Σ (z)))) (x))a.
Here (x))& is the activation at spatial position within the i-th channel of the L-th layer of network that is fed with the image x at its input. W and H are the width and height of the this map.

---

-10
-10
Dissinilarity
Dissimilariiy
5
0.5
3
ReLU12 ReLU21 ReLUzz ReLUn ReLU,1
n=5
Random Same class dieimmiissmillartty
Random
Same class Self-dissimilarity
(a) Deep Dissimilarity
(b) RGB Dissimilarity
Figure 2: Dissimilarity of image patches and deep features. We compare the dissimilarities between images (blue & red) the self-dissimilarity within images (green), both for deep features and for small image patches|p [b) In contrast to image patches and shallow network layers, the large
average self-dissimilarity levels (green) deep network layers (right columns in (a)) are comparable
to the cross-image dissimilarities (blue & red) in those layers. This shows that internal feature distributions deep layers are not similar across image scales.
We define the deep self-dissimilarity (DSD) of an image æ as the difference between the é-th Gram matrices corresponding to two image scales, r and æ Ja.
DSD (1) Ge(x) Ge(xta).
In other words, DSD (r) a matrix whose (i, )-th entry measures the extent to which channels i and of the f-th layer are correlated when feeding the network, than when feeding Lo. We often choose the scaling factor a to be 2 or 4. We sometimes omit the and & subscripts for brevity.
Dissimilarity RGB space vs. feature space
The strong tendency of small image patches (in pixel space) to recur across different scales of an image [53] manifested in similarity between distributions of patches across different image scales. If that was the case also for deep features, then DSD would always equal (approximately) the zero matrix, and thus would not constitute a unique fingerprint for images. However, as we show bellow, deep features behave very differently from small image patches in this respect.
We perform comparison between deep feature self dissimilaritss and pixel-space dissimilarities for = 4. For the latter, we replace the deep feature Gram matrices by pixel-space patch Gram matrices,
(GRGn(x))ss WH Σ
1
where [æ*]& j is the i-th pixel in a column-stacken n x n patch, centered at the k-th pixel of the image W and H here correspond to the width and height of the image. For both deep features and pixel-space patches, we depict in Fig. 2 (i) the average & distance between Gz(x) and Gr Lt) over random images sampled from ImageNet 42) (green bars), (ii) the average l distance between Ge(z) and Ge(y) over random image pairs (z, y) from ImageNet (blue bars), and (iii) the average l distance between Ge(x) and Ge(y) over random image pairs (x. y) sampled from the same class in ImageNet (red bars). Distances are computed for several VGG layers & and several patch sizes n.
As can be seen, the self dissimiiiiiil values (green) computed over image patches and shallow net- work layers are significantly lower than their pross-image counterparts (blue and red), in accordance with the well known ccoss-scale patch recurrence phenomenon in natural images. However, when it

---

Low DSD
High DSD
Figure 3: Visualising DSD. Images from the ImageNet validation set corresponding to lowest (top) and highest (bottom) DSD values. We observe that natural images with the lowest DSD tend to contain large smooth regions, whereas images with the largest DSD contain high frequency textures.
comes deeper network layers (Fig. 2ap. the average Gram dissimilarities between different scales of the same image are roughly as large as their cross-image counterparts, suggesting that deep features do exhibit self similarity like pixel-spae space patches.
This conclusion is also supported by Fig. 6 Here, we optimized the RGB values of an image x so as to minimize [|DSD la for layer = ReLUU31 As can be seen in the right image, too many (5K) gradient descent steps result in the image exhibiting unnatural artifacts. This illustrates again that natural images are in fact not characterized by small DSD values.
3.3 Visualizing DSD
What does the DSD fingerprint capture, then? We answer this question through several visualizations. Fig. presents the images from the ImageNet validations set, which have the lowest and the highest DSD values layer at e ReLU31. As be seen, images with higher DSD values (bottom row) typically contain finer details and textures, compared to low DSD value images (top row). This makes sense, as computing DSD involves downsampling the image, which results in erasing fine image details. This significantly changes the Gram matrix computed over deep features corresponding to the downsampled image, which in turn increases the difference between the two Gram matrices, computed in (2.. Examining the statistics of DSD values across different classes in the ImageNet dataset reveals similar behaviour. For example the class "sandbar" has an average DSD of 2.1 10-3 0.83 10 10-3 while the average DSD of the class "window screen" is 13.5 10 + 11.0 10
Next, we visualize the effect of modifying a specific content image x. such that its DSD fingerprint is as similar as possible to that of reference image xy. We do this once by minimizing [IDSD2(x.)
DSD2 (ax,) | and once by minimizing [DDSD2 (2) DSDa 42) Namely, we try match the images' deep self-dissimilarities between scales 1 and 0.5, and between scales 0.5 and 0.25. Here DSD, is calculated by aggregating the DSD matrices corresponding to VGG layers l (ReLU,,, ReLU22LU2222 ReLUgi) For comparison, we perform the same experiment, but with the style losses [|G(xz) G(z,) 112 |G(r 12) G(I, 12) and [[G(x, 44) G(æ, L4)ll1. The latter experiment can be thought of as style pyramid, whereas the former as a style-differences pyramid (in analogy to the Gaussian and Laplacian pyramids). The results of the two experiments are presented in Fig. 4 As can be seen, while style dissimilarity (upper row) can be minimized by merely augmenting vague patterns from the reference image, matching the DSD fingerprints (bottom row) requires adding sharper, more detailed visual structures. This suggests that the DSD loss is more intricate fingerprint, and can serve as a better loss for image quality assessment and for image restoration.

---

Content
Style a 1
Style a =
Style a 4
Reference
(DDSDa(E) DSD
(D,SD2(* DSDa(>, 42)l
Figure 4: The effect of changing the DSD of an image. To visualize what the DSD fingerprint captures, we modify the content image have similar fingerprint to that of the reference image (bottom row). We do this for two pairs of scales (between full res. and a = 2, and between = 2 and a 4). For comparison, scales (top row). As can be seen, matching DSD fingerprints requires modifying finer details.
also perform the same experiment using the style loss for the three
3.4 DSD as full-reference measure
The DSD can be used define a full-reference distortion measure simply by computing the & distance between the DSD's of the two images we wish to compare,
CDSD. (x, v) DDSD. (z) DSD
(4)
This distance quantifies the extent to which the differences between the deep feature distributions different scales of x are similar to those of (see Fig. [5] right pane).
We evaluate this measure in perceptual image quality assessment task. We use the PieAPP dataset [40]. which contains 4800 pairs of images, where one image the reference and the other is distorted version of that reference. The dataset includes a variety of distortions, along with scores for their distortion level obtained from human ranking. We compare CDSD with several other full-reference measures that are based on deep network features: the perceptual loss [|ő(z) d(y)l the style loss [[G(x) G(y) I| and the projected distribution loss (PDL) 10. To isolate the effect of using multiple image scales in DSD, we also add comparison to multi-scale variant of the style loss, [[G(z) G + (2) - G2yl2)) la (see schematic illustration in Fig. 5) Note that none of these metrics were specifically designed for (or even fine-tuned on) the PieAPP dataset.
We measured the correlation between each metric and the human preferences. Specifically, for every two distorted versions Ta; Ty of the same reference image Ir, the PieAPP dataset (40) provides the probability that a human rater would prefer version v, over version æ5. We calculated the Pearson correlation between these probabilities and the loss differences [( x,,) [( T.,), for each of the full-reference losses C(.-)) -) we examined Table presents the results for several different layers of the VGG network. Correlation with the LPIPS measure [51), which uses a different network, is also reported. Note that DSD values computed using the deeper network layers (two right columns) have the highest correlation with human evaluation scores, compared to all other measures. This illustrates that the DSD fingerprint can be used as an effective perceptual distortion measure.
3.5
DSD as no-reference quality measure
The DSD fingerprint can also serve to define no-reference image quality measure. Let us revisit Fig. 6 in which we optimized an image x so to minimize (DSD(x))l+ Here, the original image a super-resolution result produced by the ESRGAN method (48) As already mentioned, after 5K gradient descent steps the optimized image contains unnatural artifacts. However, note that an interesting phenomenon occurs if we stop the optimization after only 500 steps. In that case, the optimized image is sharper and more naturally looking than the original ESRGAN result. This
'The Pearson correlation measures the extent to which the loss difference is a monotonic function of the preference probability.
5

---

I+
1
I
I
I
L
I
I
I
I
Comparing deep feature distributions (style loss)
Comparing deep feature distributions at multiple scales (multi-scale style)
Comparing the cross-scale dissimilarity of deep feature distributions (our)
Deep self-dissimilarities as a full reference distortion measure. As opposed to the standard style loss, which compares two images according to the distance between their Gram matrices at single image scale (leftmost), we wish to exploit the dissimilarity between different image scales. A naive approach is to sum the style loss across scales (middle). However, we show that comparing the self dissimillaryy of deep features between image scales (rightmost) is much more powerful metric.
Figure 5:
Method
ReLU21
ReLU31
ReLU41
ReLUs11
Perceptual
0.448
0.449
0.461
0.630
PDL.
0.622
0.636
0.620
0.661
Style
0.645
0.714
0.711
0.706
Multi-scale Style
0.645
0.698
0.709
0.714
CDSD2 (ours)
0,463
0.608
0.769
0.761
LDSD, (ours)
0.570
0.667
0.607
0.600
LPIPS
0.641
Table 1: DSD as a full reference measure. We calculate the Pearson correlation between full- reference image distortion measures and human evaluation scores using the PieAPP dataset [40]. As can be seen, the DSD calculated on deep network layers achieves the highest correlation.
suggests that for this particular scene, there is a specific optimal |IDSD(z)]] value, which is in between that of the original image, and the 5K minimization result.
propose to DSD for computing no-reference image quality score. asses the quality of potentially degraded image y, we would like to measure the difference between (DDS (y)llı and [DDDa the no-reference case we do not have access to æ. One could conceive replacing [[DSD, (x) by its average value for natural images. The problem with this approach is that different natural images have very different DDDD (z))l1 values, as evident from the histogram in Fig. 7 A different approach would be train regression model predict |JDSD, (x)]l1 from y. However, restrict our measure to specific degradations, and thus would like to avoid using datasets of degraded images for training. Our solution from the downsampled clean input, & fo- At test time, we input to our model the downsampled degraded image, y Va- and use the model's output as our estimate for |JDSD (x) The reasoning behind this choice, is that for many types of degradations ta and x La are quite similar (e.g. noise, blur, compression artifacts, etc.). We train the regression network z (we use convolutional network with 12 residual blocks (17)) on the BSD training set [30] containing 400 clean images, using the mean square error loss. Our no-reference measure is therefore given by
Following this observation,
where is the degradation-free version of y. Obviously, in
do not want to
train light-weight regression model v/r to predict DSD., (x:)) []
@OSD. (v) 1 la) |DSD |.
(5)

---

DSD too high (ESRGAN)
Optimal DSD (500 iterations)
DSD too low (5K iterations)
Figure 6: Optimal deep self-dissimilarity. We minimize [[DSD, starting from output by the ESRGAN (48 method (left), having [IDSDa(x)II = 6.3 10-5 The result (right) after 5K iterations (with [x)] dropping 1.2 10-3) contains undesired artifacts. However, stopping the optimization earlier (after 500 steps, when [DDDa(>)]l = 2.2 10-9) yields a sharper and more
natural looking image (middle), compared to the original ESRGAN output.
Method NIQE
PCC
180
0.223
Ma
0.284
BRISQUE
0.363
DSD2
0.330
DSD1
0.444
DEDs
Figure 7: DSD distribution. His- togram of [D)D2(x)]la calculated 200 natural images from the PieAPP dataset [40]. DSD values strongly vary across different images.
Table 2: DSD as a no-reference measure. We report the Pearson correlation between human evaluation scores from [40] and various no-reference metrics, including our DSD, measure, with = 2 and 4. As be seen, DSD4 significantly outperforms the other quality measures.
We compare our proposed measure to the leading existing no-reference image quality measures, including the naturalness image quality evaluator (NIQE) (36), the blind/referenceless image spatial quality evaluator (BRISQUE) (35 and the score proposed by Ma et al. [28]. Similarly to the full- reference case, compute the Pearson correlation between the human evaluation scores from (40] and the evaluated scores, and present the results in Tab, 2 DSD, significantly outperforms the competitors terms of correlation with human assessment.
4 Experiments
We use the DSD fingerprint for the tasks of single image super-resolution (SR) and motion debluring. In both cases our goal is recover clean image from its degraded observation y. To this end, train a restoration network on pairs of training examples {x, y} by minimizing the DSD-based full reference loss term (4) between the restored image & and its ground-truth counterpart æ. We combine our loss with two other popular loss terms, as common in restoration methods (e.g. [24 48),
L(x,2) = Cper(2e xr. 2) + A Lree (x, ②) + ApSD LusD(z, 2))
(6)
Here (xe( || 1/1,, Eper (l)e(*) de (z))) is the perceptual loss computed over layer Conv54 of VGG, and Ares; ADSn are weighting coefficients. For LDSD we use layers Ć (ReLU21, ReLU22, ReLUUi1 (as in Sec. 3.3) and 2. We train our networks for 300K epochs using the Adam optimizer with a batch size of and an initial learning-rate of 2 10-4, which halved after 90K, 180K and 270K steps. See Supplementary Material (SM) for full training details.
7

---

Low resolution
ESRGAN
SANGAN
Ours
Figure 8: DSD for 4x super resolution. We compare method the state-of-the-an ESR- GAN 48 and SANGAN (1I. As can be seen, our approach leads to sharper and more photo-realistic images comparing to the GAN based methods, although
adversarial training.
4.1 Single Image Super Resolution
Here our goal is to predict high-resolution (HR) image x from its low-resolution (LR) version We focus on 4x SR, where the LR image corresponds to bicubic downsampling of the HR image. Training is done over 800 LR-HR image pairs from the DIVK2K dataset []]. Our SR network utilizes consisting of 10 residual blocks.
the xSRResNet architecture
22
Figure 8 presents a qualitative comparison with ESRGAN A8] and SANGAN 21]. two state-of-the- art GAN-based methods that target perceptual quality. Note that although our method does not require any adversarial training, it manages to restore finer image details, and to produce more realistic textures. Please refer the SM for many additional visual examples.
We next follow 7 and quantitatively evaluate the performance of our method by reporting perceptual quality (calculated using NIQE [36), lower better) and image distortion (using SSIM [49] higher is better). In Fig. 19 we compare our model (red dot) with the lcading SR methods EDSR (26]
VDSR (20). SRResNet (24), xSRResNet 222] Deng [I]. ESRGAN (48). SRGAN 244 ENET 43
and SinGAN [1] (black dots). We also compare variants of our model, trained by replacing PDD with other deep
based losses (blue dots): the projected distribution loss (PDL.
[0]]]
and the style loss (STY) [15]. Finally, we present results for our model trained using only Lree and Eper. or only free (indicated by "per" and empty subscripts respectively). All scores are calculated over the BSD100 test set [30]. As can be seen, method is among the best in terms of perceptual quality, while maintaining relatively low distortion. Note that achieving such perceptual quality has been possible to date only with GAN based methods. Our method achieves such performance without the need adversarial training, which is known to be unstable and challenging in practice.
To further investigate the contribution of the DSD loss, we perform quantitative evaluation of our method when LDSD replaced by other loss terms that measure distances between deep feature distributions, including the contextual loss (CX) (32), projected distribution loss (PDL) [10, style loss [5] and the multi-scale style loss (MSS) defined in Sec.33.4 For each of these loss terms, we adjust the value of Apsp as to make the perceptual and feature distribution loss terms equal on average. The results are presented Tab. 3 which reports PSNR, LPIPS (51] and NIQE (36). In each experiment we calculate all losses using the same VGG layers (indicated at the top of each column). as well as the same training procedure and network architecture (here we use shallow xSRResNet [22) with 5 residual blocks). In terms of perceptual quality (NIQE and LPIPS), the model

---

VDSR
EDDSR Rt
DeblarG N-v2
RRt DIP
SRReaNet
3.5
Deng
NIQE
DeepDeblur
Net
ENET
SRRR
SinGAN
R
ESRGAN
xStRReNee....
SRRoaNNtoood
X SSRResNetard
SRGAN
SANGAN
2
0.75
0.7
0.65
0.6
0.95
0.9
0.85
0.8
0.75
0.7
SSIM
SSIM
(a) super-resolution
(b) motion debluring
Figure 9: Perception-distortion evaluation. We report perceptual quality (NIQE, lower is better) and image distortion (SSIM, higher better) for the tasks of (a) super-resolution and (b) motion debluring. We compare models (red), existing slate-of-the-att algorithms (black) and variants of our model trained by replacing LDDD with other deep feature distribution loss terms (blue). For SR, method is among the best perceptual quality, while requiring GAN training. For motion debluring, our model improves over existing methods in perceptual quality by a large margin, while maintaining relatively low distortion.
Method
ReLU21
ReLU22
ReLU31
CX
25.35 / 0.251 /7.8
25.77
6.02
25.69 / (..26 5.98
PDL
25.21 0.208 5.80
25.22 / 0.212 4.33
25.25 0.211 4.50
Style
24.93 / 0. 19197 3.64
24.84 00.18 / 3.73
24.66 / 0.190 / 4.20
MSS
24.77 / 0.1 / 3.86
24.72 0.190 4.01
24.87 / 0.196 3.87
DSD (Ours)
24.88 / 0.194/ 3.49
3.49
25.11 / 0.187 / 3.78
24.99 / 0.190 4.06
Table 3: Comparison with other feature distribution losses in 4x SR. PSNR LPIPS / NIQE
scores for different loss functions utilizing different VGG layers. DSD based models (bottom row) perform among the best in terms of perceptual quality (NIQE and LPIPS).
trained with our DSD loss (bottom row) is among the best for all examined activation layers. This also supported by Fig- [0 which shows a visual comparison between the methods. Our result contains less grid artifacts compared to the other losses (exemplified here for layer ReI.U22)).
4.2 Single Image Motion Deblurring
Here we aim to recover sharp, blur free image x from a blurry image y. We use the same loss function, training protocol and architecture (xSRResNet [21] with 10 res-blocks) in our SR experiments. Training is done using the REDS dataset (39] consisting of 30, 000 image pairs (x, y} from 300 different scenes. Figure II presents an example result of our motion deblurring, compared with the state-of-the-art DeblurGAN-y loss [5] instead of LDDD The result using the DSD loss is sharper and contains no ghosting artifacts. Please see many more results in the SM.
[23] and a variant of our method trained with the style
This observation is further supported quantitatively. In Fig. we report perceptual quality and distortion (using the same measures as in Fig, 92) over 100 random images from the REDS validation We compare method (red dot) with the state-of-the-art DeblurGAN-v2 and DeepDeblur 38 (black dots), as well as with three variants of our method (blue dots) trained by replacing LDDD with either the style loss (STY) [15], the multi-scale style loss (MSS) (from Sec. 3.4). and when omitting LDSD and keeping only Liee and Lper (denoted by a subscript "per"). Here as well, the perception-distortionn plot indicates that our DSD-based model obtains the best perceptual quality (significantly better than most of the competition), while exhibiting low distortion.
2 An official newer version available at: hutp://gthbb..oonSuuuggguaaDDbee

---

Low resolution
PDL
Style
Multi-scale Style
Ours
Figure 10: Looking closer into the effect of DSD. Substituting LISD with other loss terms that are based on feature distributions results in undesired grid artifacts (middle images). These artifacts get significantly reduced when using the DSD loss term (right).
Blurred
DeblurGAN-v2
Style
Ours
Figure 11: DSD for motion debluring. We visually compare result with DeblurGAN-v2 and with variant of our method where the DSD loss is replaced by the style loss. As can be seen, DSD achieves better visual quality, leading to sharper, more natural appearing result.
Conclusion
Deep features corresponding different image scales exhibit meaningful dissimilarity. We prove this to be powerful image fingerprint, highly correlated with human preference in both full-reference and no-reference image quality assessment, and leading a GAN-like highly photo-realistic image restoration (while avoiding unstable adversarial training). We believe that deep self eelf-dissimilarity can benefit additional image restoration and manipulation tasks.
Acknowledgments This research was supported by the Israel Science Foundation (grant 852/17) and by the Technion Ollendorff Minerva Center.
10

---

References
Agustsson, E., Timofte. R.: Ntir 2017 challenge single image super-resolution: Dataset and study. In: The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops (July 2017)
[2] Arar, M., Danon, D., Cohen-Or Or, D., Shamir, A.: Image resizing reconstruction from deep features. arXiv preprint arXiv:1904.08477 (2019)
[3] Bahat, Y., Efrat, N., Irani, M.: Non-uniform blind deblurring by reblurring. In: 2017 IEEE International Conference on Computer Vision (2017)
[4] Bahat, Y., Irani, M.: Blind dehazing using internal patch recurrence. In: 2016 IEEE International Confer- Computational Photography (ICCP). pp. 1-9. TEEE (2016)
[5] Bansal, A., Sheikh, Y., Ramanan, D.: Pixelnn: Example-based image synthesis. In: International Confer- Learning Representations (2018)
[6] Bamsley, M.F., Sloan, A.D.: A better way compress images. BYTE 13(1), 215-223 (Jan 1988), http://dl. wwt.:/a.............///.. wwtp///gtttiiio cfm?idd44935..... 44950
[7] Blau, Y., Michaeli, T.: The perceptionn-disortion tradeoff. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
[8] Buades, A., Coll, B., Morel, J.M.: A noc-local algorithm for image denoising. In: CVPR (2005)
[9] Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing 16(8), 2080-2095 (2007)
[10] Delbracio, M., Talebi, H., Milanfar, P:: Projected distribution loss for image enhancement. arXiv preprint arXiv:2012.09289 (2020)
[11] Deng, X.: Enhancing image quality via style transfer for single image super-resolution. IEEE Signal Processing Letters 25(4), 571-575 (2018)
[12] Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the seventh TEFE international conference on computer vision. vol. 2, pp. 1033-1038. IEEE (1999)
[13] Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM Transactions on Graphics 30(2), 12 (2011)
[14] Gatys, L.,, Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. Advances in neural information processing systems 28, 262-270 (2015)
[15] Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)
[16] Glasmer, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference Computer Vision (ICCV), pp. 349-356. IEEE (2009)
[17] He. K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 (2016)
[18] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)
[19] Katzir, O., Lischinski, D., Cohen-Or, D.: Cross-domain cascaded deep feature translation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
[20] Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016)
[21] Kligvasser, I., Michaeli, T.: Sparsity aware normalization for gans. AAAI (2021)
[22] Kligvasser. Shaham, T.R., Michaeli, T. xunit: Learning spatial activation function for efficient image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(June 2018)
[23] Kupyn, O., Martyniuk, T., Wu, J.. Wang, Z.: Deblurgan-v2: Deblurring (orders-of of oaagmiatuuue) faster and better. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8878 8887 (2019)

---

[24] Ledig. C., Theis, L., Huszár, F., Caballero, J., Cunningham, A.. Acosta, A., Aitken, A.. Tejani, A., Totz, J..
Wang. Z.. et al.: Photo-realistic single image super-resolution using generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1681-46900 (2017)
[25] Lee, C.H., Liu, Z., Wu, L.,, Luo, P:: Maskgan: Towards diverse and interactive facial image manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549-5558 (2020)
[26] Lim, B., Son, S., Kim, H., Nah, S., Mu Lec, K.: Enhanced deep residual networks for single image super- resolution. In: The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops (July 2017)
[27] Liu, G., Reda, F.A., Shib, K.J., Wang. T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp- 85-100 (2018)
[28] Ma. C., Yang. C.Y., Yang. X., Yang, M.H.: Learning no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding 158, 1-16 (2017)
[29] Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5188 5196 (2015)
[30] Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statisties. In: Proc. 8th Int'l Conf. Computer Vision. vol. pp. 416-423 (July 2001)
[31] Mechrez, R., Shechtman, E., Zelnik-Manor L.: Saliency driven image manipulation. In: 2018 IEEE Winter Conference on Applications Computer Vision (WACV). pp. 1368-1376. IEEE (2018)
[32] Mechrez, R., Talmi, L., Zelnik-Manor, L.: The contextual loss for image transformation with non-aligned data. In: Proceedings of the European conference on computer vision (ECCV). pp. 768-783 (2018)
[33] Michaeli, T., Irani, M.: Nonparametric blind super-resolution. In: The IEBE International Conference on Computer Vision (ICCV) (December 2013)
[34] Michacli, T., Trani, M.: Blind deblurring using internal patch recurrence. In: European Conference on Computer Vision. pp. 783-798. Springer (2014)
[35] Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing 21()2), 4695-4708 (2012)
[36] Mittal, A., Soundararajan, R., Bovik, A.C.: Making "completely blind" image quality analyzer. IEEE Signal Processing Letters 20(3), 209 212 (2012)
[37] Mosseri, I,, Zontak, M., Irani, M.: Combining the power of internal and external denoising. In: ICCP (2013)
[38] Nah, S., Hyun Kim, T., Mu Lec, K.: Deep multi-scale convolutional neural network for dynamic seene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3883-3891 (2017)
[39] Nah. S., Son, S., Timofte, R., Lee, K.M.: Ntire 2020 challenge image and video deblurring. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops. pp-
416-417 (2020)
[40] Prashnani, E., Cai, H., Mostofi, Y., Sen, P.: Pieapp: Perceptual image-error assessment through pairwise preference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1808 1817 (2018)
[41] Rott Shaham, T., Dekel, T., Michaeli, T.: Singan: Learning generative model from single natural image. Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4570-4580 (2019)
[42] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge (2015)
[43] Sajjadi, M.S. Scholkopf, B., Hirsch, M.: Enhancenet: Single image supoutiesolution through automated texture synthesis. In: The IEEE International Conference on Computer Vision (ICCV) (Oet 2017)
12

---

[44] Shocher, A.. Gandelsman, Y., Mosseri, L,, Yarom, M. Irani, M., Freeman, W.T., Dekel, T.: Semantic pyramid for image generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 7457-7466 (2020)
[45] Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image
classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
[46] Simonyan, K., Zisserman, A:: Very deep convolutional networks for large-scale image recognition. arXiv preprint wrXiv:1499.155 (2014)
[47] Song, Y., Yang. C.. Lin, Z.,, Liu, X., Huang, Q.. Li, H., Kuo, C.C.J.: Contextual-based image inpainting: Infer, match, and translate. In: Proceedings the European Conference on Computer Vision (ECCV). pp. 3-19 (2018)
[48] Wang, X., K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, Esrgan: Enhanced super- resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 0-0 (2018)
[49] Wang. Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P. Image quality assessment: from emoor visibility to structural similarity. IEEE transactions on image processing 13(4), 600 612661266 (2004)
[50] Yosinski. J., Clune, J.. Nguyen, A., Fuchs, T., Lipson. H.: Understanding neural networks through deep visualization. arXiv preprint 1rXiv:1550606655 (2015)
[51] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang. O.: The unreasonable effectiveness of deep features as perceptual metric. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 586-586555555 (2018)
[52] Zhou, Y., Zhu, Z.. Bai, X.. Lischinski, D., Cohen-Or, D., Huang, H.: Non-stationary texture synthesis by adversarial expansion. ACM Transactions Graphics (TOG) 37(4), 1-13 (2018)
[53] Zontak. M., Irani, M.: Internal statistics of single natural image. In: CVPR 2011. 977-984, IEEE (2011)
[54] Zontak, M., Mosseri, I., Irani, M.: Separating signal from noise using patch recurrence across scales. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1195-1202 (2013)
13


Reviews:
[{"review_id": "xQPE0HSFDm", "reviewer": "Reviewer_n5Q1", "summary": "The authors introduce a measure, which they deem the deep-self-dissimilarity (DSD), which measures the dissimilarity between deep-features from the same image presented at different scales.  They show that this can be used as a powerful visual fingerprint, and can be adapted to both a full-reference and no-reference image quality metric.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a clever and novel result, as far as I can tell, no other authors have suggested a measure of scale-wise dissimilarity of deep features as a descriptive measure of images, or one that has practical uses. The key idea, that despite images being self-similar in different scales in the pixel space, they are in fact quite dissimilar in different scales in deep feature response space, is also very novel and interesting on its own.\n\nThe performance of the full-reference metric on the pie-app dataset (outperforming LPIPS) is itself very interesting and impressive.\n\nThe visual results from the images are also impressive and suggest the measure is aligned with perception about images, though it would be fascinating to understand better how and why it aligns with perception.\n\nA few comments:\n\nI disagree with the description of it as a no-reference metric.  It is more accurately described as a reduced-reference quality metric.  Though it is true that the full scale reference image is not available to it, extracted statistics from a version of the image are available, which is more accurately reduced-reference. For the test case of super-resolution, it just turns out the degradation also coincides with the reduction to the reference.  This would not hold necessarily for other tasks requiring a no-reference metric.\n\nAdditionally, if you optimized a super resolution network just for the DSD metric, the output images will necessarily be related to the input images given no other constraints unlike other no-reference metrics which are comparisons between the output images and a distribution of natural images only, that will be optimized by output images that have no relation to the inputs.  This is because the DSD metric is actually a reduced-reference that also depends on the input, which is a reduced version of the reference.\n\nThis doesn't diminish the usefulness or novelty of the measure, especially in the case of super-resolution, but it should be clarified.\n\n\nI'm not sure that I understand the experiment being run in figure 3 or what purpose it serves for the larger argument of the paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "hrw5F2REfWT", "reviewer": "Reviewer_ooMi", "summary": "This paper proposes the use of \"deep self-dissimilarities\" (DSD) - meaning differences in feature activations at deeper layers of a network when computed on an image versus on a downsampled version of that image.  DSD is applied to several tasks - serving as a measure for predicting human preference of image quality/distortion, used as a loss-term for single image super-resolution, and as a loss term for motion deblurring.  In all cases, the proposed method performs well relative to existing state-of-the-art.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper begins by making the interesting observation that feature activations in later stages of a deep network do not exhibit consistency/similarity when computed over different scales/downsamplings of an image.  This is in contrast to raw image patches or early feature activations of the network.  Therefore, this paper proposes using this difference in activations, \"deep self-dissimilarity\" (DSD), as a visual fingerprint, and apply it to several tasks.\n\nFirst, DSD is used as a full reference distortion measure, as well as a no-reference quality measure, and in both cases is shown to correlate better to human preferences than existing methods.  The latter requires making an assumption that, given a degraded version y of an image x, DSD(X) can be well-predicted by DSD(y|alpha) where y|alpha is a version of y downsampled by a factor of alpha.  As a note, it would be interesting to have be given some numbers/sense of how well this assumption holds (eg the test accuracy of the regression network).\n\nSecond, DSD is used as an additional loss term for single image super-resolution SR and motion deblurring.  In both cases, DSD performs well relative to existing state-of-the-art, and, unlike many methods for SR, without requiring GAN training.  \n\nOverall, I felt that the paper was nice, in terms of making an interesting observation regarding deep self-dissimilarity, and based on this observation, proposing a novel measure that is empirically validated to give good performance on several tasks.\n\nHowever, as a primary negative, from the paper, I don't have a strong sense or intuition for why the method works well.  What is the measure really keying in on, and what are possible failure cases/error modes?  Some explanation is given in section 3.3, but the overall argument seems to be quite high-level - that DSD focuses on smaller details compared with something like style dissimilarity.  A natural question that I don't think was addressed by the paper is how much the method depends on the specifics of the network from which the DSD is being computed?  For instance, how would the results change if that underlying network was trained with much more aggressive data augmentation or some architectural change to make it more scale-invariant?  Or, in another direction, Figure 2 gives mean dissimilarity, but how much variance is there?  Are there certain classes of images for which DSD is more/less informative?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Y2DgogIaE9j", "reviewer": "Reviewer_C5ui", "summary": "The authors observe that Gram matrices of image patches and of features extracted from convolutional networks behave differently. Namely that the latter are not invariant to image scale changes. Taking advantage of this invariance, they propose an image descriptor computed as a difference of Gram matrices of convnet features at two different image scales. The authors show the obtained descriptors can be used to compute an image quality measure in a full-reference setup that is in good alignment with human preference. An alternative no-reference image quality measure is also formulated and show cased in experiments for single-image super-resolution and motion debluring with promising results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is simple yet could be interesting, however, understanding the paper is harder than necessary, some choices seem to be ad-hoc and reporting of the results seems to be selective. See the suggestions below.\n\nUPDATE: Thanks to the authors for addressing the concerns raised. After adapting the text as discussed, I would consider the paper to be acceptable.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Possible improvements:\n- Fig. 2 - the caption should be improved, currently it does not give enough details to understand what is plotted, even though this information is in the main text. Consider labeling the x-axis of the plots clearer or mentioning in the caption (a) shows VGG layers and (b) patch sizes. Consider labeling the green bars as \"Same image\" instead of \"Dissimilarity\". And also consider replacing the y-label with \"L_1 distance\".\n- Fig. 2 should include error bars.\n- The definition (2) omits the layer from which the features are used. This makes it harder than necessary to think about the statements like the one on lines 91-92.\n- In Table 1 I would expect to see results computed on the test set, however, the results for LPIPS are different from what was reported before. Make it clear which data is used to compute the results in Table 1.\n- Connected to LPIPS results in Table 1, it is not clear which network was used for it, i.e., is this LPIPS-VGG,  LPIPS-Alex, LPIPS-Squeeze?\n- Why does the Table 1 miss PieAPP results?\n- The main text should be made more self-contained, definitely add details on the training/experiments setup, e.g., which resnet is meant on line 154, what the training setup is to allow for training a resnet with just 400 images, etc.\n- Check that DeepDeblur and DeblurGAN-v2 are not mixed up in Fig. 8 (b).\n\nQuestions:\n- Why is Fig. 2 showing the current selection of vgg layers, missing any layer deeper than 3-1?\n- In Table 2, DSD4 performs good but DSD2 significantly worse, compared to this Table 1 shows values only for DSD2. What was the reasoning to report one or the other scale?\n- Along the same lines as the point above, how is it selected which of the VGG layers is used to compute DSD?\n- Were any other backbones than VGG tested?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QLcWnjieih", "reviewer": "Reviewer_CTwK", "summary": "This paper proposes a new image descriptor, called DSD (Deep Self-Dissimilarities). The authors observe that the very same image may happen to be classified in different ways by deep neural networks when viewed at different scales. For example (Figure 1) an image is classified by the same network as \"Jean\" at full resolution and as \"Constrictor\" at a lower scale. Therefore, to characterize an image, they consider the activations at a given layer of the net for the original image and its resized version, and compute some statistics that describe how they change. More precisely, the descriptor is computed as the difference between the Gram matrices of activations at full and reduced scale. Several experiments are carried out to illustrate the potential of the proposed descriptor.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper moves from some interesting considerations. In fact, the very same network makes different decisions for the same image depending on the image scale, and these variations may be descriptive of the image by themselves. The proposed descriptor seems to be able to capture such variations. However, using deep features to characterize images is hardly new, as the authors themselves acknowledge. So prior work on this topic should be analyzed adequately (just a few lines in the Related Work now) and similarities and differences with other proposal should be clearly pointed out. This would help putting the proposal in a correct perspective with respect to the state of the art. Along this line, it would be interesting to investigate in more depth what these observed dissimilarities depend upon, that is, which image features appear to be invariant/variant across scales.\n\nExperiments on the possible uses of DSD provide mixed support for this proposal. Those concerning its use as full-reference and no-reference quality measures show an impressive performance and maybe should be expanded. The performance observed for image super-resolution and motion deblurring instead is less convincing and not above the state of the art. This is confirmed by the selected visual examples, Figure 3, 7, 9 and 10 where, contrary to what suggested by the comments, hardly any improvement can be noticed with respect to the references.\n\nIn summary, DSD is an interesting descriptor, with good potential, but should be better positioned with respect to the relevant literature and more deeply analyzed and justified.\n\nMinor\n- Eq.(2): I believe some constant may be missing to make the two terms, computed on image of different size, comparable\n- Figure 2: what does rx_y stands for?\n- typos: debluring\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nUPDATES\n\nFirst of all, I want to thank the authors for answering my questions.\nI appreciate the considerations about which features are invariant across image scales.\nThe analysis of the images with the highest and lowest DSD values is very interesting and can be of benefit for the paper. I also better analyzed the visual results for super-resolution and motion deblurring and I can see some improvements,\nhowever I believe that a pointer in the text would definitely help.\nFinally, for what concerns the prior work, I don't think the paper needs additional references or comparisons, but just an expansion of the related work analysis made in Section 2. This would allow a non-expert reader to better position the work with respect to prior art.\n\nOverall, the authors addressed my major concerns and for this reason I improve my final rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not describe limitations and potential negative societal impacts of their work. For what concern limitations it would be interesting to better understand why for the image super-resolution and motion deblurring tasks, the proposed descriptor is not very effective. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a new image descriptor, referred to as deep self dissimilarity (DSD) which measures the dissimilarity between deep-features from the same image presented at different scales. Reviewers agree that the paper presents a novel idea which suggests a  measure of scale-wise dissimilarity of deep features as a descriptive measure of images. The paper is well written and the key idea is appreciated. The rebuttal addressed most of concerns raised by reviewers, leading that two of reviewers raised the score during the discussion period. I believe that the paper is deserved to be presented in the conference. 


Decision: Accept (Spotlight)