Deep Self-Dissimilarities as Powerful Visual Fingerprints
Year: 2021

Abstract:
Features extracted from deep layers of classification networks are widely used as image descriptors. Here, we exploit an unexplored property of these features: their internal dissimilarity. While small image patches are known to have similar statistics across image scales, it turns out that the internal distribution of deep features varies distinctively between scales. We show how this deep self dissimilarity (DSD) property can be used as a powerful visual fingerprint. Particularly, we illustrate that full-reference and no-reference image quality measures derived from DSD are highly correlated with human preference. In addition, incorporating DSD as a loss function in training of image restoration networks, leads to results that are at least as photo-realistic as those obtained by GAN based methods, while not requiring adversarial training.

heading: 1 Introduction
text: Features extracted from deep layers of classification networks are widely used as powerful image descriptors. These features are known to capture high level semantics [44] as well as low-level textural cues [14, 31] and are thus exploited in numerous applications, including image enhancement [24, 18, 10, 47], synthesis [52, 44, 5, 19], and editing [15, 2, 25], both in the form of per-element similarity measures (e.g. the perceptual loss [15, 18] and LPIPS [51]) and for comparing internal image distributions (e.g. the style loss [15], contextual loss [32], and projected distribution loss [10]).
In this paper, we propose to exploit a surprisingly dominant property of deep features: the dissimilarity between their internal distributions at different image scales. As opposed to small patches in pixel space, which are known to exhibit similar characteristics across scales [53, 16], here we show that deep features tend to vary significantly for different scales of the same image. A glimpse into this phenomenon is provided in Fig. 1 for the VGG-19 network [46]. As can be seen, the network often outputs completely different classification results for the same image at different resolutions. We show that this behavior is also characteristic of earlier stages within the network, and can therefore serve as a powerful image fingerprint.
A naive approach to account for this phenomenon would be to aggregate deep feature descriptors from multiple image scales. For example, to measure discrepancy between two images, one could accumulate deep feature distribution distances, measured separately at different scales, as schematically illustrated in Fig. 5 (middle) for the case of two scales. However, here we propose a different approach, which as we show, is far more powerful as a visual fingerprint. Specifically, we present the deep self-dissimilarity (DSD) image descriptor, which captures differences between internal
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
feature distributions at different scales of an image. As we show, this descriptor can serve both as a full reference image fidelity metric (i.e. by comparing the DSD’s of two images) and as a no-reference measure (quantifying the “naturalness” of an image according to its DSD). In both cases, our metrics are highly correlated with human perception, as we verify on user-annotated image quality assessment datasets. Furthermore, we demonstrate the effectiveness of DSD as a loss function for image restoration tasks. Our approach leads to high perceptual quality reconstructions that are at least comparable to GAN-based methods, while completely avoiding adversarial training.

heading: 2 Related work
text: Deep features for similarity measures. Using deep features of a pre-trained classification network for measuring fidelity between images, was first introduced in the context of network inversion and visualization [29, 45, 50]. This idea was quickly adopted as a perceptual similarity in tasks such as texture synthesis [14], style transfer [15, 18], super resolution [18, 24, 24], image inpainting [27] and more. Non-local (distribution based) variants of this per-element perceptual loss were also proposed. These include the style loss [15], the contextual loss [32], and the projected distribution loss [10]. These approaches compare between the internal deep-feature distributions of two inputs, at a single resolution. Here, we show that there is a more powerful way to use deep features: comparing their between-scale self-dissimilarities.
Internal recurrence and self-similarity Small image patches tend to recur within natural images, both in the same scale [8, 9, 6, 12], and across different scales [16]. Cross-scale recurrence has been used as a prior for various tasks, including super resolution [16, 13], denoising [54, 37], dehazing [4], blind deblurring [34, 3], and blind super resolution [33]. In this work we show that as opposed to image patches, deep features do not exhibit strong similarities across scales, implying that deep self-similarities cannot serve as a prior e.g. for restoration tasks. Nonetheless, we show that the deep self dissimilarity pattern of an image is meaningful, and can therefore serve as a powerful fingerprint.

heading: 3 The Self-Dissimilarity Between Deep Features at Different Scales

heading: 3.1 Deep self-dissimilarity
text: We characterize deep feature distributions through their second-order statistics. More concretely, we make use of Gram matrices of the channel activations (which are proportional to the empirical covariance of the channels),
[G`(x)]i,j = 1
WH ∑ k [φi`(x)]k [φ j `(x)]k. (1)
Here [φi`(x)]k is the activation at spatial position k within the i-th channel of the `-th layer of a network φ that is fed with the image x at its input. W and H are the width and height of the this map.
We define the deep self-dissimilarity (DSD) of an image x as the difference between the `-th Gram matrices corresponding to two image scales, x and x↓α,
DSD`α(x) = G`(x)−G`(x↓α). (2)
In other words, DSD`α(x) is a matrix whose (i, j)-th entry measures the extent to which channels i and j of the `-th layer are more correlated when feeding x to the network, than when feeding x↓α. We often choose the scaling factor α to be 2 or 4. We sometimes omit the α and ` subscripts for brevity.

heading: 3.2 Dissimilarity in RGB space vs. feature space
text: The strong tendency of small image patches (in pixel space) to recur across different scales of an image [53] is manifested in the similarity between distributions of patches across different image scales. If that was the case also for deep features, then DSD would always equal (approximately) the zero matrix, and thus would not constitute a unique fingerprint for images. However, as we show bellow, deep features behave very differently from small image patches in this respect.
We perform a comparison between deep feature self-dissimilarities and pixel-space self-dissimilarities for α = 4. For the latter, we replace the deep feature Gram matrices by pixel-space patch Gram matrices,
[GRGB(x)]i,j = 1
WH ∑ k [xi]k [x j ]k, (3)
where [xi]k is the i-th pixel in a column-stacked n × n patch, centered at the k-th pixel of the image x. W and H here correspond to the width and height of the image. For both deep features and pixel-space patches, we depict in Fig. 2: (i) the average `1 distance between G`(x) and G`(x↓4) over random images x sampled from ImageNet [42] (green bars), (ii) the average `1 distance between G`(x) and G`(y) over random image pairs (x, y) from ImageNet (blue bars), and (iii) the average `1 distance between G`(x) and G`(y) over random image pairs (x, y) sampled from the same class in ImageNet (red bars). Distances are computed for several VGG layers ` and several patch sizes n.
As can be seen, the self -dissimilarity values (green) computed over image patches and shallow network layers are significantly lower than their cross-image counterparts (blue and red), in accordance with the well known cross-scale patch recurrence phenomenon in natural images. However, when it
comes to deeper network layers (Fig. 2a), the average Gram dissimilarities between different scales of the same image are roughly as large as their cross-image counterparts, suggesting that deep features do not exhibit self similarity like pixel-space patches.
This conclusion is also supported by Fig. 6. Here, we optimized the RGB values of an image x so as to minimize ‖DSD`4(x)‖1 for layer ` = ReLU31. As can be seen in the right image, too many (5K) gradient descent steps result in the image exhibiting unnatural artifacts. This illustrates again that natural images are in fact not characterized by small DSD values.

heading: 3.3 Visualizing DSD
text: What does the DSD fingerprint capture, then? We answer this question through several visualizations. Fig. 3 presents the images from the ImageNet validations set, which have the lowest and the highest DSD values layer at ` = ReLU31. As can be seen, images with higher DSD values (bottom row) typically contain finer details and textures, compared to low DSD value images (top row). This makes sense, as computing DSD involves downsampling the image, which results in erasing fine image details. This significantly changes the Gram matrix computed over deep features corresponding to the downsampled image, which in turn increases the difference between the two Gram matrices, computed in (2). Examining the statistics of DSD values across different classes in the ImageNet dataset reveals similar behaviour. For example the class “sandbar” has an average DSD of 2.1 · 10−3 ± 0.83 · 10−3, while the average DSD of the class “window screen” is 13.5 · 10−3 ± 11.0 · 10−3. Next, we visualize the effect of modifying a specific content image xc such that its DSD fingerprint is as similar as possible to that of a reference image xr. We do this once by minimizing ‖DSD2(xc)− DSD2(xr)‖1 and once by minimizing ‖DSD2(xc ↓2) − DSD2(xr ↓2)‖1. Namely, we try to match the images’ deep self-dissimilarities between scales 1 and 0.5, and between scales 0.5 and 0.25. Here DSD2 is calculated by aggregating the DSD matrices corresponding to VGG layers ` ∈ {ReLU21,ReLU22,ReLU31}. For comparison, we perform the same experiment, but with the style losses ‖G(xc) − G(xr)‖1, ‖G(xc ↓2) − G(xr ↓2)‖1, and ‖G(xc ↓4) − G(xr ↓4)‖1. The latter experiment can be thought of as a style pyramid, whereas the former as a style-differences pyramid (in analogy to the Gaussian and Laplacian pyramids). The results of the two experiments are presented in Fig. 4. As can be seen, while style dissimilarity (upper row) can be minimized by merely augmenting vague patterns from the reference image, matching the DSD fingerprints (bottom row) requires adding sharper, more detailed visual structures. This suggests that the DSD loss is a more intricate fingerprint, and can serve as a better loss for image quality assessment and for image restoration.

heading: 3.4 DSD as a full-reference measure
text: The DSD can be used to define a full-reference distortion measure simply by computing the `1 distance between the DSD’s of the two images we wish to compare,
LDSDα(x, y) = ‖DSDα(x)− DSDα(y)‖1. (4) This distance quantifies the extent to which the differences between the deep feature distributions at different scales of x are similar to those of y (see Fig. 5, right pane).
We evaluate this measure in a perceptual image quality assessment task. We use the PieAPP dataset [40], which contains 4800 pairs of images, where one image is the reference and the other is a distorted version of that reference. The dataset includes a variety of distortions, along with scores for their distortion level obtained from human ranking. We compare LDSD with several other full-reference measures that are based on deep network features: the perceptual loss ‖φ(x)− φ(y)‖1, the style loss ‖G(x)−G(y)‖1, and the projected distribution loss (PDL) [10]. To isolate the effect of using multiple image scales in DSD, we also add a comparison to a multi-scale variant of the style loss, ‖G(x)−G(y)‖1 + ‖G(x↓2)−G(y↓2)‖1 (see schematic illustration in Fig. 5). Note that none of these metrics were specifically designed for (or even fine-tuned on) the PieAPP dataset.
We measured the correlation between each metric and the human preferences. Specifically, for every two distorted versions xa, xb of the same reference image xr, the PieAPP dataset [40] provides the probability that a human rater would prefer version xa over version xb. We calculated the Pearson correlation between these probabilities and the loss differences L(xb, xr)−L(xa, xr), for each of the full-reference losses L(·, ·) we examined1. Table 1 presents the results for several different layers ` of the VGG network. Correlation with the LPIPS measure [51], which uses a different network, is also reported. Note that DSD values computed using the deeper network layers (two right columns) have the highest correlation with human evaluation scores, compared to all other measures. This illustrates that the DSD fingerprint can be used as an effective perceptual distortion measure.

heading: 3.5 DSD as a no-reference quality measure
text: The DSD fingerprint can also serve to define a no-reference image quality measure. Let us revisit Fig. 6, in which we optimized an image x so as to minimize ‖DSD(x)‖1. Here, the original image is a super-resolution result produced by the ESRGAN method [48]. As already mentioned, after 5K gradient descent steps the optimized image contains unnatural artifacts. However, note that an interesting phenomenon occurs if we stop the optimization after only 500 steps. In that case, the optimized image is sharper and more naturally looking than the original ESRGAN result. This
1The Pearson correlation measures the extent to which the loss difference is a monotonic function of the preference probability.
suggests that for this particular scene, there is a specific optimal ‖DSD(x)‖1 value, which is in between that of the original image, and the 5K minimization result.
Following this observation, we propose to use DSD for computing a no-reference image quality score. To asses the quality of a potentially degraded image y, we would like to measure the difference between ‖DSDα(y)‖1 and ‖DSDα(x)‖1, where x is the degradation-free version of y. Obviously, in the no-reference case we do not have access to x. One could conceive replacing ‖DSDα(x)‖1 by its average value for natural images. The problem with this approach is that different natural images have very different ‖DSDα(x)‖1 values, as evident from the histogram in Fig. 7. A different approach would be to train a regression model to predict ‖DSDα(x)‖1 from y. However, we do not want to restrict our measure to specific degradations, and thus would like to avoid using datasets of degraded images for training. Our solution is to train a light-weight regression model ψ to predict ‖DSDα(x)‖1 from the downsampled clean input, x ↓α. At test time, we input to our model the downsampled degraded image, y↓α, and use the model’s output as our estimate for ‖DSDα(x)‖1. The reasoning behind this choice, is that for many types of degradations y↓α and x↓α are quite similar (e.g. noise, blur, compression artifacts, etc.). We train the regression network ψ (we use a convolutional network with 12 residual blocks [17]) on the BSD training set [30] containing 400 clean images, using the mean square error loss. Our no-reference measure is therefore given by
QDSDα(y) = |ψ(y↓α)− ‖DSDα(y)‖1 | . (5)
We compare our proposed measure to the leading existing no-reference image quality measures, including the naturalness image quality evaluator (NIQE) [36], the blind/referenceless image spatial quality evaluator (BRISQUE) [35] and the score proposed by Ma et al. [28]. Similarly to the fullreference case, we compute the Pearson correlation between the human evaluation scores from [40] and the evaluated scores, and present the results in Tab. 2. DSD4 significantly outperforms the competitors in terms of correlation with human assessment.

heading: 4 Experiments
text: We use the DSD fingerprint for the tasks of single image super-resolution (SR) and motion debluring. In both cases our goal is to recover a clean image x from its degraded observation y. To this end, we train a restoration network on pairs of training examples {x, y} by minimizing the DSD-based full reference loss term (4) between the restored image x̂ and its ground-truth counterpart x. We combine our loss with two other popular loss terms, as common in restoration methods (e.g. [24, 48]),
L(x, x̂) = Lper(x, x̂) + λrec · Lrec(x, x̂) + λDSD · LDSD(x, x̂). (6) Here Lrec(x, x̂) = ‖x − x̂‖1, Lper = ‖φ`(x̂) − φ`(x)‖1 is the perceptual loss computed over layer Conv54 of VGG, and λrec, λDSD are weighting coefficients. For LDSD we use layers ` ∈ {ReLU21,ReLU22,ReLU31} (as in Sec. 3.3) and α = 2. We train our networks for 300K epochs using the Adam optimizer with a batch size of 16 and an initial learning-rate of 2 · 10−4, which is halved after 90K, 180K and 270K steps. See Supplementary Material (SM) for full training details.

heading: 4.1 Single Image Super Resolution
text: Here our goal is to predict a high-resolution (HR) image x from its low-resolution (LR) version y. We focus on 4× SR, where the LR image corresponds to bicubic downsampling of the HR image. Training is done over 800 LR-HR image pairs from the DIVK2K dataset [1]. Our SR network utilizes the xSRResNet architecture [22] consisting of 10 residual blocks.
Figure 8 presents a qualitative comparison with ESRGAN [48] and SANGAN [21], two state-of-theart GAN-based methods that target perceptual quality. Note that although our method does not require any adversarial training, it manages to restore finer image details, and to produce more realistic textures. Please refer to the SM for many additional visual examples.
We next follow [7] and quantitatively evaluate the performance of our method by reporting perceptual quality (calculated using NIQE [36], lower is better) and image distortion (using SSIM [49], higher is better). In Fig. 9a we compare our model (red dot) with the leading SR methods EDSR [26], VDSR [20], SRResNet [24], xSRResNet [22], Deng [11], ESRGAN [48], SRGAN [24], ENET [43] and SinGAN [41] (black dots). We also compare variants of our model, trained by replacingLDSD with other deep internal-distribution based losses (blue dots): the projected distribution loss (PDL) [10] and the style loss (STY) [15]. Finally, we present results for our model trained using only Lrec and Lper, or only Lrec (indicated by “per” and empty subscripts respectively). All scores are calculated over the BSD100 test set [30]. As can be seen, our method is among the best in terms of perceptual quality, while maintaining relatively low distortion. Note that achieving such perceptual quality has been possible to date only with GAN based methods. Our method achieves such performance without the need of adversarial training, which is known to be unstable and challenging in practice.
To further investigate the contribution of the DSD loss, we perform quantitative evaluation of our method when LDSD is replaced by other loss terms that measure distances between deep feature distributions, including the contextual loss (CX) [32], projected distribution loss (PDL) [10], style loss [15] and the multi-scale style loss (MSS) defined in Sec. 3.4. For each of these loss terms, we adjust the value of λDSD so as to make the perceptual and feature distribution loss terms equal on average. The results are presented in Tab. 3, which reports PSNR, LPIPS [51] and NIQE [36]. In each experiment we calculate all losses using the same VGG layers (indicated at the top of each column), as well as the same training procedure and network architecture (here we use a shallow xSRResNet [22] with 5 residual blocks). In terms of perceptual quality (NIQE and LPIPS), the model
trained with our DSD loss (bottom row) is among the best for all examined activation layers. This is also supported by Fig. 10, which shows a visual comparison between the methods. Our result contains less grid artifacts compared to the other losses (exemplified here for layer ReLU22).

heading: 4.2 Single Image Motion Deblurring
text: Here we aim to recover a sharp, blur free image x from a blurry image y. We use the same loss function, training protocol and architecture (xSRResNet [21] with 10 res-blocks) as in our SR experiments. Training is done using the REDS dataset [39] consisting of 30, 000 image pairs {x, y} from 300 different scenes. Figure 11 presents an example result of our motion deblurring, compared with the state-of-the-art DeblurGAN-v2 [23] and a variant of our method trained with the style loss [15] instead of LDSD. The result using the DSD loss is sharper and contains no ghosting artifacts. Please see many more results in the SM.
This observation is further supported quantitatively. In Fig. 9b we report perceptual quality and distortion (using the same measures as in Fig. 9a) over 100 random images from the REDS validation set. We compare our method (red dot) with the state-of-the-art DeblurGAN-v2 and DeepDeblur2 [38] (black dots), as well as with three variants of our method (blue dots) trained by replacing LDSD with either the style loss (STY) [15], the multi-scale style loss (MSS) (from Sec. 3.4), and when omitting LDSD and keeping only Lrec and Lper (denoted by a subscript “per”). Here as well, the perception-distortion plot indicates that our DSD-based model obtains the best perceptual quality (significantly better than most of the competition), while exhibiting low distortion.
2An official newer version available at: https://github.com/SeungjunNah/DeepDeblur-PyTorch

heading: 5 Conclusion
text: Deep features corresponding to different image scales exhibit meaningful dissimilarity. We prove this to be a powerful image fingerprint, highly correlated with human preference in both full-reference and no-reference image quality assessment, and leading to a GAN-like highly photo-realistic image restoration (while avoiding unstable adversarial training). We believe that deep self-dissimilarity can benefit additional image restoration and manipulation tasks.
Acknowledgments This research was supported by the Israel Science Foundation (grant 852/17) and by the Technion Ollendorff Minerva Center.

Review summaries:
This paper proposes a new image descriptor, called DSD (Deep Self-Dissimilarities). The authors observe that the very same image may happen to be classified in different ways by deep neural networks when viewed at different scales. For example (Figure 1) an image is classified by the same network as "Jean" at full resolution and as "Constrictor" at a lower scale. Therefore, to characterize an image, they consider the activations at a given layer of the net for the original image and its resized version, and compute some statistics that describe how they change. More precisely, the descriptor is computed as the difference between the Gram matrices of activations at full and reduced scale. Several experiments are carried out to illustrate the potential of the proposed descriptor. 

The authors observe that Gram matrices of image patches and of features extracted from convolutional networks behave differently. Namely that the latter are not invariant to image scale changes. Taking advantage of this invariance, they propose an image descriptor computed as a difference of Gram matrices of convnet features at two different image scales. The authors show the obtained descriptors can be used to compute an image quality measure in a full-reference setup that is in good alignment with human preference. An alternative no-reference image quality measure is also formulated and show cased in experiments for single-image super-resolution and motion debluring with promising results.

This paper proposes the use of "deep self-dissimilarities" (DSD) - meaning differences in feature activations at deeper layers of a network when computed on an image versus on a downsampled version of that image.  DSD is applied to several tasks - serving as a measure for predicting human preference of image quality/distortion, used as a loss-term for single image super-resolution, and as a loss term for motion deblurring.  In all cases, the proposed method performs well relative to existing state-of-the-art.