Deep Self-Dissimilarities as Powerful Visual Fingerprints

1 Introduction . Features extracted from deep layers of classification networks are widely used as powerful image descriptors . These features are known to capture high level semantics [ 44 ] as well as low-level textural cues [ 14 , 31 ] and are thus exploited in numerous applications , including image enhancement [ 24 , 18 , 10 , 47 ] , synthesis [ 52 , 44 , 5 , 19 ] , and editing [ 15 , 2 , 25 ] , both in the form of per-element similarity measures ( e.g . the perceptual loss [ 15 , 18 ] and LPIPS [ 51 ] ) and for comparing internal image distributions ( e.g . the style loss [ 15 ] , contextual loss [ 32 ] , and projected distribution loss [ 10 ] ) . In this paper , we propose to exploit a surprisingly dominant property of deep features : the dissimilarity between their internal distributions at different image scales . As opposed to small patches in pixel space , which are known to exhibit similar characteristics across scales [ 53 , 16 ] , here we show that deep features tend to vary significantly for different scales of the same image . A glimpse into this phenomenon is provided in Fig . 1 for the VGG-19 network [ 46 ] . As can be seen , the network often outputs completely different classification results for the same image at different resolutions . We show that this behavior is also characteristic of earlier stages within the network , and can therefore serve as a powerful image fingerprint . A naive approach to account for this phenomenon would be to aggregate deep feature descriptors from multiple image scales . For example , to measure discrepancy between two images , one could accumulate deep feature distribution distances , measured separately at different scales , as schematically illustrated in Fig . 5 ( middle ) for the case of two scales . However , here we propose a different approach , which as we show , is far more powerful as a visual fingerprint . Specifically , we present the deep self-dissimilarity ( DSD ) image descriptor , which captures differences between internal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feature distributions at different scales of an image . As we show , this descriptor can serve both as a full reference image fidelity metric ( i.e . by comparing the DSD ’ s of two images ) and as a no-reference measure ( quantifying the “ naturalness ” of an image according to its DSD ) . In both cases , our metrics are highly correlated with human perception , as we verify on user-annotated image quality assessment datasets . Furthermore , we demonstrate the effectiveness of DSD as a loss function for image restoration tasks . Our approach leads to high perceptual quality reconstructions that are at least comparable to GAN-based methods , while completely avoiding adversarial training . 2 Related work . Deep features for similarity measures . Using deep features of a pre-trained classification network for measuring fidelity between images , was first introduced in the context of network inversion and visualization [ 29 , 45 , 50 ] . This idea was quickly adopted as a perceptual similarity in tasks such as texture synthesis [ 14 ] , style transfer [ 15 , 18 ] , super resolution [ 18 , 24 , 24 ] , image inpainting [ 27 ] and more . Non-local ( distribution based ) variants of this per-element perceptual loss were also proposed . These include the style loss [ 15 ] , the contextual loss [ 32 ] , and the projected distribution loss [ 10 ] . These approaches compare between the internal deep-feature distributions of two inputs , at a single resolution . Here , we show that there is a more powerful way to use deep features : comparing their between-scale self-dissimilarities . Internal recurrence and self-similarity Small image patches tend to recur within natural images , both in the same scale [ 8 , 9 , 6 , 12 ] , and across different scales [ 16 ] . Cross-scale recurrence has been used as a prior for various tasks , including super resolution [ 16 , 13 ] , denoising [ 54 , 37 ] , dehazing [ 4 ] , blind deblurring [ 34 , 3 ] , and blind super resolution [ 33 ] . In this work we show that as opposed to image patches , deep features do not exhibit strong similarities across scales , implying that deep self-similarities can not serve as a prior e.g . for restoration tasks . Nonetheless , we show that the deep self dissimilarity pattern of an image is meaningful , and can therefore serve as a powerful fingerprint . 3 The Self-Dissimilarity Between Deep Features at Different Scales . 3.1 Deep self-dissimilarity . We characterize deep feature distributions through their second-order statistics . More concretely , we make use of Gram matrices of the channel activations ( which are proportional to the empirical covariance of the channels ) , [ G ` ( x ) ] i , j = 1 WH ∑ k [ φi ` ( x ) ] k [ φ j ` ( x ) ] k. ( 1 ) Here [ φi ` ( x ) ] k is the activation at spatial position k within the i-th channel of the ` -th layer of a network φ that is fed with the image x at its input . W and H are the width and height of the this map . We define the deep self-dissimilarity ( DSD ) of an image x as the difference between the ` -th Gram matrices corresponding to two image scales , x and x↓α , DSD ` α ( x ) = G ` ( x ) −G ` ( x↓α ) . ( 2 ) In other words , DSD ` α ( x ) is a matrix whose ( i , j ) -th entry measures the extent to which channels i and j of the ` -th layer are more correlated when feeding x to the network , than when feeding x↓α . We often choose the scaling factor α to be 2 or 4 . We sometimes omit the α and ` subscripts for brevity . 3.2 Dissimilarity in RGB space vs. feature space . The strong tendency of small image patches ( in pixel space ) to recur across different scales of an image [ 53 ] is manifested in the similarity between distributions of patches across different image scales . If that was the case also for deep features , then DSD would always equal ( approximately ) the zero matrix , and thus would not constitute a unique fingerprint for images . However , as we show bellow , deep features behave very differently from small image patches in this respect . We perform a comparison between deep feature self-dissimilarities and pixel-space self-dissimilarities for α = 4 . For the latter , we replace the deep feature Gram matrices by pixel-space patch Gram matrices , [ GRGB ( x ) ] i , j = 1 WH ∑ k [ xi ] k [ x j ] k , ( 3 ) where [ xi ] k is the i-th pixel in a column-stacked n × n patch , centered at the k-th pixel of the image x. W and H here correspond to the width and height of the image . For both deep features and pixel-space patches , we depict in Fig . 2 : ( i ) the average ` 1 distance between G ` ( x ) and G ` ( x↓4 ) over random images x sampled from ImageNet [ 42 ] ( green bars ) , ( ii ) the average ` 1 distance between G ` ( x ) and G ` ( y ) over random image pairs ( x , y ) from ImageNet ( blue bars ) , and ( iii ) the average ` 1 distance between G ` ( x ) and G ` ( y ) over random image pairs ( x , y ) sampled from the same class in ImageNet ( red bars ) . Distances are computed for several VGG layers ` and several patch sizes n. As can be seen , the self -dissimilarity values ( green ) computed over image patches and shallow network layers are significantly lower than their cross-image counterparts ( blue and red ) , in accordance with the well known cross-scale patch recurrence phenomenon in natural images . However , when it comes to deeper network layers ( Fig . 2a ) , the average Gram dissimilarities between different scales of the same image are roughly as large as their cross-image counterparts , suggesting that deep features do not exhibit self similarity like pixel-space patches . This conclusion is also supported by Fig . 6 . Here , we optimized the RGB values of an image x so as to minimize ‖DSD ` 4 ( x ) ‖1 for layer ` = ReLU31 . As can be seen in the right image , too many ( 5K ) gradient descent steps result in the image exhibiting unnatural artifacts . This illustrates again that natural images are in fact not characterized by small DSD values . 3.3 Visualizing DSD . What does the DSD fingerprint capture , then ? We answer this question through several visualizations . Fig . 3 presents the images from the ImageNet validations set , which have the lowest and the highest DSD values layer at ` = ReLU31 . As can be seen , images with higher DSD values ( bottom row ) typically contain finer details and textures , compared to low DSD value images ( top row ) . This makes sense , as computing DSD involves downsampling the image , which results in erasing fine image details . This significantly changes the Gram matrix computed over deep features corresponding to the downsampled image , which in turn increases the difference between the two Gram matrices , computed in ( 2 ) . Examining the statistics of DSD values across different classes in the ImageNet dataset reveals similar behaviour . For example the class “ sandbar ” has an average DSD of 2.1 · 10−3 ± 0.83 · 10−3 , while the average DSD of the class “ window screen ” is 13.5 · 10−3 ± 11.0 · 10−3 . Next , we visualize the effect of modifying a specific content image xc such that its DSD fingerprint is as similar as possible to that of a reference image xr . We do this once by minimizing ‖DSD2 ( xc ) − DSD2 ( xr ) ‖1 and once by minimizing ‖DSD2 ( xc ↓2 ) − DSD2 ( xr ↓2 ) ‖1 . Namely , we try to match the images ’ deep self-dissimilarities between scales 1 and 0.5 , and between scales 0.5 and 0.25 . Here DSD2 is calculated by aggregating the DSD matrices corresponding to VGG layers ` ∈ { ReLU21 , ReLU22 , ReLU31 } . For comparison , we perform the same experiment , but with the style losses ‖G ( xc ) − G ( xr ) ‖1 , ‖G ( xc ↓2 ) − G ( xr ↓2 ) ‖1 , and ‖G ( xc ↓4 ) − G ( xr ↓4 ) ‖1 . The latter experiment can be thought of as a style pyramid , whereas the former as a style-differences pyramid ( in analogy to the Gaussian and Laplacian pyramids ) . The results of the two experiments are presented in Fig . 4 . As can be seen , while style dissimilarity ( upper row ) can be minimized by merely augmenting vague patterns from the reference image , matching the DSD fingerprints ( bottom row ) requires adding sharper , more detailed visual structures . This suggests that the DSD loss is a more intricate fingerprint , and can serve as a better loss for image quality assessment and for image restoration .

Summary: This paper proposes the use of "deep self-dissimilarities" (DSD) - meaning differences in feature activations at deeper layers of a network when computed on an image versus on a downsampled version of that image.  DSD is applied to several tasks - serving as a measure for predicting human preference of image quality/distortion, used as a loss-term for single image super-resolution, and as a loss term for motion deblurring.  In all cases, the proposed method performs well relative to existing state-of-the-art.