Deep Self-Dissimilarities as Powerful Visual Fingerprints

1 Introduction . Features extracted from deep layers of classification networks are widely used as powerful image descriptors . These features are known to capture high level semantics [ 44 ] as well as low-level textural cues [ 14 , 31 ] and are thus exploited in numerous applications , including image enhancement [ 24 , 18 , 10 , 47 ] , synthesis [ 52 , 44 , 5 , 19 ] , and editing [ 15 , 2 , 25 ] , both in the form of per-element similarity measures ( e.g . the perceptual loss [ 15 , 18 ] and LPIPS [ 51 ] ) and for comparing internal image distributions ( e.g . the style loss [ 15 ] , contextual loss [ 32 ] , and projected distribution loss [ 10 ] ) . In this paper , we propose to exploit a surprisingly dominant property of deep features : the dissimilarity between their internal distributions at different image scales . As opposed to small patches in pixel space , which are known to exhibit similar characteristics across scales [ 53 , 16 ] , here we show that deep features tend to vary significantly for different scales of the same image . A glimpse into this phenomenon is provided in Fig . 1 for the VGG-19 network [ 46 ] . As can be seen , the network often outputs completely different classification results for the same image at different resolutions . We show that this behavior is also characteristic of earlier stages within the network , and can therefore serve as a powerful image fingerprint . A naive approach to account for this phenomenon would be to aggregate deep feature descriptors from multiple image scales . For example , to measure discrepancy between two images , one could accumulate deep feature distribution distances , measured separately at different scales , as schematically illustrated in Fig . 5 ( middle ) for the case of two scales . However , here we propose a different approach , which as we show , is far more powerful as a visual fingerprint . Specifically , we present the deep self-dissimilarity ( DSD ) image descriptor , which captures differences between internal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . feature distributions at different scales of an image . As we show , this descriptor can serve both as a full reference image fidelity metric ( i.e . by comparing the DSD ’ s of two images ) and as a no-reference measure ( quantifying the “ naturalness ” of an image according to its DSD ) . In both cases , our metrics are highly correlated with human perception , as we verify on user-annotated image quality assessment datasets . Furthermore , we demonstrate the effectiveness of DSD as a loss function for image restoration tasks . Our approach leads to high perceptual quality reconstructions that are at least comparable to GAN-based methods , while completely avoiding adversarial training . 2 Related work . Deep features for similarity measures . Using deep features of a pre-trained classification network for measuring fidelity between images , was first introduced in the context of network inversion and visualization [ 29 , 45 , 50 ] . This idea was quickly adopted as a perceptual similarity in tasks such as texture synthesis [ 14 ] , style transfer [ 15 , 18 ] , super resolution [ 18 , 24 , 24 ] , image inpainting [ 27 ] and more . Non-local ( distribution based ) variants of this per-element perceptual loss were also proposed . These include the style loss [ 15 ] , the contextual loss [ 32 ] , and the projected distribution loss [ 10 ] . These approaches compare between the internal deep-feature distributions of two inputs , at a single resolution . Here , we show that there is a more powerful way to use deep features : comparing their between-scale self-dissimilarities . Internal recurrence and self-similarity Small image patches tend to recur within natural images , both in the same scale [ 8 , 9 , 6 , 12 ] , and across different scales [ 16 ] . Cross-scale recurrence has been used as a prior for various tasks , including super resolution [ 16 , 13 ] , denoising [ 54 , 37 ] , dehazing [ 4 ] , blind deblurring [ 34 , 3 ] , and blind super resolution [ 33 ] . In this work we show that as opposed to image patches , deep features do not exhibit strong similarities across scales , implying that deep self-similarities can not serve as a prior e.g . for restoration tasks . Nonetheless , we show that the deep self dissimilarity pattern of an image is meaningful , and can therefore serve as a powerful fingerprint . 3 The Self-Dissimilarity Between Deep Features at Different Scales . 3.1 Deep self-dissimilarity . We characterize deep feature distributions through their second-order statistics . More concretely , we make use of Gram matrices of the channel activations ( which are proportional to the empirical covariance of the channels ) , [ G ` ( x ) ] i , j = 1 WH ∑ k [ φi ` ( x ) ] k [ φ j ` ( x ) ] k. ( 1 ) Here [ φi ` ( x ) ] k is the activation at spatial position k within the i-th channel of the ` -th layer of a network φ that is fed with the image x at its input . W and H are the width and height of the this map . We define the deep self-dissimilarity ( DSD ) of an image x as the difference between the ` -th Gram matrices corresponding to two image scales , x and x↓α , DSD ` α ( x ) = G ` ( x ) −G ` ( x↓α ) . ( 2 ) In other words , DSD ` α ( x ) is a matrix whose ( i , j ) -th entry measures the extent to which channels i and j of the ` -th layer are more correlated when feeding x to the network , than when feeding x↓α . We often choose the scaling factor α to be 2 or 4 . We sometimes omit the α and ` subscripts for brevity . 3.2 Dissimilarity in RGB space vs. feature space . The strong tendency of small image patches ( in pixel space ) to recur across different scales of an image [ 53 ] is manifested in the similarity between distributions of patches across different image scales . If that was the case also for deep features , then DSD would always equal ( approximately ) the zero matrix , and thus would not constitute a unique fingerprint for images . However , as we show bellow , deep features behave very differently from small image patches in this respect . We perform a comparison between deep feature self-dissimilarities and pixel-space self-dissimilarities for α = 4 . For the latter , we replace the deep feature Gram matrices by pixel-space patch Gram matrices , [ GRGB ( x ) ] i , j = 1 WH ∑ k [ xi ] k [ x j ] k , ( 3 ) where [ xi ] k is the i-th pixel in a column-stacked n × n patch , centered at the k-th pixel of the image x. W and H here correspond to the width and height of the image . For both deep features and pixel-space patches , we depict in Fig . 2 : ( i ) the average ` 1 distance between G ` ( x ) and G ` ( x↓4 ) over random images x sampled from ImageNet [ 42 ] ( green bars ) , ( ii ) the average ` 1 distance between G ` ( x ) and G ` ( y ) over random image pairs ( x , y ) from ImageNet ( blue bars ) , and ( iii ) the average ` 1 distance between G ` ( x ) and G ` ( y ) over random image pairs ( x , y ) sampled from the same class in ImageNet ( red bars ) . Distances are computed for several VGG layers ` and several patch sizes n. As can be seen , the self -dissimilarity values ( green ) computed over image patches and shallow network layers are significantly lower than their cross-image counterparts ( blue and red ) , in accordance with the well known cross-scale patch recurrence phenomenon in natural images . However , when it comes to deeper network layers ( Fig . 2a ) , the average Gram dissimilarities between different scales of the same image are roughly as large as their cross-image counterparts , suggesting that deep features do not exhibit self similarity like pixel-space patches . This conclusion is also supported by Fig . 6 . Here , we optimized the RGB values of an image x so as to minimize ‖DSD ` 4 ( x ) ‖1 for layer ` = ReLU31 . As can be seen in the right image , too many ( 5K ) gradient descent steps result in the image exhibiting unnatural artifacts . This illustrates again that natural images are in fact not characterized by small DSD values . 3.3 Visualizing DSD . What does the DSD fingerprint capture , then ? We answer this question through several visualizations . Fig . 3 presents the images from the ImageNet validations set , which have the lowest and the highest DSD values layer at ` = ReLU31 . As can be seen , images with higher DSD values ( bottom row ) typically contain finer details and textures , compared to low DSD value images ( top row ) . This makes sense , as computing DSD involves downsampling the image , which results in erasing fine image details . This significantly changes the Gram matrix computed over deep features corresponding to the downsampled image , which in turn increases the difference between the two Gram matrices , computed in ( 2 ) . Examining the statistics of DSD values across different classes in the ImageNet dataset reveals similar behaviour . For example the class “ sandbar ” has an average DSD of 2.1 · 10−3 ± 0.83 · 10−3 , while the average DSD of the class “ window screen ” is 13.5 · 10−3 ± 11.0 · 10−3 . Next , we visualize the effect of modifying a specific content image xc such that its DSD fingerprint is as similar as possible to that of a reference image xr . We do this once by minimizing ‖DSD2 ( xc ) − DSD2 ( xr ) ‖1 and once by minimizing ‖DSD2 ( xc ↓2 ) − DSD2 ( xr ↓2 ) ‖1 . Namely , we try to match the images ’ deep self-dissimilarities between scales 1 and 0.5 , and between scales 0.5 and 0.25 . Here DSD2 is calculated by aggregating the DSD matrices corresponding to VGG layers ` ∈ { ReLU21 , ReLU22 , ReLU31 } . For comparison , we perform the same experiment , but with the style losses ‖G ( xc ) − G ( xr ) ‖1 , ‖G ( xc ↓2 ) − G ( xr ↓2 ) ‖1 , and ‖G ( xc ↓4 ) − G ( xr ↓4 ) ‖1 . The latter experiment can be thought of as a style pyramid , whereas the former as a style-differences pyramid ( in analogy to the Gaussian and Laplacian pyramids ) . The results of the two experiments are presented in Fig . 4 . As can be seen , while style dissimilarity ( upper row ) can be minimized by merely augmenting vague patterns from the reference image , matching the DSD fingerprints ( bottom row ) requires adding sharper , more detailed visual structures . This suggests that the DSD loss is a more intricate fingerprint , and can serve as a better loss for image quality assessment and for image restoration .

Summary: The authors observe that Gram matrices of image patches and of features extracted from convolutional networks behave differently. Namely that the latter are not invariant to image scale changes. Taking advantage of this invariance, they propose an image descriptor computed as a difference of Gram matrices of convnet features at two different image scales. The authors show the obtained descriptors can be used to compute an image quality measure in a full-reference setup that is in good alignment with human preference. An alternative no-reference image quality measure is also formulated and show cased in experiments for single-image super-resolution and motion debluring with promising results.