On the relation between statistical learning and perceptual distances

1 INTRODUCTION . The relationship between the internal representations of supervised learning models and biological systems has previously been explored ( Cadieu et al. , 2014 ) , and this connection can be explained by both systems being optimized to perform the same object recognition task . A much less studied area is comparing modern representations learned in an unsupervised manner to those of biological systems . As one of the most influential ideas in this area , the efficient coding hypothesis states that the internal representations of the brain have evolved to efficiently represent the stimuli ( Attneave , 1954 ; Barlow , 1961 ) and has been validated against statistical models for images ( Simoncelli & Olshausen , 2001 ; Malo & Laparra , 2010 ) . Similarly , the explicit constraints of dimensionality reduction or compression present in many unsupervised representation learning models , mainly autoencoders ( Ballé et al. , 2018 ; 2016 ; Baldi , 2012 ) , impose a type of parsimony on the representation . To understand properties of these representations of interest , here we consider distance measurements between the representations of pairs of stimuli . Such distance measurements can be thought of as perceptual distances , which are rooted in psychophysics : a good perceptual distance mimics the human rating of similarity between the two stimuli with high accuracy . Traditionally , perceptual distances have been hand-designed models with few adjustable parameters , inspired by the physiology or observations in visual psychology , as the Multi-Scale Structural SIMilarity index ( MS-SSIM ) ( Wang et al. , 2003 ) . More recently , it has become common to use an explicit image representation and ’ induce ’ a distance from it . For instance , comparing the internal representations of models trained for image classification for pairs of stimuli has been used for perceptual judgments and was shown to correlate well with human opinions ( Zhang et al. , 2018 ; Ding et al. , 2020 ) . This is also true for unsupervised representations that focus on learning features of natural scenes which are information efficient . For example , in the normalized Laplacian pyramid distance ( NLPD ) ( Laparra et al. , 2016 ) , the representation is learned based on redundancy reduction in neighboring pixels . The Perceptual Information Metric ( PIM ) ( Bhardwaj et al. , 2020 ) uses a contrastive representation learning technique based on compression and slowness . With regards to training autoencoders , a particular type of model that can be used to unsupervisedly learn an explicit representation , here we examine three distinct types of induced distances : the reconstruction distance Dr , the inner distance Din and the self-distance Ds . These distances correspond to different representations learned by the autoencoder ( see Fig . 1 ) . While the connection between the biological response and image probability has been examined before ( Laughlin , 1981 ; Twer & MacLeod , 2001 ; Malo & Gutiérrez , 2006 ; Malo & Laparra , 2010 ; Laparra et al. , 2012 ; Laparra & Malo , 2015 ; Hyvärinen et al. , 2009 ) , the relation between perceptual distances , unsupervised image representations , and the statistics of natural images has not been studied in depth . The current understanding is simply that distances induced by representations relevant for image classification or compression are useful for perceptual judgments . We show that the relation is deeper than that , linking it to image statistics . Furthermore , we examine the unexpected effects of utilizing perceptual distances in the loss functions of autoencoders , which is a common approach in designing neural image compression methods ( Ballé et al. , 2018 ) . One would expect that using a perceptual distance that is closely related to the internal image representation of the brain would give rise to a representation inside the model that is much more closely tied to humans , but that does not seem to be the case . This is surprising given the limited ability of Euclidean distances like Mean Squared Error ( MSE ) to reproduce human opinion ( Girod , 1993 ; Wang & Bovik , 2009 ) , compared to successful perceptual distances . We argue that this is because of a double counting effect where the distribution of natural images is taken into account twice in the training of the autoencoder ; once in the training data and again in the perceptual distance , leading to an over-stressing of high density regions in the data . Conversely , where data is sparse or contains non-representative samples , this effect can result in regularization by discounting losses from outliers . Our specific contributions are : 1 We demonstrate that good perceptual distances , i.e . distances that are good at predicting human psychophysical responses , are also correlated with image likelihoods obtained using a recent probabilistic image model ( PixelCNN++ ( Salimans et al. , 2017 ) ) . This underlines indirectly the idea that part of the biology is informed by efficient representation , as conjectured by Barlow . 2 The distances induced by autoencoders trained to minimize an Euclidean loss are correlated with the probability of the training data . Moreover , when autoencoders are trained using natural images , these induced distances are highly correlated with human perception . 3 Using a perceptual distance instead of a Euclidean loss in the optimization of autoencoders implies taking into account the data distribution twice : one in the perceptual distance and another through the empirical risk minimization procedure . We call this the double counting effect . This effect may explain the limited improvements obtained when using perceptual distances in some machine learning applications . We find that perceptual distances lead to models that over-stress high density regions . We emphasize this by showing that image autoencoders can be trained without image data ( just using uniform random noise as input ) if a proper perceptual distance is used . 2 ON THE RELATION BETWEEN PERCEPTUAL DISTANCE AND IMAGE PROBABILITY . The broadest formulation of the classical efficient coding hypothesis in neuroscience ( Barlow , 1961 ) states that the organization of the sensory system can be explained by the regularities of the natural signals ( Barlow , 2001 ) . This hypothesis has inspired statistical derivations of the nonlinear response of biological vision based on equalization : maximization of channel capacity ( Laughlin , 1981 ) and error minimization under noise ( Twer & MacLeod , 2001 ) are one-dimensional equalization techniques that directly relate signal probability , ppxq , with the response of the system to the visual input , y “ Spxq , where S is the given system . These univariate equalization techniques have been generalized to multivariate scenarios to explain the nonlinearities of the biological responses to spatial texture ( Malo & Gutiérrez , 2006 ) , color ( Laparra et al. , 2012 ) , and motion ( Laparra & Malo , 2015 ) . Interestingly , biological networks optimized to reproduce subjective image distortion display statistical properties despite using no statistical information in fitting their parameters . Divisive normalization obtained from perceptual distances substantially reduces the redundancy in natural images ( Malo & Laparra , 2010 ) , and captures a large fraction of the available information in the input ( Malo , 2020 ) . A similar factorization is obtained in Wilson-Cowan networks fitted to reproduce subjective distortions ( Gomez-Villa et al. , 2020a ) . Here the perceptual distance between an image x1 and its distorted version , x2 , will be referred to as Dppx1 , x2q and is computed in the response representation y . For small distortions ( i.e . x1 close to x2 ) , equalization models imply that there is an explicit relation between the perceptual distance , Dppx1 , x2q , and the probability of the original image according to the distribution of natural images , ppx1q ( see Appendix A ) : Dppx1 , x2q ||x2 ´ x1||2 “ Dppx1 , x2q ? m RMSE « ppxqγ ( 1 ) where γ ą 0 , and the Euclidean length ||x2 ´ x1||2 is just the Root Mean Squared Error ( RMSE ) scaled by the dimensionality of the data m. As we are interested in correlations , ignoring the scaling term , this relation leads to the first observation that we check experimentally in this work : Observation 1 The perceptual quantity Dppx1 , x2qRMSEpx1 , x2q is correlated with ppx1q for small distortions ( i.e . when ||x1 ´ x2||2 ă δ for small δ ) . This quantity is related to the sensitivity of the perceptual distance to displacements in the signal domain , BDpBx . In fact , the ( n-dimensional ) gradient reduces to Dp RMSE in the 1-d case ( Appendix A.1 ) . The observation states that the perceptual distance is more sensitive in high probability regions . This observation is experimentally illustrated in Appendix A.2 , that shows a strong connection between the sensitivity of perceptual distances ( either NLPD or MS-SSIM ) and the distribution of natural images , as opposed to the sensitivity of Euclidean RMSE , which is not related to the data . 3 ON THE RELATION BETWEEN DISTANCES INDUCED BY AUTOENCODERS AND PROBABILITY . In section 2 we showed the relation between perceptual distances and the distribution of natural images , and here we connect the distance induced by a statistical model and the probability of the data used to train it . We first discuss our observations , followed by describing the experimental setup . Statistical learning is based on risk minimization which connects the model , the loss to minimize and the distribution of the data . As a result , the trained model captures the statistics of the data . In this section we elaborate on this connection by computing Euclidean distances in the representations induced by the learned model . Here we focus on autoencoders , although the considerations apply to other statistical learning settings . We will consider the three different options to define the induced distances by an autoencoder shown in Fig . 1 . We will refer to them as self-reconstruction distance Ds “ ||x´ x̂||2 , reconstruction distance Dr “ ||x̂1 ´ x̂2||2 , and inner distance Din “ ||y1 ´ y2||2 . The autoencoder model will be denoted as f , which is consists of an encoder ep¨q and a decoder dp¨q , f “ d ˝ e. The reconstructed data is , x̂ “ fpxq , and y is the data in the inner domain , y “ epxq . We will explicitly show how all these three distances depend in different ways on the probability of the input data . Given samples x from a distribution ppxq a generic autoencoder minimizes this risk : R “ ExrLpfpxq , xqs “ ż Lpfpxq , xqdP pxq “ ż ppxq ¨ Lpx̂ , xqdx ( 2 ) where Exr¨s stands for expectation over variable x L is a loss between the input x and the reconstruction fpxq “ x̂ . Since ppxq is unknown and often intractable , the risk is approximated using the empirical risk minimization principle ( Devroye et al. , 1996 ; Vapnik , 1999 ) : Remp “ 1n řn i “ 1 Lpfpxiq , xiq , where the xi are samples from the training dataset of size n. Although it is well known , it is important for the reasoning of this work to stress that , the function fpxq that minimizes the empirical risk Remp is not the same function that would minimize the loss function L uniformly over all space . For example , if we choose L “ ||xi ´ fpxiq||2 , minimizing Remp is not minimizing the RMSE , but the RMSE weighted by the probability of samples , Eq . 2 . Therefore , the distance an autoencoder induces will be different to the loss function used for training . Once the model , fpxq , is trained , it inherits some properties of the data distribution , ppxq . This implies that the model ’ s behavior depends on the probability of the region of the input space where it is applied . In what follows , we make some observations on the relation between the distances Ds , Dr , and Din and the distribution ppxq . We assume that L is the Euclidean distance ( or RMSE ) , but similar arguments qualitatively apply to distances monotonically related to RMSE . Observation 2 The self-reconstruction distance in an autoencoder is correlated to the inverse of the probability : Ds “ ||x´ x̂||29 1ppxq . The difference between the original and the reconstructed data point in denoising and contractive autoencoders has been related with score matching : x ´ x̂ “ BlogpppxqqBx ( Vincent , 2011 ; Alain & Bengio , 2014 ) . This expression can be formulated using the distribution of the noisy data instead ( Miyasawa , 1961 ; Raphan & Simoncelli , 2011 ) . Here we argue that for a family of distributions the modulus of the difference ( ||x ´ x̂||2 ) can be related with the distribution itself ( Obs . 2 ) . This is true for Gaussian distribution , and a good proxy in general for unimodal distributions as the ones proposed to describe the distribution of natural images ( Portilla et al. , 2003 ; Hyvärinen et al. , 2009 ; Lyu & Simoncelli , 2009 ; Malo & Laparra , 2010 ; Lyu , 2011 ; van den Oord & Schrauwen , 2014 ; Ballé et al. , 2016 ) ( see details in Appendix B ) . Our intuition comes from Eq . 2 that enforces small reconstruction errors in high probability regions , i.e . when ppxq is high , ||x´ fpxq||2 should be low . This implies a dependency for the allowed error ( variance of the error ) on the probability : V arxp||x´fpxq||2q9 1ppxq , where V arx is computed around the point x . This is the idea behind the use of autoencoders for anomaly detection . An autoencoder trained for data coming from a particular distribution ppxq , obtains high self-reconstruction error when it is evaluated on data from a different distribution . Analogous to the distance used in Observation 1 , we argue that the quantity DrRMSE is correlated with the probability . The rationale is that if fp.q was trained to minimize the average distortion introduced in the reconstruction , then Dr has to be more sensitive in high density regions . Observation 3 The sensitivity of the reconstruction distance induced by an autoencoder in x1 , Dr RMSE “ ||fpx1q´fpx2q||2 ||x1´x2||2 , is correlated with ppx1q when ||x1 ´ x2||2 ă δ for small δ . While the autoencoder aims to reconstruct the original data , there are some restrictions which make the learned transformation different from the identity . This transformation is going to put more resources in regions where training data is more likely . This enforces a relation between the sensitivity ( derivative ) of the model and the probability distribution , i.e . | BfBx pxq| M ppxq ( see an example in Appendix C ) . Our hypothesis is that this relation should be positively ( possibly nonlinearly ) correlated . In particular we will be reasoning in the direction of using ppx1q||x1 ´ x2||2 as a proxy for the induced reconstruction distance Dr when x1 and x2 are close . While the distance in the inner domain , Din “ ||y1 ´ y2||2 , is highly used ( Ding et al. , 2020 ; Zhang et al. , 2018 ; Gatys et al. , 2016 ) , it is the most difficult to interpret . The main problem is to decide which representation inside the model to use . For autoencoders , the inner representation is usually selected as the layer with the least dimensions , although it is arbitrary what is defined as the inner representation . In compression autoencoders one has two suitable candidate representations , before or after the quantisation step . Both have different interesting properties , however in either cases the sensitivity conjecture also makes qualitative sense for Din . Observation 4 The sensitivity of the distance induced by a compression autoencoder in the inner domain at point x1 , DinRMSE “ ||epx1q´epx2q||2 ||x1´x2||2 , correlates with ppx1q when ||x1 ´ x2||2ăδ for small δ . This observation has a strong connection with density estimation , independent component analysis and data equalization . The density target under a transformation is ppxq “ ppepxqq| BeBx pxq| . If one wants to describe p ( x ) , a suitable target distribution for the transform domain is the uniform distribution . Therefore , if one assumes ppepxqq constant , the determinant of the Jacobian of the transformation is equal to the probability distribution as assumed in the equalization model ( Observation 1 ) , and similarly , one would have ||epxq´epx ` ∆xq||2||x´x ` ∆x||2 « ppxq . A similar result can be obtained modeling the noise in the inner domain ( Berardino et al. , 2017 ) . Note this observation has parallelisms with the procedure that the human visual system has to do in order to process the natural images as stated in Sec . 2 . Experiment . Here we explore the relation of the previously defined distances with the probability distribution of the training data using a compression autoencoder in a toy 2D example . Compression using autoencoders is presented as follows ( Ballé et al. , 2018 ; 2016 ) ; an input x is transformed using the encoder , or analysis transform , epxq to an encoding y. Scalar quantisation is then applied to y to create a vector of integer indices q and a reconstruction ŷ . The quantisation is approximated during training with additive uniform noise ỹ “ y ` ∆y to get a continuous gradient . It is then transformed back into the image domain using the decoder dpŷq to create the reconstructed image x̂ . Given x , the aim is to find a quantised representation q that minimizes the approximated entropy , or rate R “ Hrppỹqs , estimated by a density model p , whilst also minimizing the distortion loss , D , usually weighted by a Lagrange multiplier in order to achieve different compression rates . Thus the loss function L “ R ` λD becomes L “ Exr´ log2 ppepxq ` ∆yqs ` λDpx , dpepxq ` ∆yqq . ( 3 ) In order to have explicit access to the data distribution , a 2D Student-t distribution is used with one dimension having high variance and the other low variance . The Student-t distribution is a reasonable approximation for the distribution of natural images ( van den Oord & Schrauwen , 2014 ) . A 2 layer MLP is used for e and d. For full experimental setup see Appendix C. The induced distances are calculated for a set of samples taken from the Student-t distribution . Fig . 2 shows that these induced distances are correlated with the probability ppx1q . The three observations hold for intermediate regimes , low rate gets bad reconstruction while high rate gets almost no compression . Observation 2 ( Ds ) holds at low and medium rates where the Spearman correlation is -0.68 to -0.4 , but starts to fail at 4.67bpp ( bits-per-pixel ) where correlation approaches -0.18 . The same occurs for Observation 3 ( Dr ) where at high entropy the correlation decreases slightly from 0.49 at 2.03bpp to 0.31 at 4.67bpp . Observation 4 ( Din ) holds over all rates but especially at high rates where the largest correlation of 0.90 is found . These results indicate that the learned representations inside an autoencoder correlate with the training distribution . Although we test at a wide range of rates , if the entropy is very restricted , f “ d ¨ e has no capacity and in the limit may even be a constant function . However , this is a very extreme case and , in practice , an autoencoder like this would never be used .

Summary:  The work presented in this paper aims to analyze the relationships between the probability distribution of the data, perceptual distances, and unsupervised machine learning. Perceptual sensitivity is correlated with the probability of an image  in its close neighborhood. The paper also explores the relation between distances induced by autoencoders and the probability distribution of the training data, as well as how these induced distances are correlated with human perception. At the end, the paper specifies that perceptual distances do not always lead to noticeable gains in performance over Euclidean distance in common image processing tasks.