On the relation between statistical learning and perceptual distances

1 INTRODUCTION . The relationship between the internal representations of supervised learning models and biological systems has previously been explored ( Cadieu et al. , 2014 ) , and this connection can be explained by both systems being optimized to perform the same object recognition task . A much less studied area is comparing modern representations learned in an unsupervised manner to those of biological systems . As one of the most influential ideas in this area , the efficient coding hypothesis states that the internal representations of the brain have evolved to efficiently represent the stimuli ( Attneave , 1954 ; Barlow , 1961 ) and has been validated against statistical models for images ( Simoncelli & Olshausen , 2001 ; Malo & Laparra , 2010 ) . Similarly , the explicit constraints of dimensionality reduction or compression present in many unsupervised representation learning models , mainly autoencoders ( Ballé et al. , 2018 ; 2016 ; Baldi , 2012 ) , impose a type of parsimony on the representation . To understand properties of these representations of interest , here we consider distance measurements between the representations of pairs of stimuli . Such distance measurements can be thought of as perceptual distances , which are rooted in psychophysics : a good perceptual distance mimics the human rating of similarity between the two stimuli with high accuracy . Traditionally , perceptual distances have been hand-designed models with few adjustable parameters , inspired by the physiology or observations in visual psychology , as the Multi-Scale Structural SIMilarity index ( MS-SSIM ) ( Wang et al. , 2003 ) . More recently , it has become common to use an explicit image representation and ’ induce ’ a distance from it . For instance , comparing the internal representations of models trained for image classification for pairs of stimuli has been used for perceptual judgments and was shown to correlate well with human opinions ( Zhang et al. , 2018 ; Ding et al. , 2020 ) . This is also true for unsupervised representations that focus on learning features of natural scenes which are information efficient . For example , in the normalized Laplacian pyramid distance ( NLPD ) ( Laparra et al. , 2016 ) , the representation is learned based on redundancy reduction in neighboring pixels . The Perceptual Information Metric ( PIM ) ( Bhardwaj et al. , 2020 ) uses a contrastive representation learning technique based on compression and slowness . With regards to training autoencoders , a particular type of model that can be used to unsupervisedly learn an explicit representation , here we examine three distinct types of induced distances : the reconstruction distance Dr , the inner distance Din and the self-distance Ds . These distances correspond to different representations learned by the autoencoder ( see Fig . 1 ) . While the connection between the biological response and image probability has been examined before ( Laughlin , 1981 ; Twer & MacLeod , 2001 ; Malo & Gutiérrez , 2006 ; Malo & Laparra , 2010 ; Laparra et al. , 2012 ; Laparra & Malo , 2015 ; Hyvärinen et al. , 2009 ) , the relation between perceptual distances , unsupervised image representations , and the statistics of natural images has not been studied in depth . The current understanding is simply that distances induced by representations relevant for image classification or compression are useful for perceptual judgments . We show that the relation is deeper than that , linking it to image statistics . Furthermore , we examine the unexpected effects of utilizing perceptual distances in the loss functions of autoencoders , which is a common approach in designing neural image compression methods ( Ballé et al. , 2018 ) . One would expect that using a perceptual distance that is closely related to the internal image representation of the brain would give rise to a representation inside the model that is much more closely tied to humans , but that does not seem to be the case . This is surprising given the limited ability of Euclidean distances like Mean Squared Error ( MSE ) to reproduce human opinion ( Girod , 1993 ; Wang & Bovik , 2009 ) , compared to successful perceptual distances . We argue that this is because of a double counting effect where the distribution of natural images is taken into account twice in the training of the autoencoder ; once in the training data and again in the perceptual distance , leading to an over-stressing of high density regions in the data . Conversely , where data is sparse or contains non-representative samples , this effect can result in regularization by discounting losses from outliers . Our specific contributions are : 1 We demonstrate that good perceptual distances , i.e . distances that are good at predicting human psychophysical responses , are also correlated with image likelihoods obtained using a recent probabilistic image model ( PixelCNN++ ( Salimans et al. , 2017 ) ) . This underlines indirectly the idea that part of the biology is informed by efficient representation , as conjectured by Barlow . 2 The distances induced by autoencoders trained to minimize an Euclidean loss are correlated with the probability of the training data . Moreover , when autoencoders are trained using natural images , these induced distances are highly correlated with human perception . 3 Using a perceptual distance instead of a Euclidean loss in the optimization of autoencoders implies taking into account the data distribution twice : one in the perceptual distance and another through the empirical risk minimization procedure . We call this the double counting effect . This effect may explain the limited improvements obtained when using perceptual distances in some machine learning applications . We find that perceptual distances lead to models that over-stress high density regions . We emphasize this by showing that image autoencoders can be trained without image data ( just using uniform random noise as input ) if a proper perceptual distance is used . 2 ON THE RELATION BETWEEN PERCEPTUAL DISTANCE AND IMAGE PROBABILITY . The broadest formulation of the classical efficient coding hypothesis in neuroscience ( Barlow , 1961 ) states that the organization of the sensory system can be explained by the regularities of the natural signals ( Barlow , 2001 ) . This hypothesis has inspired statistical derivations of the nonlinear response of biological vision based on equalization : maximization of channel capacity ( Laughlin , 1981 ) and error minimization under noise ( Twer & MacLeod , 2001 ) are one-dimensional equalization techniques that directly relate signal probability , ppxq , with the response of the system to the visual input , y “ Spxq , where S is the given system . These univariate equalization techniques have been generalized to multivariate scenarios to explain the nonlinearities of the biological responses to spatial texture ( Malo & Gutiérrez , 2006 ) , color ( Laparra et al. , 2012 ) , and motion ( Laparra & Malo , 2015 ) . Interestingly , biological networks optimized to reproduce subjective image distortion display statistical properties despite using no statistical information in fitting their parameters . Divisive normalization obtained from perceptual distances substantially reduces the redundancy in natural images ( Malo & Laparra , 2010 ) , and captures a large fraction of the available information in the input ( Malo , 2020 ) . A similar factorization is obtained in Wilson-Cowan networks fitted to reproduce subjective distortions ( Gomez-Villa et al. , 2020a ) . Here the perceptual distance between an image x1 and its distorted version , x2 , will be referred to as Dppx1 , x2q and is computed in the response representation y . For small distortions ( i.e . x1 close to x2 ) , equalization models imply that there is an explicit relation between the perceptual distance , Dppx1 , x2q , and the probability of the original image according to the distribution of natural images , ppx1q ( see Appendix A ) : Dppx1 , x2q ||x2 ´ x1||2 “ Dppx1 , x2q ? m RMSE « ppxqγ ( 1 ) where γ ą 0 , and the Euclidean length ||x2 ´ x1||2 is just the Root Mean Squared Error ( RMSE ) scaled by the dimensionality of the data m. As we are interested in correlations , ignoring the scaling term , this relation leads to the first observation that we check experimentally in this work : Observation 1 The perceptual quantity Dppx1 , x2qRMSEpx1 , x2q is correlated with ppx1q for small distortions ( i.e . when ||x1 ´ x2||2 ă δ for small δ ) . This quantity is related to the sensitivity of the perceptual distance to displacements in the signal domain , BDpBx . In fact , the ( n-dimensional ) gradient reduces to Dp RMSE in the 1-d case ( Appendix A.1 ) . The observation states that the perceptual distance is more sensitive in high probability regions . This observation is experimentally illustrated in Appendix A.2 , that shows a strong connection between the sensitivity of perceptual distances ( either NLPD or MS-SSIM ) and the distribution of natural images , as opposed to the sensitivity of Euclidean RMSE , which is not related to the data . 3 ON THE RELATION BETWEEN DISTANCES INDUCED BY AUTOENCODERS AND PROBABILITY . In section 2 we showed the relation between perceptual distances and the distribution of natural images , and here we connect the distance induced by a statistical model and the probability of the data used to train it . We first discuss our observations , followed by describing the experimental setup . Statistical learning is based on risk minimization which connects the model , the loss to minimize and the distribution of the data . As a result , the trained model captures the statistics of the data . In this section we elaborate on this connection by computing Euclidean distances in the representations induced by the learned model . Here we focus on autoencoders , although the considerations apply to other statistical learning settings . We will consider the three different options to define the induced distances by an autoencoder shown in Fig . 1 . We will refer to them as self-reconstruction distance Ds “ ||x´ x̂||2 , reconstruction distance Dr “ ||x̂1 ´ x̂2||2 , and inner distance Din “ ||y1 ´ y2||2 . The autoencoder model will be denoted as f , which is consists of an encoder ep¨q and a decoder dp¨q , f “ d ˝ e. The reconstructed data is , x̂ “ fpxq , and y is the data in the inner domain , y “ epxq . We will explicitly show how all these three distances depend in different ways on the probability of the input data . Given samples x from a distribution ppxq a generic autoencoder minimizes this risk : R “ ExrLpfpxq , xqs “ ż Lpfpxq , xqdP pxq “ ż ppxq ¨ Lpx̂ , xqdx ( 2 ) where Exr¨s stands for expectation over variable x L is a loss between the input x and the reconstruction fpxq “ x̂ . Since ppxq is unknown and often intractable , the risk is approximated using the empirical risk minimization principle ( Devroye et al. , 1996 ; Vapnik , 1999 ) : Remp “ 1n řn i “ 1 Lpfpxiq , xiq , where the xi are samples from the training dataset of size n. Although it is well known , it is important for the reasoning of this work to stress that , the function fpxq that minimizes the empirical risk Remp is not the same function that would minimize the loss function L uniformly over all space . For example , if we choose L “ ||xi ´ fpxiq||2 , minimizing Remp is not minimizing the RMSE , but the RMSE weighted by the probability of samples , Eq . 2 . Therefore , the distance an autoencoder induces will be different to the loss function used for training . Once the model , fpxq , is trained , it inherits some properties of the data distribution , ppxq . This implies that the model ’ s behavior depends on the probability of the region of the input space where it is applied . In what follows , we make some observations on the relation between the distances Ds , Dr , and Din and the distribution ppxq . We assume that L is the Euclidean distance ( or RMSE ) , but similar arguments qualitatively apply to distances monotonically related to RMSE . Observation 2 The self-reconstruction distance in an autoencoder is correlated to the inverse of the probability : Ds “ ||x´ x̂||29 1ppxq . The difference between the original and the reconstructed data point in denoising and contractive autoencoders has been related with score matching : x ´ x̂ “ BlogpppxqqBx ( Vincent , 2011 ; Alain & Bengio , 2014 ) . This expression can be formulated using the distribution of the noisy data instead ( Miyasawa , 1961 ; Raphan & Simoncelli , 2011 ) . Here we argue that for a family of distributions the modulus of the difference ( ||x ´ x̂||2 ) can be related with the distribution itself ( Obs . 2 ) . This is true for Gaussian distribution , and a good proxy in general for unimodal distributions as the ones proposed to describe the distribution of natural images ( Portilla et al. , 2003 ; Hyvärinen et al. , 2009 ; Lyu & Simoncelli , 2009 ; Malo & Laparra , 2010 ; Lyu , 2011 ; van den Oord & Schrauwen , 2014 ; Ballé et al. , 2016 ) ( see details in Appendix B ) . Our intuition comes from Eq . 2 that enforces small reconstruction errors in high probability regions , i.e . when ppxq is high , ||x´ fpxq||2 should be low . This implies a dependency for the allowed error ( variance of the error ) on the probability : V arxp||x´fpxq||2q9 1ppxq , where V arx is computed around the point x . This is the idea behind the use of autoencoders for anomaly detection . An autoencoder trained for data coming from a particular distribution ppxq , obtains high self-reconstruction error when it is evaluated on data from a different distribution . Analogous to the distance used in Observation 1 , we argue that the quantity DrRMSE is correlated with the probability . The rationale is that if fp.q was trained to minimize the average distortion introduced in the reconstruction , then Dr has to be more sensitive in high density regions . Observation 3 The sensitivity of the reconstruction distance induced by an autoencoder in x1 , Dr RMSE “ ||fpx1q´fpx2q||2 ||x1´x2||2 , is correlated with ppx1q when ||x1 ´ x2||2 ă δ for small δ . While the autoencoder aims to reconstruct the original data , there are some restrictions which make the learned transformation different from the identity . This transformation is going to put more resources in regions where training data is more likely . This enforces a relation between the sensitivity ( derivative ) of the model and the probability distribution , i.e . | BfBx pxq| M ppxq ( see an example in Appendix C ) . Our hypothesis is that this relation should be positively ( possibly nonlinearly ) correlated . In particular we will be reasoning in the direction of using ppx1q||x1 ´ x2||2 as a proxy for the induced reconstruction distance Dr when x1 and x2 are close . While the distance in the inner domain , Din “ ||y1 ´ y2||2 , is highly used ( Ding et al. , 2020 ; Zhang et al. , 2018 ; Gatys et al. , 2016 ) , it is the most difficult to interpret . The main problem is to decide which representation inside the model to use . For autoencoders , the inner representation is usually selected as the layer with the least dimensions , although it is arbitrary what is defined as the inner representation . In compression autoencoders one has two suitable candidate representations , before or after the quantisation step . Both have different interesting properties , however in either cases the sensitivity conjecture also makes qualitative sense for Din . Observation 4 The sensitivity of the distance induced by a compression autoencoder in the inner domain at point x1 , DinRMSE “ ||epx1q´epx2q||2 ||x1´x2||2 , correlates with ppx1q when ||x1 ´ x2||2ăδ for small δ . This observation has a strong connection with density estimation , independent component analysis and data equalization . The density target under a transformation is ppxq “ ppepxqq| BeBx pxq| . If one wants to describe p ( x ) , a suitable target distribution for the transform domain is the uniform distribution . Therefore , if one assumes ppepxqq constant , the determinant of the Jacobian of the transformation is equal to the probability distribution as assumed in the equalization model ( Observation 1 ) , and similarly , one would have ||epxq´epx ` ∆xq||2||x´x ` ∆x||2 « ppxq . A similar result can be obtained modeling the noise in the inner domain ( Berardino et al. , 2017 ) . Note this observation has parallelisms with the procedure that the human visual system has to do in order to process the natural images as stated in Sec . 2 . Experiment . Here we explore the relation of the previously defined distances with the probability distribution of the training data using a compression autoencoder in a toy 2D example . Compression using autoencoders is presented as follows ( Ballé et al. , 2018 ; 2016 ) ; an input x is transformed using the encoder , or analysis transform , epxq to an encoding y. Scalar quantisation is then applied to y to create a vector of integer indices q and a reconstruction ŷ . The quantisation is approximated during training with additive uniform noise ỹ “ y ` ∆y to get a continuous gradient . It is then transformed back into the image domain using the decoder dpŷq to create the reconstructed image x̂ . Given x , the aim is to find a quantised representation q that minimizes the approximated entropy , or rate R “ Hrppỹqs , estimated by a density model p , whilst also minimizing the distortion loss , D , usually weighted by a Lagrange multiplier in order to achieve different compression rates . Thus the loss function L “ R ` λD becomes L “ Exr´ log2 ppepxq ` ∆yqs ` λDpx , dpepxq ` ∆yqq . ( 3 ) In order to have explicit access to the data distribution , a 2D Student-t distribution is used with one dimension having high variance and the other low variance . The Student-t distribution is a reasonable approximation for the distribution of natural images ( van den Oord & Schrauwen , 2014 ) . A 2 layer MLP is used for e and d. For full experimental setup see Appendix C. The induced distances are calculated for a set of samples taken from the Student-t distribution . Fig . 2 shows that these induced distances are correlated with the probability ppx1q . The three observations hold for intermediate regimes , low rate gets bad reconstruction while high rate gets almost no compression . Observation 2 ( Ds ) holds at low and medium rates where the Spearman correlation is -0.68 to -0.4 , but starts to fail at 4.67bpp ( bits-per-pixel ) where correlation approaches -0.18 . The same occurs for Observation 3 ( Dr ) where at high entropy the correlation decreases slightly from 0.49 at 2.03bpp to 0.31 at 4.67bpp . Observation 4 ( Din ) holds over all rates but especially at high rates where the largest correlation of 0.90 is found . These results indicate that the learned representations inside an autoencoder correlate with the training distribution . Although we test at a wide range of rates , if the entropy is very restricted , f “ d ¨ e has no capacity and in the limit may even be a constant function . However , this is a very extreme case and , in practice , an autoencoder like this would never be used .

Summary: This paper presents a mainly theoretical explication of the relationship between natural image statistics and perceptual distances for small image distortions. The paper presents a number of observations linking distances in natural images, autoencoders, and perceptual similarity for humans. The paper finally explores some implications of these observations, including impressive-seeming results on training with no data using perceptual distances as regularizers.