On the relation between statistical learning and perceptual distances
Authors: Alexander Hepburn
Valero Laparra
Raul Santos-Rodriguez
Jona Ballé
Jesus Malo
Venue: ICLR 2022 Spotlight (2022)

Abstract: It has been demonstrated many times that the behavior of the human visual system is connected to the statistics of natural images. Since machine learning relies on the statistics of training data as well, the above connection has interesting implications when using perceptual distances (which mimic the behavior of the human visual system) as a loss function. In this paper, we aim to unravel the non-trivial relationships between the probability distribution of the data, perceptual distances, and unsupervised machine learning. To this end, we show that perceptual sensitivity is correlated with the probability of an image in its close neighborhood. We also explore the relation between distances induced by autoencoders and the probability distribution of the training data, as well as how these induced distances are correlated with human perception. Finally, we find perceptual distances do not always lead to noticeable gains in performance over Euclidean distance in common image processing tasks, except when data is scarce and the perceptual distance provides regularization. We propose this may be due to a double-counting effect of the image statistics, once in the perceptual distance and once in the training procedure.

Published as a conference paper at ICLR 2022
ON THE RELATION BETWEEN STATISTICAL LEARNING AND PERCEPTUAL DISTANCES
Alexander Hepburn
Valero Laparra
Engineering Mathematics
Image Processing Lab
University of Bristol
Universitat de Valencia
alex. hww..heaabrr@@bbittll.....
valero. waw.raappaaaee.e...
Raul Gantos-Rodriguu
Johannes Ballé Google Research
Jesús Malo
Engineering Mathematics University of Bristol enrsrâbristen ac. .uk
Image Processing Lab Universitat de Valencia jesus , ma uduv. es
jballe@gpogle.com com
ABSTRACT
It has been demonstrated many times that the behavior the human visual system is connected to the statistics of natural images. Since machine learning relies on the statistics of training data as well, the above connection has interesting implications when using perceptual distances (which mimic the behavior of the human visual system) as a loss function. In this paper, we aim to unravel the non-trivial relationships between the probability distribution of the data, perceptual distances, and unsupervised machine learning. To this end, we show that perceptual sensitivity is correlated with the probability of an image its close neighborhood. We also explore the relation between distances induced by autoencoders and the probability distribution the training data, as well as how these induced distances are correlated with human perception. Finally, we find perceptual distances do not always lead to noticeable gains in performance over Euclidean distance in common image processing tasks, except when data is scarce and the perceptual distance provides regularization. We propose this may be due to a double-counting effect of the image statistics, once in the perceptual distance and once in the training procedure.
INTRODUCTION
The relationship between the internal representations supervised learning models and biological systems has previously been explored (Cadieu al. 2014). and this connection can be explained by both systems being optimized to perform the same object recognition task. much less studied area is comparing modern representations learned in unsupervised manner those of biological systems. As one of the most influential ideas in this area, the efficient coding hypothesis states that the internal representations of the brain have evolved efficiently represent the stimuli (Attneavel 1954 Barlow 1961) and has been validated against statistical models for images (Simoncelli & Olshausen |001
Malo & Laparra 2010). Similarly, the explicit constraints of dimensionality reduction compression
present in many unsupervised representation learning models, mainly autoencoders (Ballé et al. 2018
2016 Baldi 2012), impose type of parsimony on the representation. To understand properties of these representations of interest, here we consider distance measurements between the representations of pairs of stimuli. Such distance measurements can be thought of as perceptual distances, which are rooted in psychophysies: a good perceptual distance mimics the human rating of similarity between the two stimuli with high accuracy.
Traditionally, perceptual distances have been hand-designed models with few adjustable parameters, inspired by the physiology or observations in visual psychology, as the Multi-Scale Structural SIMilarity index (MS-SSIM) (Wang et al.) 2003). More recently, it has become common to explicit image representation and 'induce' a distance from it. For instance, comparing the internal representations of models trained for image classification for pairs of stimuli has been used for

---

Published as a conference paper at ICLR 2022
perceptual judgmentss and was shown to correlate well with human opinions (Zhang et al. 2018 Ding et al. 2020). This is also true for unsupervised representations that focus on learning features of natural scenes which information efficient. For example, in the normalized Laplacian pyramid distance (NIPDD) (Laparra et al., 2016), the representation is learned based on redundancy reduction in neighhoring pixels. The Perceptual Information Metric (PIM) (Bhardwaj et al. (020) uses contrastive representation learning technique based on compression and slowness. With regards to training autoencoders, a particular type of model that can be used unsupervisedly learn an explicit representation, here we examine three distinct types of induced distances: the reconstruction distance D,, the inner distance Din and the self-distance D,. These distances correspond different representations learned by the autoencoder (see Fig. D.
While the connection between the biological response and image probability has been examined before (Laughlin| 1981;Twer & Macl 200T; Malo & Gutiérrez, 2006), Malo & Laparra 2010) Laparra et al. | Laparra & Malo 2013 Hyvärinen et al. 2000),. the relation between perceptual distances, unsupervised image representations, and the statistics of natural images has not been
studied in depth. The current understanding is simply that distances induced by representations relevant for image classification or compression are useful for perceptual judgments. We show that the relation is deeper than that, linking it to image statistics.
Furthermore, examine the unexpected effects of utilizing perceptual distances in the loss functions of autoencoders, which is a common approach in designing neural image compression methods (Ballé let al. 2018). One would expect that using a perceptual distance that is closely related to the internal image representation of the brain would give rise to a representation inside the model that is much more closely tied to humans, but that does not seem to be the case. This is surprising given the limited ability of Euclidean distances like Mean Squared Error (MSE) to reproduce human opinion (Girod 1993; Wang Bovik 2009), compared to successful perceptual distances. We argue that this is because of a double counting effect where the distribution of natural images is taken into account twice in the training of the autoencoder; once in the training data and again in the perceptual distance, leading to an over-stressing of high density regions in the data. Conversely, where data is sparse or contains non-representative samples, this effect can result in regularization by discounting losses from outliers. Our specific contributions are:
1 We demonstrate that good perceptual distances, i.e. distances that are good at predicting human psychophysical responses, also correlated with image likelihoods obtained using a recent probabilistic image model (PixeICNN++ (Salimans et al. 2017p. This underlines indirectly the idea that part of the biology is informed by efficient representation, as conjectured by Barlow.
2 The distances induced by autoencoders trained to minimize an Euclidean loss are correlated with the probability of the training data. Moreover, when autoencoders trained using natural images, these induced distances are highly correlated with human perception.
3 Using a perceptual distance instead of a Euclidean loss in the optimization of autoencoders implies taking into account the data distribution twice: one in the perceptual distance and another through the empirical risk minimization procedure. We call this the double counting effect. This effect may explain the limited improvements obtained when using perceptual distances in some machine learning applications. We find that perceptual distances lead to models that over-stress high density regions. We emphasize this by showing that image autoencoders can be trained without image data (just using uniform random noise as input) if proper perceptual distance is used.
ON THE RELATION BETWEEN PERCEPTUAL DISTANCE AND IMAGE PROBABILITY
The broadest formulation of the classical efficient coding hypothesis neuroscience (Barlow 1961 ) states that the organization of the sensory system can be explained by the regularities of the natural signals (Barlow] 2001). This hypothesis has inspired statistical derivations the nonlinear response of biological vision based on equalization: maximization of channel capacity (Laughlin) 1981) and error minimization under noise (Twer& & MacLeod] 200T) are one-dimensional equalization techniques that directly relate signal probability, p(æ), with the response of the system to the visual input, y = S(z), where S is the given system. These univariate equalization techniques have been generalized to multivariate scenarios to explain the nonlinearities of the biological responses to
2

---

Published as a conference paper at ICLR 2022
Original domain X
Inner domain Y
e(x)
De
f d
X2
D
Dg
in
d(y)
Y2
Figure I: Distances in autoencoders. D. is the euclidean distance (green), D, is the reconstruction distance (blue), D2 is the self-distance, Din is the inner distance. Red circles illustrate the allowed enror for the points X₃ and X2, i... the reconstructed points (x1 and Xy) should live inside the circle. The sizes are inversely proportional to the density (blue background). The autoencoder function, f. is the composition of the encoder function, and the decoder function, d.
spatial texture (Malo & Gutiérrez, 2006), color (Laparra et al. 2012), and motion (Laparra & Malo 2015). Interestingly, biological networks optimized to reproduce subjective image distortion display statistical properties despite using statistical information in fitting their parameters. Divisive normalization obtained from perceptual distances substantially reduces the redundancy in natural images (Malo & Laparra 2010), and captures a large fraction of the available information in the input (Malo] (220). A similar factorization obtained in Wilson-Cowan networks fitted to reproduce subjective distortions (Gomez- Villa et al.] 2020a).. Here the perceptual distance between an image X1 and its distorted version, X2. will be referred to as (x(x1, X2) and is computed in the response representation y. For small distortions (i.e. X1 close to x2), equalization models imply that there an explicit relation between the perceptual distance, (x(, and the probability of the original image according to the distribution of natural images, p(x1) (see Appendix Ap:
X2) Dp(x1.X2)
p(x)7
||[ X1
V RMSE
where y > 0, and the Euclidean length X2 X1 lll is just the Root Mean Squared Error (RMSE) scaled by the dimensionality of the data m. As we are interested in correlations, ignoring the scaling term, this relation leads to the first observation that we check experimentally in this work:
Observation 1 The perceptual quantity (x) is correlated with p(x1) ) for small distortions (i.e. when ||x; -xyll2 < for small 5).
This quantity is related to the sensitivity of the perceptual distance to displacements the signal D2 domain, dæ In fact, the (n-dimensional) gradient reduces to RMSE the 1-d case (Appendix A The observation states that the perceptual distance is more sensitive in high probability regions.
This observation is experimentally illustrated in Appendix 2 that shows a strong connection between the sensitivity of perceptual distances (either NLPD or MS-SSIM and the distribution of natural images, as opposed to the sensitivity of Euclidean RMSE, which is not related to the data.
ON THE RELATION BETWEEN DISTANCES INDUCED BY AUTOENCODERS AND PROBABILITY
In section 2 we showed the relation between perceptual distances and the distribution of natural images, and here we connect the distance induced by statistical model and the probability of the data used to train it. We first discuss our observations, followed by describing the experimental setup.
Statistical learning is based on risk minimization which connects the model, the loss to minimize and the distribution of the data. As result, the trained model captures the statistics of the data. In this section we elaborate on this connection by computing Euclidean distances in the representations induced by the learned model. Here we focus on autoencoders, although the considerations apply to other statistical learning settings. We will consider the three different options to define the induced distances by an autoencoder shown in Fig. [ We will refer to them as relf-recomstruction distance D. = ||x x||22. reconstruction distance D, = || - Kell2, and inner distance Din = |ly: y2ll2. The autoencoder model will be denoted as f. which is consists of encoder e(-) and decoder d(-), f = o e. The reconstructed data is, & = f(x), and y is the data in the inner domain, y = e(x). We will explicitly show how all these three distances depend in different ways the probability of the
3

---

Published as a conference paper at ICLR 2022
input data.
Given samples x from a distribution p(x)= a generic autoencoder minimizes this risk:
== Ex[[[((xx),
( x)d(xx)
p(x) -(x, x)dx
(2)
where [xx stands for expectation over variable x L is loss between the input x and the reconstruc- tion f(x) X. Since p(x) is unknown and often intractable. the risk is approximated using the empir- ical risk minimization principle (Devroye et al. 1996: Vapnik |999): Remp = -ー= (x)((x), x.), where the x, are samples from the training dataset of size n. Although it is well known, it is important for the reasoning of this work to stress that, the function [(x) that minimizes the empirical risk Remp is not the same function that would minimize the loss function L uniformly over all space. For example, if we choose L = ||x] f(x))ll, minimizing Remp is not minimizing the RMSE, but the RMSE weighted by the probability of samples, Eq. [ Therefore, the distance an autoencoder induces will be different to the loss function used for training. Once the model, f fxx), is trained, it inherits some properties of the data distribution, p(x). This implies that the model's behavior depends on the probability of the region of the input space where it is applied.
In what follows, we make some observations on the relation between the distances D,. Dy, and Din and the distribution p(x). We assume that L is the Euclidean distance (or RMSE), but similar arguments qualitatively apply to distances monotonically related to RMSE.
Observation 2 The relffrecconttruction distance in an autoencoder is correlated to the inverse of the probability: D3 ||x x||2 p(x)
The difference between the original and the reconstructed data point in denoising and contractive
autoencoders has been related with score matching: x - x [log(p(x)) (Vincent 2011 1 Alain & Bengio| 2014). This expression can be formulated using the distribution of the noisy data instead (Miyasawal [961 Raphan & Simoncelli 2011). Here we argue that for family of distributions the modulus of the difference (||x- x|(2) can be related with the distribution itself (Obs. 2 This is true for Gaussian distribution, and good proxy in general for unimodal distributions the ones
proposed to describe the distribution of natural images (Portilla et al.] 2003 Hyvärinen et al. 2009
Lyu & Simoncelli] [2009] Malo & Laparra 2010 Lyu 2011
van den Oord & Schrauwen 2014 Ballé
ct al. 2016) (see details in Appendix
2 that enforces small reconstruction errors in high probability regions, i.e. when p(x) is high, ||x [x)ll2 should be low. This implies a dependency for the allowed error (variance of the error) on the probability: Varz (x))= p(x) where V Yarz is computed around the point x. This is the idea behind the use of autoencoders for anomaly detection. An autoencoder trained for data coming from particular distribution p(x), obtains high self-reconstruutiin error when it is evaluated on data from a diffcrent distribution.
Our intuition comes from Eq.
Analogous to the distance used in Observation I we argue that the quantity RMSE is correlated with the probability. The rationale is that if f-) was trained to minimize the average distortion introduced in the reconstruction, then D, has to be more sensitive in high density regions.
D.
Observation 3 The sensitivity of the reconstruction distance induced by an autoencoder in X1, D. RMSE
[xx] ) (xxa)]l is correlated with p(x1) when ||x] -xxl2 < x xal1a
for small 5.
While the autoencoder aims to reconstruct the original data, there are some restrictions which make the learned transformation different from the identity. This transformation is going to put more resources regions where training data is more likely. This enforces a relation between the sensitivity (derivative) of the model and the probability distribution, i.e. j (x)] & p(x) (see an example in Appendix C. Our hypothesis is that this relation should be positively (possibly non- linearly) correlated. In particular we will be reasoning the direction of using p(x1) proxy for the induced reconstruction distance D, when x1 and x2 close.
ix
X2|12 as
While the distance in the inner domain, Din = ||y y2ll2. 2. is highly used (Ding et al. 2020 Zhang et al. 2018: Gatys al. 2016), it is the most difficult to interpret. The main problem to decide which representation inside the model to use. For autoencoders, the inner representation is usually selected as the layer with the least dimensions, although it is arbitrary what is defined as the inner representation. In compression autoencoders one has two suitable candidate representations, before or after the quantisation step. Both have different interesting properties, however in either cases the
4

---

Published as a conference paper at ICLR 2022
D3 = 1|x1 81112
D: RMSE
||xs
kallz Xalb
Dp RMSE
[
X Xall
0.4
0.2
0.0
0.2
-0.4
Rate (bpp)
Rate (bpp)
Rate (bpp)
Figure 2: Illustrating Observations 2, 3, and Spearman correlations ps Ps p(x1)) between different distances and the probability of point X1 are shown. Each point corresponds to the correlation for one autoencoder trained for a particular rate regime.
sensitivity conjecture also makes qualitative sense for Din-
Observation 4 The sensitivity of the distance induced by a compression autoencoder the inner domain at point X1,
Dan ee(x))=)(xx>llz correlates with p(x1) when |x] X2|12 < for small 6.
RMSE
1 22
This observation has strong connection with density estimation, independent component analysis and data equalization. The density target under a transformation is p(x) = p(e(x)) Xx (x)] If one wants to describe p(x), suitable target distribution for the transform domain is the uniform distribution. Therefore, if one assumes ple(x)) constant, the determinant of the Jacobian of the transformation equal to the probability distribution as assumed in the equalization model (Observation [ and [((x))<(x++Aa)ll x p(x). A similar result can be obtained modeling the noise in the inner domain (Berardino et al. 2017). Note this observation has parallelisms with the procedure that the human visual system has to in order to process the natural images as stated in Sec.
similarly, one would have x
Experiment. Here we explore the relation of the previously defined distances with the probability distribution of the training data using compression autoencoder in a toy 2D example. Compression using autoencoders presented as follows (Ballé et al., 2018 2016): an imput x transformed using the encoder, or analysis transform, e(x) to an encoding y. Scalar quantisation is then applied to y to create vector of integer indices q and a reconstruction y. The quantisation approximated during training with additive uniform noise ý y + Ay to get a continuous gradient. It is then transformed back into the image domain using the decoder d()) to create the reconstructed image &
Given the aim is to find quantised representation q that minimizes the approximated entropy, or rate R = H [r())] estimated by a density model p, whilst also minimizing the distortion loss, D, usually weighted by Lagrange multiplier in order to achieve different compression rates. Thus the loss function C = R + XD becomes
L Ex [- 1og2 p(e(x) Ay)] XD(x, d(e(x) Ay)).
In order to have explicit access to the data distribution, a 2D Student-t distribution used with one dimension having high variance and the other low variance. The Student-t distribution is reasonable approximation for the distribution of natural images (van den Oord & Schrauwen 2014). A layer MLP is used for and d. For full experimental setup Appendix C The induced distances calculated for set of samples taken from the Student-t distribution. Fig. 2 shows that these induced distances are correlated with the probability p(x1). The three observations hold for intermediate regimes, low rate gets bad reconstruction while high rate gets almost no compression. Observation 2 (D,) holds at low and medium rates where the Spearman correlation -0.68 to -0.4, but starts to fail at 4.67bpp (bits-per, per-pxeel where correlation approaches -0.18. The same occurs for Observation (D,) where at high entropy the correlation decreases slightly from 0.49 at 2.03bpp to 0.31 at 4.6 4.67bpp. Observation 4 (D holds over all rates but especially at high rates where the largest correlation of 0.90 is found. These results indicate that the learned representations inside an autoencoder correlate with the training distribution. Although we test at wide range of rates, if the entropy is very restricted, f = d e has no capacity and in the limit may even be a constant function. However, this is very extreme case and, in practice, autoencoder like this would never be used.
5

---

Published as a conference paper at ICLR 2022
4 ON THE RELATION BETWEEN DISTANCES INDUCED BY AUTOENCODERS AND PERCEPTION
As stated above, machine learning models trained on natural images for classification or compres- sion can lead to successful perceptual distances (Zhang et al.) 2018 Ding et al. 2020 Bhardwajj (ct al., 2020). Similarly, human- -like Contrast Sensitivity Functions arise in autoencoders for image
enhancement (Gomez- Villa et al. 2020b; Li et al. (02I). Reproducing CSFs is key in percep-
tual distances (Mannos & Sakrison 1974 Watson 1993). The latent space of a machine learning
model can also be used for smooth interpolation between images (Connor & Rozell [020 [Berthelot et al. 2018) whilst the statistics of activations in different layers can be used to interpolate between textures Vaccher al. 2020
The emergence of perceptually relevant distances (or key features to compute perceptual distances) in image classifiers and autoencoders trained optimize Euclidean distortions over natural images should not be a surprise given the correlations found in Sections 2 and 3: both the perceptual distances and the autoencoder-induuce distances are correlated with the probability. Therefore, these two distances should be correlated as well, and more importantly, with the actual opinion of humans. We indeed find this to be the case:
Observation 5 Both the reconstruction distance induced by an autoencoder, Dr(x1,x2) - ((x)lle.. and the inner distance, Din(x1, = [|e(x1) e(x2)]l2 are correlated with the subjective opinion given by humans.
Experiment. The TID 2013 dataset (Ponomarenko et al. 2013) contains 25 reference images, X1, and 3000 distorted images, X2, with 24 types and 5 levels of severity of distortion. Also provided is the mean opinion score (MOS) for each distorted image. This psychophysical MOS represents the subjective distortion rating given by humans which is the ground truth for perceptual distances precisely what human vision models try to explain.
For all experiments, networks to compute D, and Din pretrained models from Ballé et al. 2016 See the architecture details in Appendix D
As in Section 3] the autoencoder induces distances in the reconstruction and inner domain. Fig. reports the Spearman correlations hetween the induced distance and MOS, p( D. MOS), for the TT) 2013 database. As reference, we include the correlation between the MOS and RMSE ||x ||xı-xx) X2 2- Results in Fig. Connfirm Observation 5 correlations are bigger for both induced distances than for RMSE, particularly for D... (with maximum of 0.77). The correlations for models trained with MSE or MS-SSIM are very similar in medium to high rates where the difference in low rates can be explained by the double counting effect described in Sec.5.1. Correlations are bigger in the inner representation than the reconstructed representation, as in the 2D example in Fig. 2 For both induced distances, the maximum correlation for the reconstructed domain occurs at the highest bit-rate (more natural images). Interestingly, for networks trained using the perceptual distance MS-SSIM the correlations remain similar. It is common practice to use Din distance as a more informative distance between two data points than D,. with LPIPS (Zhang et al. 2018). DISTS (Ding et al. 2020)) and PIM (Bhardwaj et al., 2020), where the correlation these methods achieve with human opinion scores is explained by the correlation in the inner domain.
should be noted that specific examples of a reference and distorted image with the same internal representation, e(x1)= e(x2), are similar to metamers in humans physically different objects that elicit the same internal representation and thus have 0 perceptual distance (Freeman & Simoncelli 201T). Whilst we could find images with maximum RMSE in the image domain but with the embedded distance being 0, the existence of these data points not included in TID does not reduce the validity of the experiment.
5 OPTIMIZING MACHINE LEARNING MODELS WITH PERCEPTUAL DISTANCES
In Sec. 2 it was established that the sensitivity of perceptual distances at a point is correlated with the probability of that point and in Sec. we showed that distances induced by autoencoder
Code taken from
under Apache I. License 2.0.

---

Published as a conference paper at ICLR 2022
D, = ||x1 82112
Din = 11
82112
0.8
pg MOS)
factorized-ne mse
factorized-man msssim
[X Xallz
0.8 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00
0.00
0.25 0.50
0.75 1.00
1.25 1.50 1.75 2.00
bits per pixel (bpp)
bits per pixel (bpp)
Figure 3: Illustrating Observation 5 Spearman correlations pg between induced distances (D, or Din) and mean opinion score (MOS) for images from TID 2013 dataset (Ponomarenko et al.) 2013). Pretrained compressive autoencoders at different bitrates were used. factorized-mse denotes networks trained using MSE D
X1 X2ll2 in Eq. 3 and factorized-msssin networks use
D = MS-SSIM(X: X2) in Eq.
representations also inherit a correlation with the probability of the point (which makes them acquire perceptual properties seen in Sec.E4) This raises the question whether perceptual distances can be used in place of the data distribution. Empirically, we find this to be the case:
Observation 6 Optimizing using perceptual distance Dp as loss function has similar ef- fect as optimizing using the RMSE weighted by the probability, arg min Em= Dp(x,, (,, &;) & arg min > p(x:). [|x,; 8:112-
Using perceptual distance in empirical risk minimization context (Eq. [ appears to have double counting effect of the distribution of natural images; once in the sensitivity of the perceptual distance and second time when optimizing over the set of images.
In Sec. 5.1 we show that using perceptual distance has a similar effect to using MSE weighted by p(x). Excessive considcration p(x) may explain why using perceptual distances instead of MSE has little gain in some machine learning tasks (Hepburn et al. 2020 Ballé et al. 2016) despite the large difference in replicating perceptual judgments.
In Sec. 5.2 we present an extreme special case of this setting, where one has no samples from the distribution to minimize over. Instead, the model has direct access the perceptual distance, which acts as proxy of p(x). This effect can be also seen in Fig. 3 where the difference of using or not using a perceptual distance can only be observed at low rates (poor quality non-natural images): when dealing with poor quality data, extra regularization is helpful. This regularization effect is consistent with what was found when using perceptual distances in denoising (Portilla 2006). Finally, in Appendix E. we show the effect when using batch size of 1 in training.
5.1 PERCEPTUAL DISTANCES AND PROBABILITY HAVE A SIMILAR EFFECT IN RISK MINIMIZATION
Here examine this effect in compression autoencoders, using either RMSE, RMSE weighted by the probability or a perceptual distance as a loss function. State of the art density estimators like PixeICNN+ can be used to estimate p(x) only on small patches of images. On the other hand, perceptual distances are designed to estimate distances with large images. As such, we choose to compare the effect in two different scenarios: the 2D example used Sec. and the image compression autoencoders used in Sec.
The rate/distortion used as a proxy for the general performance of network. The relative performance is obtained by dividing the performance of the network optimized for the RMSE weighted by the probability (or the perceptual distance) by the performance of the network using RMSE. A relative performance below 1.0 means a performance gain using the probability (or perceptual distance). We evaluate the performance with data sampled from a line through the distribution from high to low probability. Fig. shows the relative performance for both the 2D example and sampled images. For the 2D case, sampling through the distribution is trivial, (æ, y) ([O, 35]. 0). For images, create more and less likely samples under the image distribution (low contrast images are generally more likely (Frazor & Geisler 2006)). dictates where in the contrast spectrum the images lie,
modify image contrast of the Kodak dataset (Kodak] 1993) in order to
7

---

Published as a conference paper at ICLR 2022
Relative performance
Relative Perormance
2.0
1.8
(doc) atly
2.5
1.6
2.0
14
1.5
(doa)
122
1.0
1.0
0.8
0.0 High
0.5
1.0
15
2.0 Low'
High
Probability
Probabilty
Probabiltt
Figure Illustrating observation 6 Figure shows the relative performance the models for
samples across the support of the respective data distributions. Both training with the probability (left) and the perceptual distance (right) cause the model to allocate more resources to high probability regions. Left: models trained with D p(x) ||x x2llă in Eq. divided by performance of models trained with D = ||x X2 22 on the 2D Student-t and evaluated using samples along z-axis MS-SSIM(X1, X2) in
(see Appendix Cp. Right: models trained for image compression with D
Eq. divided by performance of models trained with D = | X2 x2|
where a 0.0 represents very low contrast (solid gray), a 1.0 the original image and a 2.0 illustrates high contrast. Details on how the performance is computed and how the samples from different probability regions are taken are in Appendix
We observe a performance gain in data points that are more probable and a performance loss away from the center of the distribution. This behavior similar in both the Student-t example when multiplying the distortion by the probability, and when using perceptual distance as the distortion term with images. In the 2D example (a) the maximum average performance gain across rates center of the distribution is 0.86, whereas with images (b) the maximum average performance gain 0.64 at a = 0.08. This leads us to the conclusion that multiplying by the probability and using a perceptual distance have similar effect the learning process.
5.2 TRAINING WITHOUT DATA
While in Appendix E.I we show the effect of using batch size of 1 during training, here we focus on the most extreme example. We show how explicit access the samples from the data distribution might not be needed. If given the probability distribution, one can weight samples drawn from a uniform distribution with the same dimensionality and evaluate the likelihood of these points belonging to the original distribution. With the 2D Student-t example, using uniform distribution and minimizing loss weighted by the likelihood the samples belong to the Student-t, u.1 where the optimization would be minx-14 C millx-14 P, (x) (R + XD), where pr (x) denotes the probability of point x belonging to the 2D Student-t distribution (see Fig. 13b in Appendix C). In the 2D example this is trivial, however this is not the case for natural images. In Sec. 2 it can be that perceptual distances encode some information about the distribution natural images and, as such, be used in similar fashion to Pr p+(x) when training on uniform distributed data. The perceptual distance can be minimized over uniform noise and should result in realistic looking images.
We will use the previous setting of compressive autoencoders but for ease of training, we approximate the rate-distortion equation in Eq. Instead of optimizing the rate, we set an upper bound on the entropy of the encoding by rounding the points to set of known code vectors L and purely optimize for the distortion (Ding et al., 2021 Agustsson et al. 2019). We use two scenarios, L = {-1, 1} leading an entropy of 0.25 bpp and L 2 2. 1, 2} 0.5 bpp. We refer the reader to Appendix
E for more details.
Fig. 5 shows the result of minimizing MSE, NLPD MS-SSIM over uniform data. Clearly, the networks trained with perceptual distances NLPD and MS-SSIM perform significantly better. The mean PSNR at 0.25 bpp over the Kodak dataset for the network trained using MSE 13.17, for MS-SSIM is 18.92 and for NLPD is 21.21. In Appendix D we report results for each network evaluated with respect to PSNR, MS-SSIM and NLPD at 0.25 bpp and at 0.50 bpp. Using these perceptual distances significantly reduces need for data. Although we have not explored how the amount of data scales affects the reconstructions, the extreme having no data from the distribution of natural images but still obtaining reasonable reconstructions is clear indication of the relationship
8

---

Published as a conference paper at ICLR 2022
(a) (in ||x R
(b) mix NLPD(x, 8)
(inu(s) MS- MS-SSIM(x. R)
Figure 5: Illustrating observation 6 The perceptual distance provides regularization when one has no samples from the training distribution. Decoded image (compressed at 0.25 bpp) encoded with networks trained using data from uniform distribution and Euclidean vs. perceptual losses.
between perceptual distances, statistical learning and probability.
FINAL REMARKS
Discussion Statistical learning and perceptual distances are related through their dependence on the probability distribution of the input data. While for statistical learning this dependence is direct and by construction, for perceptual distances the dependence comes from the evolution of human vision to efficiently process natural images. We are not the first (at all) arguing that human perception has to be related with the statistics of the natural images. However, the results presented here explicitly compare reasonable perceptual distances with accurate models of the PDF of natural images for the first time.
Taking inspiration from equalization based perception models, we show that the sensitivity of perceptual distances is related to the distribution of natural images (Sec. 2). This is further shown experimentally in Appendix A We also show that the distances induced by an autoencoder are correlated with the distribution used to train it. In particular, similarly to perceptual distances, the distances are positively correlated with the PDF of images (Sec. 3 Fig. 2}. In image compression, without imposing perceptual properties all, the sitoencoder-induccee distances present high correlation with human perception maximum Spearman correlation of 0. (Sec. [ Fig. 3). The goal in compression highly related with the efficient coding hypothesis proposed by Barlow to explain the organization of human vision. Note that a widely accepted perceptual distance like SSIM has substantially lower Spearman correlation with psychophysical MOS in the same experiment (p(SSTM, MOS) 0.63 (Ponomarenko et al. 2013
sensitivity of some of these nutoencoder-induce
while max(pt ( Din, MOS)) = 0.77).
Our observations suggest that there is double counting effect of the probability distribution when optimizing models empirically and using perceptual distances. This can be explained by the use of the data in the expectation, but also implicitly in the perceptual distance. Consequently, perceptual distances can cause the model to over-allocate resources to high likelihood regions (Sec. (Sec.(5.1) Fig. 4) On the positive side, they can act helpful regularizers the case of limited 5). By minimizing a perceptual distance over uniform noise, we show it is possible to successfully reconstruct images with compressive autoencoder at low entropy. This effect has implications the design of generative models, showing that perceptual distances can help when training with limited datasets where the the statistics of the dataset hand might not match those of natural images.
data (Sec. 5.2
Conclusion We present an empirical exploration of the three-way relationship between (a) image representations (either learned as autoencoders, or natural, as perception models), (b) the distribution of natural images, and (c) actual human perceptual distances. We describe the experimental setup we used to examine this relationship and summarize our findings in six concise observations. Naturally, our experiments have limitations, such as regarding choices of perceptual and density models (MS- SSIM, PixeICNN++, etc.). However, the presented observations represent step towards better understanding of how both machine learning and biological perception is informed by the distribution of natural images.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was partially funded by EPSRC grant EP/N509619/1, UKRI Turing Al Fellowship EP/VO24817/1, Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund under grants PID2020-11 11w.11111111BBB and DP12017-89867-C8 19987--2-2-R, and the regional grant GV/2021/074.
REFERENCES
Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool. Generative adversarial networks for extreme learned image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 221-231, 2019.
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data data-generating distribution. The Journal of Machine Learning Research, 15()):3563-3593, 2014.
F. Attneave. Some informational aspects of visual perception. Psychological Review, 61(3), 1954. doi: 10.1037/h0054663.
P. Baldi. Autoencoders, unsupervised learning, and deep architectures, In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 37-49. JMLR Workshop and Conference Proceedings, 2012.
Ballé, V. Laparra, and Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv:1611.0170,, 2016.
Ballé, D. Minnen, Singh, S. J. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In International Conference on Learning Representations, 2018.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using generalized normalization transformation. In 4th Int. Conf. Learn. Repr., ICLR 2016, 2016. URL
http:://axxit wttp:awwwww...r//////.a///. 066881
H. B. Barlow. Possible principles underlying the transformation of sensory messages. Sensory Communication, pp. 217-234, 1961.
H. B. Barlow. Redundancy reduction revisited. Network: Comp. Neur. Syst., 12(3):241-2533 2001.
A. Berardino, V. Laparra, J. Ballé, and E. P. Simoncelli. Figen-distortions. of hierarchical representa- tions. In Adv. Neur. Inf. Proc. Syst. (NeurIPS), volume 30, 2017.
D. Berthelot, C. Raffel, A. Roy, and I. Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In Int. Conf. Learn. Repr. ICIR, 2018.
S. Bhardwaj, I. Fischer, J. Ballé, and T. Chinen. An unsupervised information- perceptual quality metric. In Ady. Neur. Inf. Proc. Syst. (NeurIPS), volume 33, pp. 13-24, 2020.
C. F. Cadieu, H. Hong, D. L.K. Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James J DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS Comput Biol, 10019)x 2014.
M. Connor and C. Rozell. Representing closed transformation paths in encoded network latent space. In Proc. AAAI Conf. Artif. Intell., volume 34, pp. 3666-3675, 2020.
L. Devroye, L. Györfi, and G. Lugosi. A probabilistic theory of pattern recognition. Springer, 1996.
K. Ding, K. Ma, S. Wang, and E. P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. arXiv preprint arXiv:2004.07728, 2020.
Keyan Ding, Kede Ma. Shiqi Wang, and Eero P Simoncelli. Comparison of full-reference image quality models for optimization of image processing systems. International Journal of Computer Vision, 129(4): 2021.
10

---

Published as a conference paper at ICLR 2022
1. Epifanio, J. Gutierrez, and J. Malo. Linear transform for simultaneous diagonalization of covariance and perceptual metric matrix in image coding. Patt. Recogn., 36:1799-1811 1811,, I 08 2003.
R. A. Frazor and W. S. Geisler. Local luminance and contrast in natural images. Vision Research, 46 (10):1585-1599) 2006. doi: https://doi.org/10...
sissres.005.06... 00.0800
Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature neuroscience, 14(9): 1195-1201, 2011.
L. Gatys, A. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2414-2423, 2016. doi: 10.11009CVR.2201.2226.
B. Girod. What's wrong with mean-squared error? Digital images and human vision, pp- 207-220, 1993.
A. Gomez- Villa, Bertalmio, and Malo. Visual information flow in wilson-cowan networks. Neurophysiol., 2336))2249-2266. 2020a.
A. Gomez-Vilke A. Martín, Vazquez-Corral, M. Bertalmío, and J. Malo. Color illusions also deceive CNNs for low-level vision tasks: Analysis and implications. Vision Research, 176:156- 174, 2020b.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
A. Hepburn, V. Laparra, R. McConville, and R. RodrigeRo Enforcing perceptual consistency on generative adversarial networks by using the normalised laplacian pyramid distance. In Proceedings of the Northern Lights Deep Learning Workshop, volume 1, pp. 6-6, 2020.
Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
E. Kodak. Kodak lossless true color image suite (photocd ped0992). us/graphics/kodak, 6, 1993.
URL http://rOk.
I. Krasin et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from 2017.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
V. Laparra and J. Malo. Visual aftereffects and sensory nonlinearities from single statistical framework. Frontiers in Human Neuroscience, 9:557, 2015, doi: 3.3389/nhumm2215.005555
V. Laparra, J. Muñoz-Marí, and J. Malo. Divisive normalization image quality metric revisited. J. Opt. Soc. Am. A, 27( -64, 2010.
V. Laparra, S. Jiménez, G. Camps, and J. Malo. Nonlinearities and adaptation of color vision from sequential principal curves analysis. Neural Comp., 24(10))2551-2788 2012.
V. Laparra, Ballé, Berardino, and Simoncelli P. Perceptual image quality assessment using a normalized laplacian pyramid. Electronic Imaging, 2016(16):166, 2016.
Laughlin. A simple coding procedure enhances neuron's 'ss information capacity. Zeit. Natur. C, Biosciences, 36(9-10):910 912, 1981. ISSN 0341-0382.
Q. Li, A. Gomez-Villa. M. Bertalmio, and J. Malo. Contrast sensitivity functions in autoencoders. Accepted in the Journal of Vision, preprint arXiv:2103.0048/ 2021.
Siwei Lyu. Divisive normalization an efficient coding transform: Justification and effectiveness.
Neural Comp., 23(11):2942-299 2973, 2011.
Siwei Lyu and Eero P. Simoncelli. Nonlinear extraction of independent components of natural images using radial gaussianization. Neural Comp., 1485::445-15111 2009.

---

Published as a conference paper at ICLR 2022
J. Malo. Spatio-chromatic information available from different neural layers via gaussianization. J. Math. Neurosci., 10(1):1-40, 2020. doi: 10.1186/s13400 00086//134800200000000 020-0009--...
J. Malo and J. Gutiérrez. VI non-linear properties emerge from local-to-global non-linear ICA. Network: Computation in Neural Systems, 17(1):85-102, 2006.
Malo and Laparra. Psychophysically tuned divisive normalization approximately factorizes the pdf of natural images. Neural computation, 22(12):317--320, 2010.
J. Malo, I. Epifanio, and Fero P Simoncelli. Nonlinear image representation for efficient perceptual coding. IEEE Trans. Im. Proc., 15(1):68-85) 2006.
J. Mannos and D. Sakrison. The effects of a visual fidelity criterion of the encoding of images. IEEE Trans. Inf. Theor, 20(4):525-5366 1974.
M. Martinez, Cyriac, T. Batard, M. Bertalmío, and J. Malo. Derivatives and inverse of cascaded linear+nonlinear neural models. PLOS ONE, 13(10): 10 2018.
K. Miyasawa. An empirical bayes estimator of the mean of normal population. Bull. Inst. Internat. Statist., 38: 181 1111-188 188, 1961.
N. Ponomarenko et al. Color image database tid2013: Peculiarities and preliminary results. In EUVIP, pp. 101. IEEE, 2013.
J Portilla, V Strela, M Wainwright, and E Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans Image Processing. 12(11):1338-135 Nov 2003. Recipient, IEEE Signal Processing Society Best Paper Award, 2008.
Javier Portilla. perceptual bayesian estimation framework and its application to image denoising. In 2006 14th European Signal Processing Conference, pp. 1-5, 2006.
Martin Raphan and Eero P. Simoncelli. Least Squares Estimation Without Priors or Supervision. Neural Comp., 23(2):374 420- 02 2011.
T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. PixeICN++:: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint 2017.
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1193-121) 2001.
T. Twer and D. MacLcod. Optimal nonlinear codes for the perception of natural colours. Network: Comp. Neur Syst., 12(3):395-407, 2001.
Jonathan Vacher, Aida Davila, Adam Kohn, and Ruben Coen-Cagli. Texture interpolation for probing visual perception. Advances in Neural Information Processing Systems, 33, 2020.
A. van den Oord and B. Schrauwen. The student-t mixture as a natural image patch prior with application to image compression. J. Mach. Learn. Res., 15:2061-2086, 2014.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Comp., 2367).66611164, 07 2011.
Z. Wang and A. C. Bovik. Mean Squared Error: Love it leave it? new look signal fidelity measures. IEEE Signal Processing Magazine, 26(1):98-117, 2009.
Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In ACSSC, volume 2, pp. 1398 1402. Ieee, 2003.
A.B. Watson. Digital images and human vision. Cambridge, Mass. MIT Press, 1993.
R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEFE CVPR, pp. 586-595, 2018.
12

---

RMSE and p(x) decreases for Published as a conference paper at ICLR 2022
A ON THE RELATION BETWEEN PERCEPTUAL DISTANCE AND IMAGE PROBABILITY
A.1 DERIVING EQUATION I
Classical literature in psychophysics (Mannos & Sakrison 1974. Watson 1993 Laparra et al. 2010) assumes that the perceptual distance (subjective distance) between two images is given by the Euclidean difference between the inner representations of the original and the distorted images, S(x1) and S (222) where S(.) is the function that represents the response of the biological visual system. Given the non-trivial nature of S, the perceptual distance in the input domain non-Euclidean, and described by non-diagonal (and eventually image dependent) perceptual metric matrix, W:
Dz x2) = A ~
(x2
x1)
W. (x²- X1)
(4)
where, following Epifanio et al. 2003 scenario for simplicty:
Malo et al.
2006). W
2s
Assuming a I-dimensional 6
dim
as
Dp(01, x2)
~ ||x] - x l|2
Zo
Now, following the perceptual equalization literature (Laughlin] 1981 Twer & MacLeod 2001 Malo
& Gutiérrez 2006 Laparra et al. 2012; Laparra & Malo 2015), , the determinant of the Jacobian of the response related to monotonie function of the probability, 05 = p(x)), and then, we get Eq.
Dp(21, 22)
22)
||x - x1ll2 vmm RMSE & p(a:)?
(6)
where y > (typically y = 1 for uniformization and 7 = for error minimization) and m is the dimensionality of the data.
The fact that Eq. is a descriptor of the sensitivity of the distance is easy to see in the 1-d case. Note aD
that the n-dimensional gradient vector is (Martinez et al. 2018
ASS
Dj
as
As in the
one-dimensional case D, ||AS||2. using Eq. 5 it holds
DP
Vmh RMSE
A.2 EXPERIMENTAL ILLUSTRATION OF OBSERVATION 1
D, To illustrate Observation we evaluate the correlation between RMSE and the probability of the natural images. The results of the experiment will depend on (a) how closely the considered Dp describes human perception, and (b) how closely human perception actually equalizes p(x). In any case, note that this correlation should be compared to the zero correlation obtained if one naively assumes that the perceptual distance is just the RMSE. The illustrations consider (a) two representative measures of perceptual distortion: the Normalized Laplacian Perceptual Distance (the NLPD (Laparra et al.. 2016)) and the classical Multi-Scale SSIM (Wang et al.] 2003), and (b) a recent model for the probability of natural images, the PixeICNN+ (Salimans et al. 2017 2 Here we take images, x, from the CIFAR-10 10 dataset (Krizhevsky et al., 2009) which consists of small samples that we can analyze according to the selected image model. We corrupt the images with additive Gaussian noise such that Ax (0, o²) for wide range of noise energies. A pretrained PixeICNN+ model is used to estimate log( (ox() of all original images.
Dy Figure 6 shows the correlations, pl RMSE p(x)). together with a convenient reference to check the quality of the selected perceptual distances for natural images: we also show the correlation of NLPD and MS- with the naive Euclidean distances PDp, RMSE). In these figures, natural images live at the left end of the horizontal axis (uncorrupted samples x). Images with progressively higher noise levels depart from naturalness. The right end of the scale corresponds to highly corrupted and hence non-natural images. For natural images (those with negligible distortion), both perceptual distances are not correlated with RMSE (RMSE is not plausible perceptual distances). For natural images the correlation considered in the Observation T is high (red/gray curves). For a = 1, the Spearman correlation is 0.51 for NLPD and 0.63 for MS-SSIM compared to RMSE (which would be pl RMSE p())) = 0). The correlation between the perceptual distances and the Euclidean RMSE increases for non-natural images, but note also that the relation between Dp
RMSE
https://gitthbb.co
-onn--nnpppppp under MIT License
13

---

Published as a conference paper at ICLR 2022
progressively non-natural images.
These results indicate a strong connection between the sensitivity of perceptual distances (either NLPD or MS-SSIM) and the distribution of natural images, opposed to the sensitivity of Euclidean RMSE, which not related to the data.
pINEPO.RMSE!
Plate (oç(000)
pOANSE. logjolsti
AfL MBSSIM, KMSS)
Voae log(pta)
NLPD
MS-SSIM
06
0.4
Figure 6: Illustration of Oberservation Spearman correlations ps between the sensitivity of the perceptual distances NLPD and MS-SSIX, and log (pg(p(x)) (in red/gray). Distances computed between x, and a distorted version with additive Gaussian noise, x- Åx, with deviation a. Correlation of RMSE with perceptual distortions (in orange/brown) and of RMSE with log(p(x)) (in blue) included for comparison. MS-SSIM is similarity index, so 1-(MS-SSIM) is a distortion measure.
A.3 DETAILS OF CIFAR 10 EXPERIMENTS
Fig 7 shows how NLPD(x1.X2) and L-MS SSIM vary with log(p(x)) for each specific image and RMSE
RMSE
the two extreme situations of very low and very high noise variance. can be seen that for low variance, a = 1. the probability and the fraction of the perceptual metric are more related than for high variance. This is coherent with the idea of the Observation
0=1
50
0.26
0.33
SSIAY
0.32
NOP01X 31
0/93
0510
6003
0.20
0.12
0.55
0001
08:0
0.02
-10000 -0000
-5250 -408C -2000
-10000 -1000 -100o -4000
-2000
-10E00
00025
-90Eo -4000 -2000
-10000 -000 -e3ce -1000 -2003
(opppi))
(og(200)
og(ptx]
(a) NLPD
(b) MS-SSIM
Figure 7: Scatter plots showing (x) and perceptual distances NL.D and MS-SSIM between images from CIFAR-10 with additive Gaussian noise for a 1,50
Fig. shows the distribution of log(p(x + (o(O for a = 20,1,2,5, 10, 20, 30, 40, 50). A pretrained PixeICNN+ model used to evaluate log(p(x)). It is clear that when more noise is introduced in the images the probability of being a natural image decrease (As expected).
14

---

Published as a conference paper at ICLR 2022
1200
x x+ 12)
1000
x 24)
x 5
x x
600
x 200,
x 300
400
x 400,
200
x+ 500)
116000
-14000
-12000
-10000
-8000
-6000
-4000
-2000
Figure 8: Distributions of log(p(x o2) for various where log(p(x)) is estimated using PixeICNN+ model.
B
SELF-RECONSTRUCTION DISTANCE AND SCORE MATCHING IN AUTOENCODERS
In (Vincent 2011) for denoising autoencoders and in (Alain & Bengio 2014) for contractive autoen- coders is stated that:
ôlog(p(x))
δx
or equivalently x X p(x) ex
p(p(x)) From this last formula we can already that the difference
between the original data and the reconstructed data is proportional to the inverse of the probability,
as stated Obs. (just ignoring the term d(p(x)) for now).
6x
In particular for the Gaussian distribution we have that the score matching:
Ölog(p(x))
Ox -
(8)
has monotonical relation with the inverse of the probability (see Fig. 9
Score matching has been proven to be more general than the Gaussian noise scenario denoising autoencoders (Alain & Bengio 2014), and our observation (based Eq. 2 is even more general than this because Eq. 4 is not attached to any particular noise/metric. Our observation does not have the same analytical expression than the score matching result, but we see is consistent with the score matching solution in the high probability regions (around the mode of the PDF, see Fig. (gg.9).Our observation observation for D, and the score matching result only differ the low probability regions. However there are several things to take into consideration. The obvious one is that we are going to have few data points that are going to fall in low probability region. This is important not only in evaluation terms but also during the training procedure of the autoencoder. Note that the autoencoder not going to be properly trained these regions and therefore will tend to interpolate what has learn in medium probability regions. For instance the autoencoder going to try project the data that from very low probability regions into the manifold, therefore the distance between the original data and the reconstructed data (|xx xl|2) going be high (although the theory, Eq, 7 inferr that should be extremely low).
15

---

Published as a conference paper at ICLR 2022
Gaunston uoa,
Generalized Gaussian
I-studient JoB, voBi1
Isttudent unt 05
Gaussian Scale
022
r20.25,0.5112
112
0.1
0.08
0.1
lngiptal)
logletst)
legjpt-i)
logletet)
Iogiptzi)
-1
Optinaal Devistion Denoising Autoencoders
Optimal Deviation Denoising Autoencodere
Optimal Deviation in Denoising Autoencoders
Optimaal Deviation in Denoising Autoensoders
Optimal Deviation Dencieing Autoenooders
0:04
di eppaciei
0.05
0:22
0.01
[200m] Deviation
0C1
[Zoom] Deviation
0.1
[Zoon]] Deviation
201
[Zomm] Deviation
0.01
Zoon] Deviation
3.5
0
0.5
0.5
0.5
2.5
0.5
0.5
0
0.5
0.5
055
Figure 9: Magnitude of the deviation introduced by denoising autoencoders in Gaussian PDF (first column) and in different PDFs proposed to model natural images (rest of the columns). The score in the mode of the distributions. Zoomed-in in the last row.
matching, [log(p(x)) is correlated to p(z)
:x
C 2D EXPERIMENTS
Throughout the paper, 2D example is used to explore the various observations in a simplified setting. Here, details can be found on the exact experimental setup.
We define the probability density function of our 2D Student-t distribution as
f4 + at
where we use μ4 (8). 0.5
0.2 and t, are samples from 2D Student-t distribution with
degrees of freedom v = 2. A dimension with high variance and one with low variance were chosen to mimic the low and high frequency bands in natural images.
3-yeyer multilayer perceptron (MLP) is used for e and d, where the dimensions through the network [2 100 100 2] for and the reverse for d. The quantisation used is rounding to the nearest integer, which is approximated during training with additive uniform noise as uniform scalar quantiastion the gradients are zero almost everywhere. Fig. I0 shows the architecture of the autoencoder. Softplus activations are used on all hidden layers but omitted from the final layers to not restrict the sign of the representation reconstruction. The network is optimized using Eq. 3 using different À values to achieve different rates. Adam optimizer was used, with a learning rate of 0.001, batch size of 4096 and 500,000 steps where for each step batch is sampled from the distribution. The large batch size was to account for the heavy tailed distribution used. Fig. 11 [hows a) samples from the 2D Student-t distribution b) density of the distribution and c) resulting compression a rate of 2.65 bits per pixel (bpp) where the dots represent code vectors and lines represent quantisation boundaries.
In Sec. 5.1 we hypothesize that one can cause model to focus high probability regions by including the probability in the loss function. In order to see the effect of including probability in the distortion term, similar to what we hypothesize perceptual metrics p(x)". D as the distortion term, with y = {-0.1, O, 0.1). 7 was chosen to have a small magnitude to guarantee stable training and avoid the model collapsing to code vector assigned at the center
doing for images,
16

---

Published as a conference paper at ICLR 2022
e
d
Dense 2x100
Dense 100x100
Dense 100x2
Dense 2x100
Dense 100x100
Dense 100x2
Output
Input
Softplus
Softplus
AE
Softplus
Softplus
*
AD
Figure 10: The network architecture for the 2D example where the encoder, d the decoder, Q is the quantisation step, AE is a arithmetic encoder and AD is the arithmetic decoder. The quantisation Q used is rounding the nearest integer, which approximated by additive uniform noise training time.
DPP: 2.65 SSC: 0.73
10
Sourcce
-10
-20
-16
-20
30
dimonsion
(a) Samples
(b) Density
(c) Compressed representation
Figure 11: The 2D Student-t distribution used throughout the paper and one compressed representation found by minimizing Eq. 3 For c) the lines represent quantisation boundaries (within bin, all points are compressed to the same point by e and the dots represent code vectors (where these points are projected to by d.
of the distribution. Thus the loss functions to optimize for are L1 R
XD
(9)
C2 R p(x) XD C3 R p(x) -0.1 XD
(10)
R
(11)
where the À parameters are tuned so that the networks have similar rate ranges and for D the sum of squared errors (SSE) used. Fig. 12 shows examples compression resulting from optimizing each of the 3 loss functions. The figure clear shows that when multiplying the distortion by p(x)0. 0.1 the quantisation bins and code vectors concentrate on the support of the distribution, whereas multiplying by (0)x=0.. enforces more uniform distribution of code vectors.
In Sec. 5.1 we observe the performance gain in using pxx) in the loss function. For a network, given set of points to evaluate we take the general performance to be the distortion over the rate,
D
P
(12)
R
Note that this is sort of normalization in order to compare networks that compress different rates. For the 2-D example, the gain in performance in including p(x)0.1 the distortion term can then be defined as
P1
(13)
PS where Pi is the performance of network trained with loss L1 (Eq. 9) and Py is the performance of network trained with loss L2 (Eq. 1O). For the experiment in Sec. 5.1 the relative performance is calculated for points samples along the positive x-axis, (x, ([O, 0) and the results shown in Fig.
In Sec. 5.2
show that autoencoders can be trained without data. Instead, provided with direct
17

---

Published as a conference paper at ICLR 2022
BPP: 2.65
SSE: 0.73
BPP. 2.65
SSE: 0.76
BPP: 2.59
SSE: 0.86
dimersoon
source dimensoon
-20
-10
SCurCH devensionn
sourcn dimensionn
snuFTO dimensiann
XD
(b) L2 R + p(x) 0.1
XD
(c) L3 = p(x)
Figure 12: Resulting compression when using the 2D Student-t distribution and including the
probability distribution in the loss function. b) is seen as including information about the distribution in the loss function and is removing it. BPP is bits per pixel (the rate) and SSE is square errors (distortion) evaluated on a validation set.
BFP: 10.07
SSE. 1.32
BPP: 2.99 SSE: 0.95
70
-00
-10
10
20
-00
-15 10 49 70
(a) mila R +
D
(b) min P. (x) R
Figure 13: Resulting compression when using a 2D uniform distribution across the space, where left) optimized the rate-distortion equation over the uniform distribution and right) optimized for the probability belonging to 2D Student-t weighting the rate-distortion equation over the uniform distribution.
access or proxy to the probability distribution, one can achieve reasonable results when training over samples from a uniform distribution. Fig [33]shows the resulting compression using the 2D Student-t distribution where the loss that has been minimized is minx-24 L = (x) (R + XD), where (x( denotes the probability of point x belonging the 2D Student-t distribution. This training procedure requires no data sampled from the Student-t distribution, only access to evaluations of the probability and results in code vectors assigned to the support of the distribution, namely the x-axis which has the highest variance for the 2D student distribution. The loss functions contains information on the probability distribution and thus does not require samples.
In Fig 4 we show the curves of the relative performance of the 2D example. In order to facilitate the visualization we use 20 degree polynomial fitting to soft the curve. Fig we show the result after and before the fitting. Note that the fitting makes the reconstruction error of close points since point close to the code vectors will be reconstructed much better than points from the same region but far from the code vector.
since there will be variability between
18

---

Published as a conference paper at ICLR 2022
2.2
2.25
Relative Performance
Relattve performance
2,00
1.8
1.75
(ddg) a1e8
1.6
[dig])
1.4
1.2
1.25
1.00
0.8
0.75
High Probability
Low Probability
High Protebility
Low Promability
(a) degree polynomial fit each curve.
(b) Raw values.
Figure 14: Relative performance of networks for samples along line through the support of the respective distributions. Left: networks trained with p(x) | - -x2l2 in Eq. 3 divided by performance of networks trained with D ||x| x2113 on the 2D Student-t and evaluated using samples along r-axis. Left) shows 20 degree polynomial fit to each network and right) shows the raw values.
D IMAGES
D.1 COMPRESSION AUTOENCODER
In Sec. 5.1 pretrained autoencoders are used, namely the factorized prior model from (Ballé ct al. 2018). This model has layers of convolution operations and generalized divisive normalization activations, the architecture show in Fig 15
Q
21
Conv2d Ne5xS 24
Con-2d NxSe5 21
21
Conv2d Nx5x5 21
Comv2d 3x5n5 21
Input
Conv2d Ne5x5
GDN
Conv2d Nx5x5
GDN
GDN
Conv2d Mr53x5
AE
IGDN
Conv2d Nx5e5
1GDN
IGDN
x
Output
AD
Figure 15: Architecture for networks used in Sec. 4 & 5.1 which is the the factorized prior model from (Ballé et al. 2018)) where is the encoder, d the decoder, Q is the quantisation step. AE is arithmetic encoder and AD is the arithmetic decoder. The quantisation Q used is rounding to the nearest integer, which is approximated by additive uniform noise at training time. GDN denotes a generalized divisive normalization activation, and Conv2d is a 2-d convolution operation. The convolution parameters filters kernel height kernel width down- or upsampling stride. For the 5 lower bit rates, N 128, M = 192 and for higher rates N 192, M = 320.
We take the performance gain again Eq. 13 where Pi the performance of network trained with using MSE as the distortion DD and P2 is the performance of network trained with using MS-SSIM as distortion (akin to p(x) -D in the 2D case).
D.2 SAMPLING THROUGH DISTRIBUTION IMAGES
Starting from the center of the distribution (high probability), direction on the support of the distribution chosen and samples are generated along this line. For the 2D distribution, we use (x, v) (10,35), 0) to generate samples along the x-axis. Fig. 4 shows the proportion defined earlier for samples generated along the x-axis. With the 2D example we have explicit access to the probability distribution, luxury
not afforded when it comes to images due to the probability distribution being intractable. Drawing a line through the distribution of natural images in the same way as in the 2D example would be ideal, although infeasible. However, it has been shown that
19

---

Published as a conference paper at ICLR 2022
contrast as lower-contrast images more likely (Frazor & Geisler 2006) and such axis to sample through the distribution of natural images. For each image in the Kodak dataset low-contrast and high-contrast version is generated. Samples interpolation between the low contrast and original, and the high contrast and original for all images in the Kodak dataset (Kodak 1993). For each image in the Kodak dataset, high contrast version Xhigh and low contrast version Xlow are created. Linear interpolation is used to get gradual shift from low contrast-original-e contrast.
then taken between a linear
(1-o0 a) x + Xhigh if > 1
(14)
(11 + a) x a Xlow if E 1
for x images in the Kodak dataset and a E [0. 2]. where a = denotes the original image. 200 a
0.0
1.0
2.0
Figure 16: Example of altering the contrast of an image from the Kodak Image dataset varying a: in Eq.
values sampled. Fig. 16 shows example for samples generated for one image. We compare networks with probability included without. The ratio defined earlier will be used, where P1 is the distortion/rate for networks optimized with MS SSIM and Po is distortion/rate for networks optimized for MSE. The ratio defined in Eq. 13, where the numerator denotes networks optimized using perceptual metric MS-SSIM and the denominator denotes networks optimized for MSE. All networks were pretrained and taken from the Tensorflow Compression package. Fig, shows the ratio of performance as we vary a.
loss function, the form of perceptual distances, to those
E ENTROPY LIMITED AUTOENCODERS
For ease of training, in Sec. 5.2 on the entropy and just minimizing for the distortion as in (Ding et al.) 2021 Agustsson et al." 2019).
simplify the rate-distortion loss function by setting upper bound
d
Convid Nx5x3 2
Convod Nx5a3 21
Conv2d Nx6x3 2
Cony2d Nx3x3 21
Conv2d Nt5x3 2t
Conv2d Nx5x5 21
Conv2d Nx5N3 21
Conv2d Sx5x3 21
Comy2d Mxxx3
Conv2d NecwS
Input
CDN
CDN
CDN
AE
IGDN
ICDN
ICDN
IGDN
Output
AD
Figure 17: Architecture of networks used in Sec. 5.2 where is the encoder, d the decoder, Q is the quantisation step, AE is arithmetic encoder and AD is the arithmetic decoder. The quantisation Q used is rounding to the nearest center defined by T., which is approximated by Eq. 15 at training time. GDN denotes generalized divisive normalization activation, and Conv2d is a 2-d convolution operation. The convolution parameters filters x kernel height x kernel width down- upsampling stride. For all rates N 128, M =
Setting upper bound on the entropy of the encoding with I. centers C1, CL. the soft differentiable
20

---

Published as a conference paper at ICLR 2022
approximation is
cxp(- sspi - -c,)²
i Σ
2-1 E exp(- - +(y cx )2) Cj
(15)
where is the quantisation scale parameter which we fix to 1. Given that we know the dimensionality of y and a maximum of L. integers that can be represented, obtained;
upper bound the entropy can be
W x H
(v:)
(L))
(16)
2 2n
where [H, W] are dimensions of the image, n is the number of downsampling layers you have in the network with stride 2, m the number of channels in your embedding and L. is the number of centers the embeddings
rounded to.
For all experiments in Sec. 5.2 an architecture used with 5 convolutional layers and GDN layers in both and d. Fig. 17 shows the entire architecture. In this case, the quantisation is performed by rounding values to the centers defined by L. Table Ssows the results training these autoencoders using different training distributions, distribution of natural images P(x) and uniform distribution U(x) and using different loss functions, perceptual metrics NLPD and MS-SSIM and the MSE. networks for each distribution and loss was trained, one compressing and another 0.5bpp. Probably the most interesting part of this table is shown in the PSNR column. Note how for high entropy the network trained to minimize NLPD gets better performance in PSNR than the one trained for MSE (which is basically PSNR). This effect matches with the double counting effect proposed in sec. S The idea is that minimizing the NLPD is related with minimizing the expected MSE which is what is at the end being evaluated. This effect is much more visible in the same column when using the uniform distribution for training. In this case the improvement when using NLPD is clear regard the MSE. This can only be the case if the NL.D has some properties of the distribution of the natural images.
an upper bound of 0.25bpp
Table 1: Evaluating autoencoders trained with both uniform distribution ((x) and the OpenImages dataset (Krasin et al., 2017) P(x) two different rates. Reported are the PSNR, MS-SSIM and NLPD evaluated for all networks. These networks use the approximate in Eq. 15 Bold values indicate the best value for each evaluation metric at a certain rate (bpp) with a certain distribution.
Bits per pixel
Distortion Loss
Training Distribution
PSNR
MS-SSIM
NLPD
(bpp)
MSE
29.39
0,9755
2.615
0.25
MS SSIM
26.91
0.9754
4.080
NLPD
27.85
0.9655
2.510
P(x)
MSE
29.73
0.9777
2,450
0.5
MS SSIM
27.53
0.9827
4.139
NLPD
30.13
0.9809
1.792
MSE
13.17
0,4555
16.50
0.25
MS-SSIM
18.92
0.7483
10.43
NLPD
21.21
0.7396
8.090
u(x)
MSE
13.15
0.4396
16.89
0.5
MS-SSIM
18.90
0.7659
10.37
NLPD 20.59
0.7335
9.132
Fig. 18 shows visual results of the effect in the reconstruction in an image from the Kodak dataset using each of the networks compressing to 0.25bpp and 0.5bpp and trained using natural images as input samples. i.e. p( x). In this case results when using a non perceptual metric as the MSE or perceptual metric (NL.P PD or MS-SSIM) are very similar. Even than the correlation of NL.D and MS-SSIM with human perception is much higher than the one of MSE. This is because optimizing to minimize the expected MSE over natural images impose propertics of the distribution of the natural images in the autoencoder (as shown in see 4 and this translates kind of perceptual behavior (as shown in sec 2
On the other hand Fig. 19 shows the reconstruction results when training the autoencoders using data coming from a uniform distribution, i.e. the networks do not see any natural image during the training. However the networks trained using NLPD and MS-SSIM obtain good performance when
21

---

Published as a conference paper at ICLR 2022
evaluated in natural image. While, expected, the network trained to minimize MSE obtains very bad result.
22

---

Published as a conference paper at ICLR 2022
(a) Original
(b) min ||x
2 at 0.25 hpp
(c) minp(x) ||x
8112 at 0.5 hpp
(d) min NLPD(x, X) 0.25 bpp
(c) minp(x) NLPD(x,, &) 0.5 hpp
minp(x) MS SSIMSSIIMIIIS. &) 0.25 bpp
minp(x) MS-SSIM(X, &) at 0.5bpp
Figure 18: The reconstruction of image from Kodak dataset from the various networks trained to compress to a maximum entropy bits per pixel (bpp) specified optimized using the OpenImages dataset (Krasin et al. 2017).
23

---

Published as a conference paper at ICLR 2022
(a) Original
(b) minzrge x
2||2 at 0.25 hpp
(c) mine(()) ||x
8112 at 0.5 hpp
(d) minure NLPD(x,, x) 0.25 bpp
(c) minu(s)) NLPD(x,, x) 0.5 bpp
(1) minute) MS-SSIM(X, &) at 0.25 bpp
gg) ming(x) MS. SSIM(XI,.I..... &) 0.5bpp
Figure 19: The reconstruction of image from Kodak dataset from the various networks trained to compress to a maximum entropy bits per pixel (bpp) specified optimized using random uniform noise.
24

---

Published as a conference paper at ICLR 2022
E. TRAINING WITH SMALL BATCH SIZES
In Sec. Sec. we explore what happens when one does not have direct access to the data. A more relaxed version of this is having access a small amount of data time; using small batches of images.
For small batch sizes, the minibatch estimate of the gradient over the whole set of images will have higher variance and more likely to be effected by outliers. From Sec. |. we can see that using a perceptual distance is similar to multiplying by the probability, which would weight gradients calculated from outliers less. In theory, this should lead more accurate estimate over several minibatches for perceptual distance rather than MSE. For stochastic gradient descent, minibatches of data are used to estimate the expected loss and aim to minimize the generalization error (Goodfellow et al. 2016). The exact gradient of the generalization error with loss function L over model / is given by
(17)
In gradient descent, g is estimated by sampling batch of size m from data distribution p(x) and computing the gradient of the loss with respect to f ma
g V, Σ [(fxx))xx.).
(18)
Gradient updates are then performed using this g. Given this estimator, we can quantify the degree of expected variation in the estimated gradients using the standard error of mean
m
SE(gm)
Var V > -(f(xx).xx)
(19)
Vm
where o² is the true variance of the gradient of the loss function for point x;, Given that the true variance o² is unknown, if batch size m small then a loss function with low variance will give us a better estimate of the generalization eтror and therefore a better estimate of the gradient too. When performing stochastic gradient descent (SGD) using extremely small batch size, e-g. 1, the gradients have higher variance as your estimation of your weight updates become more inaccurate when attempting to estimate the ideal weight update for the entire training set. For example, with a batch size of weight updates that come from outliers have the potential to move the weight vector far away from the optimum value it was in the process of meeting. In order to reduce the effect of outliers, one would need explicit access to the underlying data distribution. This is exactly the type of regularization that perceptual distances can perform. Given that they are proportional to the probability distribution of natural images (Section 2 the perceptual distance between an outlier and it's reconstruction will be weighted less than images coming from high density regions, and thus achieve a lower standard error for the estimator of the generalization error (Eq. 19) compared to Euclidean distances like MSE.
Tho illustrate this effect we performed an experiment where we train the same network using a batch size of one to minimize MSE and one is trained to minimize NLPD. We evaluate them on the expected MSE over the Kodak image dataset (Kodak) 1993). This means that for the network optimized for NLPD, the loss function we are minimizing and the function we use to measure generalization error are different. This quantity can be evaluated after every batch of size and Fig. reprrss the ratio of generalization error over the test set for the network trained for NL.D over MSE. Random sceds are fixed so that the two networks are given the exact same initialization and same order of images. This autoencoder acts purely as dimensionality reduction and has no quantisation in the embedded domain. It follows the same architecture as in Fig. 17 without the quantisation, AE and AD steps. 5 different random seeds were used, i.e. different network initialization and different order of training images, and reported is the mean and standard deviation across runs. The true expected MSE over the test dataset is given as Erest M EE,, x). This quantity is evaluated after every batch of size and Fig. reports the ratio of the expected MSE over the test set for the network trained for NLPD over MSE.
On average, the network trained using NLPD achieves a lower expected MSE than the network optimized for MSE, even though we are evaluating using the same function as it's loss. This ability to estimate the test set expected MSE is indicative of NLPD being getting a better estimate of the gradient of the generalization error.
25

---

Published as a conference paper at ICLR 2022
1.2
ErerMSE Ho)
10000
20000
30000
40000
50000
60000
70000
Batch
Figure 20: Gain in using NLPD over MSE loss function evaluated in terms of MSE loss test (Kodak dataset) using batch size of and a small learning rate, fixing random seeds. XNLPD denotes the reconstruction of x with a network optimized for NLPD,, and &MEE for network optimized for MSE. The mean (solid line) and standard deviation (solid fill) was taken over runs with different random seeds, ie. different network initialization and training image ordering. The dashed line represents if the two networks had the same expected MSE on the test set.
26


Reviews:
[{"review_id": "h2Z0NhoA5CM", "reviewer": "Reviewer_2c14", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper was nice to read. Overall there are six (6) observations and 3 Eqns in main body of the paper, which formed the core part of the work, while 13 more in Appendices were supporting as proofs and extra information.\nSpecific contributions are: \n(i) Pixel-CNN++ based image likelihoods are correlated with human psycho-physical distances;\n(ii) Distances induced by AEs are correlated with Prob(Trng data), as well as with human perception.\n(ii) double counting effect in perceptual distance based loss function\nOverall, as nice 3-way relationship being empirically explored.\n\n\nCons:\nLet me start with some generic queries first:\nI would like a comment from authors- these 6 observations are like soft theories (ala Perception theories), \nor they can be raised to the level of Results/Lemmas (in applied maths )?\nWhat can be the overall conclusion for theorists and experimental application domain experts, from these 6 (+ 3) contributions ? \nHow can one exploit them for better design of DL architectures/algos. to solve certain problems ?\nI am trying to get an overall picture, if possible, of what is the gain in technical know-how or analytics this paper has produced.\n\nThe words Perceptual and psycho-physical distances - are they meaning the same ? A reference would be ideal.\n\nYou have used AE as your target application, what about other deep-CNN models - transformers, LSTM, GAN based models for generation or discrimination?\n\nWhat does one need to do to counter this double-counting effect ? Modify loss function, dataset samples or  architecture ?\nIt appears that you have identified a tricky fallacy, but the remedy has not been explored ? Or, have I missed something.\nYou mention using training without image data - using uniform random noise as input- is this not same as various GAN-based models ?\n\nEqn (2) appears to be a special/modified form of Vapnik's theory of empirical risk minimization (ERM), applied to VAE ? What is the value addition then?\n\nEqn (3) - Ex is not defined ? Or I missed - although appears to be the trivial  - Empirical mean ? (I assume).\n\nAny particular reason of Using  softplus (yes, recently popular) rather than softmax ? Pixel-CNN+ also uses softmax. \n\nsec. 5.2, 1st para, pp 7 - this part of the (long) sentence may need rewording:\n......minimizing a loss weighted by the \"likelihood the samples\" belong to the Student-t,....\n\nSame within Sec. 6, 1st para, pp 8:\n......by construction, for perceptual distances we are , considering that the human visual system has....\nNow, this evolution process has been over a very long time, say more than few centuries  - and with no empirical results feasible/available involving ancestors of humans - my opinion is that:\n - this hypothesis used by visual perceptions scientists may be kept aside, in the context of this paper.\n\nThe last sentence in Conclusion section, pp 9, says:\n....both machine learning and biological perception is informed by the distribution of natural images.\n\nWell as scientists and engineers, should we not be more interested in the gap between the two, rather than the commonality ?\nThat would help us to bridge the gap and build more efficient, robust, autonomous, intelligent perception-machines, I believe, \nas our main target to design.\n\n\nWell the term \"Appendices\" is missing as a sub-heading before Appendix A - left me initially wondering if these were supplementary materials or Appendices of the paper. or extension.\nI am not happy with the purple background used in figures 11-13. Quite difficult to isolate out the central part (hoping that is significant?)  visually - eye-stressing I must admit.\nFigs. 18-19: traditional Image quality experts provide a zoomed-in part of pics to highlight the distortions. The overall pictures here look bland, and quite difficult to identify regions/parts of images with good vs bad quality (as evaluated often using SSIM or PSNR) - mostly, the pair in last row, \nThis is not the case for samples in Fig. 16.\n\nSec. E- Entropy limited AE: is this your idea, or a variant of any used elsewhere  ?\nDid not see a reference - hence checking.\n\nI enjoyed reading the paper, which had a nice flow, although a bit long (as per expectations in a Conf.) \ndue to supplementary/Appendices.\n\n", "summary_of_the_paper": "\nThe work presented in this paper aims to analyze the relationships between the probability distribution of the data, perceptual\ndistances, and unsupervised machine learning. Perceptual sensitivity is correlated with the probability of an image \nin its close neighborhood. The paper also explores the relation between distances induced by autoencoders and the\nprobability distribution of the training data, as well as how these induced distances\nare correlated with human perception. At the end, the paper specifies that perceptual distances do not\nalways lead to noticeable gains in performance over Euclidean distance in common\nimage processing tasks.", "summary_of_the_review": "\nThe paper has provided sufficient and substantial evidence of the observations made of the 3-way connections between - \nlearned or hand-crafted image representations (autoencoders and perceptual models), the distribution of\nnatural images, and human perception.\n\nThis paper may initiate good discussions/work along the domains overlapping visual perception and ML/DL scientists .\nIn terms of technical know-how, although the paper takes a small step, but does not really contribute to a big\ngain in analytics and new formulations/algorithms.\n\nAlthough there are some unanswered questions and issues, the paper appears quite interesting.\nMinor improvements would be necessary for it to get accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XEhGsithtlc", "reviewer": "Reviewer_Rmvg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper is dense, I found it nevertheless to be well structured and the figures to be generally helpful. While at first I thought that the result might be trivial (the human visual system compresses natural input, so it makes sense that autoencoders and perceptual distances are related), I believe that the paper draws out the value of making these connections explicit --- though I am not an expert in the theoretical side of this field. I have minor comments.\n\n- The observations presented are limited to small image differences $\\delta$. I would be interested to know (1) if the theory can be extended to estimate bounds on $\\delta$ in some meaningful dimension and (2) the authors' speculations on whether the assumption of perceptual distances as metric spaces breaks down when $\\delta$ is large (see e.g. Tversky, 1977, *Features of Similarity*), as in the suprathreshold regime comparing different types of image manipulations.\n\n- Could the \"double-counting\" effect be mitigated by reweighting the relative contributions of perceptual loss and training?\n\n- Figure 3: why is the RMSE correlation with MOS (dashed line) flat as a function of bpp? \n\n- Figure 4: there seems to be quite some kernel smoothing happening here, making it unclear how much data this actually represents.\n\n- P. 8 typo \"for perceptual distances we are ,\"\n\n- A relevant reference for the introduction linking efficient coding and compressed unsupervised representations is: Storrs, K. R., Anderson, B. L., & Fleming, R. W. (2021). Unsupervised learning predicts human perception and misperception of gloss. *Nature Human Behaviour.*", "summary_of_the_paper": "This paper presents a mainly theoretical explication of the relationship between natural image statistics and perceptual distances for small image distortions. The paper presents a number of observations linking distances in natural images, autoencoders, and perceptual similarity for humans. The paper finally explores some implications of these observations, including impressive-seeming results on training with no data using perceptual distances as regularizers.\n", "summary_of_the_review": "A theoretical explanation of the link between natural image statistics, autoencoders and perceptual distances, with some seemingly important implications. This is not exactly my field so I may be missing relevant background.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O92p9ItcWyE", "reviewer": "Reviewer_z8xf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe authors investigate a very interesting and important problem that lies in the intersection between perception and statistics.\n\nWeaknesses:\n\nFor Observation 1:  $\\frac{D_p(\\mathbf{x}_1, \\mathbf{x}_2)}{\\Vert \\mathbf{x}_2 - \\mathbf{x}_1\\Vert_2}$ is symmetric. Therefore, it should be correlated with both $p(\\mathbf{x}_1)$ and $p(\\mathbf{x}_2)$. One subtlety is one image (e.g., $\\mathbf{x}_2$) is a \"distorted\" version of the other (e.g., $\\mathbf{x}_1$). In other words, $p(\\mathbf{x}_1)$ is the distribution over undistorted data and $p(\\mathbf{x}_2)$ is the distribution over distorted data. Does it make sense that $\\frac{D_p(\\mathbf{x}_1, \\mathbf{x}_2)}{\\Vert \\mathbf{x}_2 - \\mathbf{x}_1\\Vert_2}$ is correlated with two different distributions. Note that although Observation 1 assumes $\\Vert \\mathbf{x}_2 - \\mathbf{x}_1\\Vert_2$ is small, $p(\\mathbf{x}_2)$ can be a distribution over very perceptually noisy data.\n\n\nFor Observation 2: This observation relies on the idea of score matching, which is derived under a particular type of distortion -  additive white Gaussian noise. Does it generalize to other types of non-additive, non-linear distortions, e.g., JPEG compression, Gaussian blurring, etc.\n\nFor Observation 3: The reviewer believes this observation will largely depend on the capacity (i.e., information bottleneck) of $f$. One extreme counterexample is that $f$ has no capacity and is a constant function.\n\nFor Observation 4: $p(\\mathbf{x}) = p(e(\\mathbf{x}))\\left\\vert\\frac{\\partial e}{\\partial \\mathbf{x}}(\\mathbf{x})\\right\\vert$ holds only for bijective mappings with identical input and output dimensions. This is not the case for compression. In addition, comments on Objective 3 can also be applied here.\n\nFor Observation 5: The reviewer respectively disagrees with this observation. For $D_\\mathrm{in}(\\mathbf{x}_1,\\mathbf{x}_2) = \\Vert e(\\mathbf{x}_1) - e(\\mathbf{x}_2)\\Vert_2$, it is not hard to find, for example using gradient-based optimization, a distorted image $\\mathbf{x}_2$ with the same compressed representation as the original image, $\\mathbf{x}_1$ of perfect quality, that is, $e(\\mathbf{x}_1) = e(\\mathbf{x}_2)$. This image may not appear in TID 2013, on which the authors justify Observtation 5.\n\nMoreover, it remains to be seen whether PixelCNN++ is a good generative model to approximate $p(\\mathbf{x})$, and how the approximation error affects the five observations.\n\nA final concern is that there is a lack of experiments/discussions on how useful these observations are for designing better generative models and/or better perceptual distances.\n\n\n", "summary_of_the_paper": "Interesting (but debatable) observations between image prior, image quality, and perceptual distances.", "summary_of_the_review": "See above for detailed comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1ug4LiQ4hLn", "reviewer": "Reviewer_CZqT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths :\n\n* Interesting topic at the cross-road of perception and machine/deep learning (ML/DL)\n* Good review of the literature\n* Well written and understandable\n\n#### Weaknesses :\n\n* No major contributions\n* List of observations that are already known in the ML/DL community\n* More like a review paper \n\n### Detailed comments\n\nOverall reading this paper is interesting as it gathers two/three fields: ML/DL, neurosciences and psychophysics. However, I don't really see any contribution to any of these fields. \n\nFor ML/DL: most of the observations are already known. No one in the ML/DL community would be surprised to hear that distances between images are correlated with their distribution. Gans, normalizing flows or NeuralODEs are approaches that takle the same goal: mimicking the dataset distribution using a latent space representation. \n\nNeurosciences: No data, nor models are provided.\n\nPsychophysics: No new data is presented. Nothing new is presented about the relation between perceptual distance and human perception. \n\nI fail to understand why the authors introduces so many distances D_s, D_r, D_e and D_in and why it is interesting. In addition, why is it insteresting to state all those observations (that are variant from each others and somehow all result from the efficient coding hypothesis) ? \n\nTo finish, on a better note. I think, the authors are working on an interesting topic and I encourage them to pursue. However, I also think this paper is too preliminary. To me the most interesting contribution (which is only a small part of the paper) is the training of auto-encoders with noise and a perceptual similarity distance. Really, is it useful for training with less data ?\n", "summary_of_the_paper": "The authors present few relation between statistical learning and perceptual distances. Specifically, they explain that\n\n1. perceptual distances correlate with image likelihoods;\n2. auto-encoder latent space induced distances of natural images are correlated with training data probability and human perception; \n3. perceptual distances are redundant with Euclidean distances in latent space.\n", "summary_of_the_review": "I think it is more a review paper and there are not important enough contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers suggest acceptance of this paper, which reports the relationship between perceptual distances, data distributions, and contemporary unsupervised machine learning methods.  I believe this paper will be of broad interest to different communities at ICLR.
Decision: Accept (Spotlight)