Perceptual Scales Predicted by Fisher Information Metrics
Authors: Jonathan Vacher
Pascal Mamassian
Venue: ICLR 2024 poster (2024)
Keywords: perceptual scale
perceptual distance
gaussian texture
naturalistic textures
texture interpolation
Primary area: applications to neuroscience & cognitive science
TL;DR: The predictions of perceptual scales based on Fisher information metrics are tested in a series of experiments. This will allow us to go beyond perceptual distances and get closer to perceptual geometry.

Abstract: Perception is often viewed as a process that transforms physical variables, external to an observer, into internal psychological variables. Such a process can be modeled by a function coined *perceptual scale*. The *perceptual scale* can be deduced from psychophysical measurements that consist in comparing the relative differences between stimuli (i.e. difference scaling experiments). However, this approach is often overlooked by the modeling and experimentation communities. Here, we demonstrate the value of measuring the *perceptual scale* of classical (spatial frequency, orientation) and less classical physical variables (interpolation between textures) by embedding it in recent probabilistic modeling of perception. First, we show that the assumption that an observer has an internal representation of univariate parameters such as spatial frequency or orientation while stimuli are high-dimensional does not lead to contradictory predictions when following the theoretical framework. Second, we show that the measured *perceptual scale* corresponds to the transduction function hypothesized in this framework. In particular, we demonstrate that it is related to the Fisher information of the generative model that underlies perception and we test the predictions given by the generative model of different stimuli in a set a of difference scaling experiments. Our main conclusion is that the *perceptual scale* is mostly driven by the stimulus power spectrum. Finally, we propose that this measure of *perceptual scale* is a way to push further the notion of perceptual distances by estimating the perceptual geometry of images i.e. the path between images instead of simply the distance between those.

Published as a conference paper at ICLR 2024
PERCEPTUAL SCALES PREDICTED BY FISHER INFOR- MATION METRICS
Jonathan Vacher* MAP5, Université Paris Cité, CNRS, F-75006, Paris, France Jonathan. wacher@u-paris.
Pascal Mamassian
LSP. Département d' d'études cognitives, École normale supérieure, PSL University, CNRS, 75005 Paris, France pascal www.mmmaassss..m..mmmmmmm.. aaiamassiin@esss fr
ABSTRACT
Perception is often viewed as process that transforms physical variables, ex- ternal to an observer, into internal psychological variables. Such a process can be modeled by function coined perceptual scale. The perceptual scale can be deduced from psychophysical measurements that consist in comparing the relative differences between stimuli (i.e. difference scaling experiments). However, this approach is often overlooked by the modeling and experimentation communities. Here, we demonstrate the value of measuring the perceptual scale of classical (spatial frequency, orientation) and less classical physical variables (interpolation between textures) by embedding it in recent probabilistic modeling of perception. First, we show that the assumption that an observer has an internal representation of univariate parameters such as spatial frequency or orientation while stimuli are high-dimensional does not lead to contradictory predictions when following the theoretical framework. Second, we show that the measured perceptual scale corre- sponds to the transduction function hypothesized in this framework. In particular, we demonstrate that is related to the Fisher information of the generative model that underlies perception and test the predictions given by the generative model of different stimuli a set a of difference scaling experiments. Our main conclu- sion is that the perceptual scale is mostly driven by the stimulus power spectrum. Finally, we propose that this measure of perceptual scale is a way to push further the notion of perceptual distances by estimating the perceptual geometry of images i.e. the path between images instead of simply the distance between those.
INTRODUCTION
Difference Scaling Difference scaling methods allow us to measure the relative perceptual differences of multiple stimuli in human observers. Such methods have been used early in the 1960s to measure the relative differences of perceived color, contrast or loudness (see Maloney & Yang (2003) and references therein). This is only the beginning of century that fitting method, called Maximum Likelihood Difference Scaling (MLDS), was developed (Maloney & Yang 2003] Knoblauch & Maloney 2008) to infer the function that maps the physical to the perceptual space. This function is called perceptual scale. The critical assumption behind the fitting methods dates back to Thurstone's 's law of comparative judgment (see case V TThurstone (9277>) ): the difference between two values along psychological dimension is corrupted by noise that has constant variance. The perceptual scale informs about how a stimulus perceived when modified along a continuous physical scale (e.g. color, contrast, ..... When the slope of the perceptual scale is steep, perception changes rapidly with small physical changes i.e. the observer is highly sensitive to physical variations. When the slope is shallow, perception is stable even for large physical variations i.c. the observer is weakly sensitive to physical variations. Recently, the MLDS method has been used to measure the perceptual scales of surface texture (Emrith et al. 2010), watercolor effect (Devinck & (Aguilar et al., 2017), lightness (Aguilar & Maertens (2020) or
Knoblauch 2012), probabilities (Zhang et al. 2020). However, perceptual scales of more fundamental physical variables such as orientation and spatial frequency have not been measured nor related to existing probabilistic
hhttpss//oontthaavsceeerr.

---

Published as a conference paper at ICLR 2024
theory of perception. Additionally, while relations between standard Two-atternative Forced Choice (2AFC) measurements and perceptual scales have been studied (Aguilar et al. 2017). it was a theory of perception that was previously described (Wei & Stocker 2017) that has been used to derive predictions.
Probabilistic modeling of perception Thurstone's 's law of comparative judgment is a first brick in the history of probabilistic modeling of perception ( [hhrstoone] 1927). Indeed, it introduced the incipient concept of random variable (Chebyshev) [867] Kolmogorofi 1933) in psychophysies. Then, the development of computer science and information theory had major impact in perception studies, bringing concepts such as redundancy reduction and information maximization (Attneave) 1954] Barlow et al.] (96I). More specifically, when applied to texture perception, these concepts led to Julesz' hypothesis that perception of textures statistical (Victor et al.) 2017) i.e. textures with similar statistics are indistinguishable. Later on, together with advances in image processing, Julesz' hypothesis led to modern texture synthesis algorithms (Portilla & Simoncelli, 2000 Gaays al., 2015). In parallel, core theorem of probabilities, namely Bayes rule, was found to efficiently predicts human perceptual behaviors (Knill & Richards 1996). Further works have been dedicated to solve the inverse problem of identifying observers' prior that best explain their perception (Stocker & Simoncelli] (2066) Girshick et al.) [011] Vacher et al., 2018) Manning et al. (023). Inspired by neural population coding models, optimal observer theory is now described in details by Wei & Stocker (2017). The main consequence of this theory is the existence of a simple relation between perceptual bias and sensitivity. Yet, the theory is limited to a hypothesized scalar perceptual variable while it is established that only part of the neurons of the primary visual cortex is tuned to scalar variables such as spatial and temporal frequencies or orientation (Olshausen & Field 2005). In higher visual areas, it is more and more difficult to identify scalar variables that uniquely drive single neurons as they respond to increasingly complex patterns (Bashivan et al., 2019). In some previous work, Wainwright (Wainwright) [199) have used higher dimensional natural image statistics (auto-correlation and power spectrum) to explain various psychophysical observations. Though, is unclear how this approach relates the univariate Bayesian framework.
Fisher information in neural populations The theory behind the work of Wei & Stocker (2017) is largely inspired by previous work on neural population coding (Brunel & Nadal 1998). In this and subsequent works (Yarrow et al.) 2012; Kanitscheider et al., [015:/Bethge et al., 22002] Wei & Stocker 2016), it is often ultimately assumed that neurons are Poisson firing neurons parameterized by the tuning curve of a scalar variable. Variants of this optimal coding framework have been explored to explain the response of single neurons [Lughhin (1983); yon der Twer & MacLeod (200T). As stated previously, it is a quite restrictive framework as all neurons are not tuned to a scalar variable (Olshausen & Field 2005). By applying this framework to perception, Wei and Stocker remove these unnecessary assumptions. They derive the relation between perceptual bias and sensitivity which at its core comes from the relation between Fisher information and prior under the optimal coding assumption (Brunel & Nadal 1998)
(s(s) ((()
(1)
where S' is stimulus variable. Fisher information is used to quantify the variance of stimulus estimator from a neural population encoding (Cramer-Rao lower bound). In contrast, priors are introduced in observer models to explain perceptual biases. Therefore, Equation (1) links neural population models and perceptual models. However, less attention dedicated to the underlying encoding model, where stimulus variable S is non-linearly related to an internal measurement M through a function yp plus an additive Gaussian noise N with constant variance,
M ((S) + N.
Interestingly, such an encoding model is very similar to the assumptions behind the observer model underlying the MLDS method. However, the precise nature of these internal measurements has so far remained abstract.
Perceptual distance A perceptual distance is a score of image quality used to quantify and to compare the performances of image restoration or generation methods. Perceptual distances have been introduced to overcome the limitation of the Signal-to-Noise Ratio (also known as SNR). Indeed, images with similar SNR could vary subjectively in quality when presented to human observers (Wang et al, 2004). The Structural SIMilarity index (SSIM) is a score that is popular to provide better account of perceptual similarity as compared to SNR. Since then, variations of SSIM
2

---

Published as a conference paper at ICLR 2024
have been proposed to more specific purposes such as estimating photo retouching (Kee & Farid 2011). However, these scores require to compare the image to be rated to a reference image. In recent years, the success of deep generative modeling have led to the emergence of new scores such as the Inception score (Salimans et al. 2016) or the Fréchet Inception Distance (FID) (Heusel et al., (017). These scores have in common that they compare the generated image distribution to the true empirical image distribution instead of a generated image to reference image. In addition, they are based on Deep Neural Network (DNN) features. Overall, has been shown that such DNN features-based scores are better aligned with human perception than SSIM or SNR for example (Zhang et al. 2018). One possible explanation is that DNN are able to capture high-order image statistics and that as hypothesized in vision, our perception is deeply related to image statistics (see Hepburn et al. (2022) and references in previous paragraphs). Yet, these coined perceptual distances are not exempt of limitations as they could be subject to bias when classes specific features are present or not (Kynkäännicmi et al. 2023). Overcoming these biases will likely require to move away from training by measuring higher-order statistics on the image directly without relying on some learned or random filters (Amir & Weiss 2021). One other limitation of perceptual distances is that they do not provide any information about how well model has captured the perceptual geometry of images. Providing a full account of perceptual geometry is more demanding, it requires to compare the path when moving from one image to another and not only their distance along this path.
Contributions Our work brings several contributions to overcome the limitations introduced above. First, we explain that convergence theorem of discrete spot noises is a way to resolve the tension between univariate Bayesian theories of perception (Wei & Stocker| (017)) and the high dimensionality of images (Wainwright 1999). More precisely, the hypothesis that an observer has a univariate representation of the distribution of the parameter of interest (e.g. spatial frequency), is compatible with the assumption that an observer is measuring the spectral energy distribution of the image in that both assumptions leads to similar predictions. Second, we show that the function v, introduced in Equation 2
can be interpreted as the perceptual scale as measured by a difference scaling experiment. Then, demonstrate again that this function & can be predicted from the Fisher information of the stimulus when using the true distribution of the noisy internal stimulus representation knowing the presented stimulus i.e. the distribution of the measurements M that give explicitly. Therefore, provide a clear link between theory and experiment. Third, we propose to go further in exploring perceptual distances by estimating how well the geometry of natural images captured by models matches the perceptual geometry. For this purpose, we empirically test the prediction given by the Fisher information of Gaussian vectors and processes in a series of experiments (code and data]] texture interpolation codef) involving stochastic stimuli characterized by their power spectrum or their higher-orde statistics captured by VGG-19 (Gatys et al.] 2013). In particular, we collapse the high dimensionality of these statistics by interpolating between single textures (Vacher et al. 2020) and we measure the corresponding perceptual scale when going from one texture to another. Finally, we propose the Area Matching Score (AMS) score to quantify the mismatch between the predicted and the measured perceptual scales providing a clear method to evaluate the perceptual alignment between generative image models and human vision.
Notations Unless stated differently, upper case letters (e.g. X) are random variables and lower case letters (e.g. 2² are samples realizations of those random variables. The probability density at X = æ is denoted Xx (x). Similarly, the conditional probability density at X = x knowing Y = is denoted Pxix (a, y). The set $ (Sints Stial] is the stimulus segment.
METHODS
2.1 STOCHASTIC VISUAL STIMULATION
We recall some theoretical results about the artificial textures we use in the current work. Firstly, we use textures that are stationary Gaussian Random Fields (GRFs) fully characterized by their scalar mean and their auto-correlation function (or equivalently their power spectrum i.e. the auto- correlation Fourier transform). Interestingly, such GRFs can be seen as the limit of high intensity discrete spot noises. This result allows one to relate the densities of local image features such as orientation and scales to the power spectrum of the image (seen as a GRF). In summary, it provides a link between scalar densities and the nigh-dimensiona. Gaussian distribution of the image. We will
3

---

Published as a conference paper at ICLR 2024
see in later sections that this result leads both approach to similar predictions for perceptual scales. Secondly, we use naturalistic textures that are obtained by imposing high-order and high- high-dimensional statisties obtained using VGG-19, 19. However, the result mentioned above and detailed below does not hold for naturalistic textures. It is unknown at this stage if similar results could be obtained with some feature under some (non-linear) transformation.
Asymptotic Discrete Spot Noise Let Eo = (1,0). Let 9o be a Gabor function defined for all a > 0
and for all x E _2 by go (a:) = 1/2n cos(æ to)e of |=1", In addition, let Ps.. be scaled rotation
defined for all (z, 0) E IR+ x [0, n by (2, 2 =x))(x) where Ro is the rotation of angle 0. Now, let Fx.o be a discrete spot noise of intensity À > 0 defined as the following random field for all x E IR2 (x) variables. Specifically (XE) 2-D Poisson process of intensity X and (Zk, ØR)ken have densities (F22, Pe).
1/V5> EKEN De (p O. (x - Xx)) where (Xx, Zk. ①x) are iid random
Proposition (Convergence and Power Spectrum), In the limit of infinite intensity () - +o0) and pure wave (a -> 0), Fx.o converges towards Gaussian field F with the following power spectrum for all & E R²2
1 7(5) = 1&|2 P2(1)))44))
where& & (1gl cos (<E), |El sin (<E))-
Proof. This a special case of Proposition 2 in Vacher et al. (018). The general result is Theorem 3.1 in Galerne (2010).
In practice, the distribution Pz and Fee are parametrized by (Zo, Σ2) and (Qu, Σe) respectively,
By providing relation between local feature statistics (orientation and scale) and the image power spectrum, Proposition will allow us to justify the common assumption made when modeling psychophysical data that is, the feature of interest is directly used by the observer instead of the image (Knill & Richards 1996, Stocker & Simoncelli 2006 Girshick et al. (011). See Section 2.3
Interpolation of Naturalistic Textures Even though for experimental purposes the GRFs described above can be parameterized by just a few scalar variables (Vacher et al.) (018)) naturalistic textures depend on the statistics of high-dimensional features extracted at different layers of VGG-19 (Gatys) et al., 2015: Vacher et al.] 2020). Previous algorithms widely used in vision studies (Portilla & Simoncelli 2000; Vacher & Briand 202T) were using fewer parameters, but the number was still too large to derive clear and interpretable results (Okazawa et al., 2015). One way to efficiently collapse the dimension parameterizing those textures is to use interpolation (Vacher et al., 2020). As
consequence the texture features of an interpolation of textures extracted at layers lt are interpreted as realizations of a random variable A& (s) with mean jw and covariance Ew (s) (see Appendix C). Assuming Gaussiannity, it will become possible to derive predictions for the perceptual scale measured along the interpolation path.
2.2
THURSTONE SCALE, FISHER INFORMATION AND MLDS
First, we define more precisely the encoding model given in Equation
2 ass follows
R + N where (S) with
x/> S S.
We use the above description to highlight the fact that M, R and N are random variables that are internal to the observer while S is external to them, it is environment variable, external stimulus. In practice, the noise N often assumed to be Gaussian with variance o². It corrupts the internal representation of the stimulus R to give what we call the internal measurement M. Then, we define Fisher information for two abstracts unidimensional random variables.
Definition (One-dimension) dimensional Fisher information). Let X and Y be two random variables defined respectively on two abstract spaces X and Y and let IP X be the conditional density of X knowing Y. The Fisher information carried by X about Y is a function I. W. - R. defined for all E V by
Ir(y) Exix ((
(Olog((xxx) (PxI))
(X,8)
(4)
Oy
4

---

Published as a conference paper at ICLR 2024
perceptual scale
(2x) bdw (5.)
oci. bdw (de)
physical scale
Figure 1: Texture samples and predicted perceptual scales for the spatial frequency mode (zo), the spatial frequency bandwidth (b,) and the orientation bandwidth (arg). Bottom-right prediction obtained by combining Appendix And Equation (5)
In statistics, Fisher information used as upper bound of the precision of an estimation (see Cramér-Rao bound). This is also how we interpret it for an observer, namely the maximal precision of their estimation of stimulus S.
The precise definition given above is helpful to realize that the Fisher information carried by M about (Ig) is different from the one carried by . about R (In). We can go one step further though, and establish the following relation between those two
Is(s) = 3' (3)
(5)
A reformulation of Thurstone law of comparative judgment (Thustonnc 1927) is to assume that the Fisher information of an observer's internal representation ZR constant. is so obvious to understand why this assumption is relevant. The idea is that an observer has only access to her internal states, she never observes any realization of an external stimulus S. Every external variable is transformed to an internal one through the psychological function tb. Therefore, without any knowledge about the external world, a fair assumption is to allocate equal resources to every possible internal state in order to be equally precise in our estimates of different states (without knowing to what they correspond to the external world). This assumption is also equivalent to assuming that internal observer's noise (a common notion used in psychophysics) is constant. If the internal Fisher information is constant we can now express the psychological function simply in terms of external Fisher information. This is summarized in the following proposition.
Proposition 2. Assume Equation 3 the internal Fisher information In constant if and only ifr all E S the psychological functioo v verifies
(s) [ VIs(I)d.
(6)
Proof. See Appendix D
Relation to the MLDS observer model. In the MLDS framework, observer has judge which pair of stimuli more similar to another. Assuming three stimuli (s,, 8j, 8k), those are transformed through the psychological scale vp and the observer responds by comparing the difference of differ- ences between the pairs di,j. = [b,(s,) - dbss) ()(ss) - (sx)).. This difference is assumed to be corrupted by an internal noise Nmids of constant variance oi i.e. A. = d + -Nmlas- Those assumptions sufficient to recover estimate of (Knoblauch & Maloney, 2008). In addition,
is often assumed that there is no specific internal ordering of the variables so that the difference d..j.k be written without absolute value. In that case, assuming the encoding model (3) is enough to recover the MLDS observer model, we have the following relation between the noise variances mlds 2
5

---

Published as a conference paper at ICLR 2024
(hielogram)
spec.)
perceptual scale
(come 3)
VGG18 (conv
VGGI8 (eoe
(weengrol)
interpolation weight
Figure 2: Texture samples and predicted perceptual scales for various interpolation between arbitrary textures. Red corresponds to the early sensitivity group (i.e. shallow-to-steep slope). Blue corre- sponds to the late sensitivity group (i.e. steep-to-shallow slope). Yellow corresponds to conflicting prediction across VGG-19 layers. Bottom-right; prediction obtained by combining Proposition and Equation (5). For pixels, images and wavelet, we also assume Gaussiannity as in Equation (pixel and wavelet) and in Equation (images). See details in Appendix
2.3 FISHER INFORMATION CARRIED BY THE IMAGE vs BY THE LOCAL FEATURES
abstract random internal In the previous section, have introduce observer model based measurement M. often unclear what those measurements are. Ideally inspired by neurophysiol- ogy, the measurements operations, even though those modeling stages are often dropped in perceptual studies. In the case of GRFs parameterized by spatial frequency (or scale) and orientation distributions, it is often accepted to assume that the measurements are samples of an appropriate distribution e.g. a Log-Normal distribution for the spatial frequency or Von-Mises distribution for the orientation. Using the notation of the previous section, these cases correspond to measurement . Z with stimulus = 'o and measurement M with stimuli S Og. We will see that in both cases this is equivalent to consider that measurements are the image itself M = F with S Zo S Og. This because Fisher information given in closed-form and that perceptual scale can be predicted using Proposition 2 Similar results hold for S Bz and S = Eg (note that (Zo. Bz) and (o,, Ee) are parameters Pz and Fe introduced in Section 2.T). The predictions are given in Figure
responses of neurons the image often modeled by Linear/Non linear
Fisher Information of Log-Normal and Von-Mises Distributions We give the precise parametriza- tion and the corresponding Fisher information the Log-Normal and Von-Mises distributions in Appendix
Fisher Information of Parametric GRFs Now, we consider GRF texture with mean μ R and autocorrelation function 7 (or equivalently power spectrum f) parameterized by S. Mathematically, the texture can be expressed for all R² and s E S, as
/
F(x, 5) = + f k( - s)d
(7)
where k(:,s) F- FFA( s)) and W is classical Wiener process. Proposition 3. The Fisher Information carried by F about S is
09 8)
Olog(f(E, s))
I(s) Ra s)
dg
dg.
as
Øs
Ra
Proof. This specific case of Whittle formula (Whittle 1953
Theorem 9).

---

Published as a conference paper at ICLR 2024
We combine Proposition with Proposition Jusing the Log-normal and the Von-Mises distributions to express the power spectrum & parameterized by S = Zu. = Bz, S Ou or S = Eg. Therefore, the Fisher information carried by mcasurements M F comes down to the Fisher information carried by measurements M = % (spatial frequency) or M = @ (orientation) as described above up multiplicative constant of 1/2. As consequence both approaches lead to similar predictions about the perceptual scales measured for these parameters.
Fisher Information of Parametric Gaussian Vectors In the case of interpolation between naturalis- tic textures, we do not have a direct generative model of the texture conditionally on the interpolation parameter s. Instead, the texture generated using gradient descent to impose the statistics of VGG-19 features at multiples layers for which we have a generative model. Therefore, at layer k and for 8 E the activation A (s) of texture Fs(()
(s)s) = (s((s) + (s) with Mi E C³ (S, and Mi E (s,R"*x*)
(8)
where N ~ NO, la.) Proposition 4. The Fisher Information carried by A about S is
) is standard normal random vector and d2 is the feature dimension of layer k.
1 [(s) [4(()) + Tr
Proof. See Appendix B
spatial freq.
spatial freq. bandwidth
ori bandwidth
As stated earlier in the manuscript, no link can be made with inter- pretable feature distributions as is the case for GRFs. Though, precise expressions for μ& and Σg are available in close forms in the Gaussian case as assumed here (see Appendix C In practice, the feature activations are not Gaus- sian /acchr et al.) 2020).
1.0
0.5
theury
0.0
0.0
0.5
1.0
0.0
0.3
1.0
0.0
0.5
1.0
Figure 3: Measured and predicted perceptual scales for the spatial frequency mode (left), the spatial frequency bandwidth (middle) and the orientation bandwidth (right).
2.4 PREDICTIONS AND EXPERIMENTAL METHODS
The calculated Fisher informations together with Proposition 2 alloow us to predict the perceptual scales corresponding to the parameters described the previous section. These predictions hold under the assumed generative models for measurements.
Predictions In the case of GRFs textures, we recall that assuming that spatial frequency or orientation are directly measured or that the image whole is measured makes no differences in the prediction (see bottom-right of Figure Tp. In the case of naturalistic textures, the measurements are assumed to be the feature activations of the texture in VGG-19 at layer 2 to 5 (see bottom- rightt of Figure 2) For the naturalistic, we alternatively propose that measurements are the single pixel gray levels, the image itself (i.e. the power spectrum as for GRFs) the wavelet activations.
Experimental Methods The experiment consists of trials where participants have to make a similarity judgment. Participants presented with stimuli with parameters 81 82 < 83 and are required to choose which of the two pairs with parameters (#1, 82) and (s2, 83) is the most similar. We used four sets of textures (see Figure and 2): (i) the first consists of parameterized artificial textures where we measured the perceptual scales of spatial frequency, spatial frequency bandwidth and orientation bandwidth (see Appendix [E for details); (ii) the other three sets consist of interpolations between arbitrary textures. A set of textures for which the perceptual scale corresponds to an early sensitivity (i.e. steep-to-shallow slope, see top-left of Figure 2 Another one for which the perceptual scale corresponds to late sensitivity (i.e. shallow-to-ee tt-tot-ttt-eepppp slope, see top-right of Figure 2) And a last set for which the predictions are inconsistent from one layer of VGG-19 to another (bottom-left of Figure 2 All stimuli had an average luminance of 128 (range [0, 255)) and an RMS contrast of 39.7. For each texture pair, we use 13 equally spaced (8, = 0.083) interpolation weights. To ensure that stimulus comparisons are around the discrimination threshold we only use triplets such that [$1,3 S2 I 35.. For each texture pair, a group of 5 naive participants performed the experiment. Participants were recruited through the platform prolific (https: htwp:://wwww... prols fic. com), performed the
7

---

Published as a conference paper at ICLR 2024
pair01
poir12
peir03
pair04
paird5
1.0
0.5
buznan sheory
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
pair06
poir07
Poir08
pair09
pair10
1.0
0.5
Intzun theory
0.0
0.0
0.5
1.0 0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Figure 4: Measured and predicted (power spectrum) perceptual scales for the early (top row) and late (bottom row) sensitivity pairs. Error bars represent 99.5% bootstrapped confidence intervals.
experiments online, and were paid 9C/hr. Monitor gamma was measured using a psychometric estimation and corrected 1. The MLDS model described the end of Section 2.2 The protocol was approved by the CER UaPaiis (IRB 00012020- 54).
3 RESULTS
3.1 ORIENTATION AND SPATIAL FREQUENCY
pairl1
peir12
1.0
The perception of spatial frequencies is well-known vision studies, its perceptual scale is expected be logarithmic. Such a scale is also predicted by Fisher information as integrating the square-root (Proposi- 2ion of squared inverse (Proposition 5) leads to logarithm. The measured perceptual scale of the spa- tial frequency mode matches correctly this prediction (left of Figure 3). The spatial frequency and orien- tation bandwidths are less studied, the predictions are qualitatively the same as for the spatial frequency mode (Proposition 2 and Appendix Proposition 6 The Fisher information of the orientation bandwidth more complex but leads to similar curve. The measured perceptual scale is more variable for the orientation bandwidth (larger eттоr bars) but is still in line with the prediction (predicted offset from linear behavior is exaggerated, see right of Figure 3) In contrast, the measured perceptual scales of spatial frequency bandwidth is approximately linear for low values while its gets supra-linear at intermediate values and even saturate for the highest values (center of Figure
Iuaman theory
0.0
0.0
0.5
1.0
0.0
0.5
1.0
Figure 5: Mcasured and predicted (auto-cor) perceptual scales for the conflicting predic- tion pairs. Error bars represent 99.5% boot- strapped confidence intervals.
3.2 INTERPOLATION BETWEEN NATURALISTIC TEXTURES
present the perceptual scales measured for the different groups natural textures in Figure and 5 On these figures, the prediction given by the auto-correlation (i.e. when considering the textures as GRFs) is shown. Predictions assuming alternative measurements (pixel, wavelet, and VGG-19) are quantitatively compared in Figure f the following Area Matching Score AMS fo ==))(((( (/tdf((()))/////()==<()a<) where fm and fu, are respectively the measured and predicted scales. Intuition about score values is given in Figure
Early and Late Sensitivity For the set of early sensitivity texture pairs, measured perceptual scales are inline with the predictions (Figure 4 in the sense that a linear (pai rr1, pa 044-05) supra-line (pair02-
perceptual scale is measured in all texture pairs. The same hold for the set of late sensitivity texture pairs (Figure 4. but this time in the sense that linear (pairO prarrs 6 and paire (8) or sub-linear (pair07 and pairr9-11 100 perceptual scale is measured in all texture
8

---

Published as a conference paper at ICLR 2024
Jasee 1 (Trus) sccee 25
pood
Image (otto oct
wasdee vigztl (Dggns 2
AMS scores
3-
2
0.5
0.0
peirtl pairt2
pairtz
pairty
pairts
pairts
pairt7
pairte
morrio pairll
pair
all pains
(a)
Figure 6: Prediction scores. (a) Intuition about score values when comparing two perceptual functions.
(b) Left scores for each pair and for different predictions. Right averaged scores for different predictions with 99.5% bootstrapped confidence intervals.
pairs. Such result is also valid for the predictions based on alternative measurement assumptions as shown in Figure 6 by the fact that all scores are positive for pai r01-1.
Confficting predictions For the set of textures with conflicting predictions (Figure 5). for both texture pairs, we observe that the GRF measurement assumption predicts a late sensitivity while the measured perceptual scale corresponds to an early sensitivity with a late saturation. Other measurement assumptions are not providing better prediction as their score is either close to 0 or negative. However, there one exception for pairll under the wavelet measurement assumption which has a ideal score, close to 1 (up to score limitation, see Section 4
Measurement Assumption Scores As previously stated, all assumptions predict correctly whether the scale corresponds to late early sensitivity (positive scores) except for the conflicting prediction texture pairs. Note that pairr. r05 has also score close to 0 under all assumptions (though this might be due to score limitation, see Section 4). On average, the GRF assumption is the best with average score (1.99.5% CI) close to 1 (0.92 0.69). The single pixel distribution assumption only predicts a linear behavior and has therefore score close to O. In contrast, the wavelet and VGG-19 assumptions often overestimate the early or late sensitivity (average scores above 1). We conducted additional experiments in which we fixed the power spectrum of all textures along path between pair to be the average of the pair's. s. In this case the power spectrum cannot explain the measured perceptual scale the operation is indeed deteriorating discriminability (see Appendix H
DISCUSSION AND CONCLUSION
In the case of GRFs, we have shown that the univariate assumption behind the Bayesian theories of perception and the absence of this assumption (i.e. the observer is using all the information in the image) lead to the same prediction for the perceptual scales of spatial frequencies, orientations and their bandwidths. Such a result is due to the fact that these local feature distributions directly appear in the power spectrum (the Fourier transform of the auto-correlation) of GRFs (Proposition [] and to our second result that is the perceptual scale is related to the Fisher information of the feature distribution (Proposition[2 2). In the case of naturalistic textures, is unknown if such a result relating a (non-linear) transform and some feature distribution holds. Therefore, it is necessary to make new hypotheses about the measurements in order to predict the perceptual scale of an observer. We tested this issue in series of difference scaling experiments involving GRF and naturalistic textures. Our main result is that the perceptual scale is mainly driven by the auto-correlation (or the power spectrum). However, it does not perfectly explain the measured perceptual scales, and in particular, the perceptual scale of bairllle appears to be driven by the wavelet representation. A highly interesting future directions is to compare the perceptual scale to neurometric scale, an equivalent scale but deduced from neurophysiological recordings as the equivalent exists for the psychometric function (Newsome et al., 1989 Berens et al.) 201T). Other limitations lie in the MLDS method. Usually, running a difference scaling experiment requires to know ahead of time an approximation of the observer's sensitivity to the parameter that one would like to test. Here, we have not estimated the sensitivity of each participant and, therefore, have not adapted the stimuli accordingly. Yet, it seems that we were near the participants sensitivity (see Appendix G)). In addition, the MLDS method limited the study of single stimulus dimension while extensions can still be developed Knoblauch et al. (2012) and compared to higher-dimensional theories Malo & Gutiérrez) (2006): Laparra & Malo (005)) All these questions demonstrate the ambition of our approach and the work that remains to be done to understand, beyond perceptual distance, perceptual metrics.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The reproducibility of our work will be ensured by the links provided to the data and code. Theoretical results supported by proofs references to proof.
REFERENCES
Guillermo Aguilar and Marianne Maertens. Toward reliable measurements of perceptual scales in multiple contexts. Journal of Vision, 20(4):19-19, 2020.
Guillermo Aguilar, Felix A Wichmann, and Marianne Maertens. Comparing sensitivity estimates from mlds and forced-choice methods in fext-fre experiment. Journal of Vision, 17(1): 37-37, 2017.
Dan Amir and Yair Weiss. Understanding and simplifying perceptual distances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12226-12235, 2021.
Fred Attneave. Some informational aspects visual perception. Psychological review, 61(3):183, 1954.
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233, 1961.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364164499):cav94336, 2019.
Philipp Berens, Alexander Ecker. Sebastian Gerwinn. Andreas S Tolias, and Matthias Bethge. Reassessing optimal neural population codes with neurometric functions. Proceedings of the National Academy of Sciences, 108(1 (1::)4444442222 2011.
Matthias Bethge, David Rotermund, and Klaus Pawelzik. Optimal short-term population coding:
When fisher information fails. Neural computation, 14(10):2317-233 2002.
Nicolas Brunel and Jean-Pierre Nadal. Mutual information, fisher information, and population coding. Neural computation, 10(7): 1731-1777 1998.
PL Chebyshev. On mean values.[o srednikh velichinakh). Matem. Shornik, pp. 1-9, 1867.
Yongxin Chen, Tryphon T Georgiou, and Allen Tannenbaum. Optimal transport for gaussian mixture models. IEEE Access, (626996278, 2018.
Frédéric Devinck and Kenneth Knoblauch. A common signal detection model accounts for both perception and discrimination of the watercolor effect. Journal of Vision, 12(3):19-19, 2012.
Khemraj Emrith, MJ Chantler, PR Green, LT Maloney, and ADF Clarke. Measuring perceived differences in surface texture due to changes in higher order statistics. JOSA A, 132-)122, 2010.
Bruno Galerne. Stochastic image models and texture synthesis. PhD thesis, École normale supérieure de Cachan-ENS Cachan, 2010.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. Advances in neural information processing systems, 28, 2015.
Ahna R Girshick, Michael S Landy, and Fero Simoncelli. Cardinal rules: visual orientation perception reflects knowledge of environmental statistics. Nature neuroscience, 14(7):926-932. 2011.
Alexander Hepburn, Valero Laparra, Raul Santos- Santos-Rodrigues Rodriguez, Johannes Ballé, and Jesus Malo. On the relation between statistical learning and perceptual distances. In 10th International Conference on Learning Representations, ICLR 2022, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Ingmar Kanitscheider, Ruben Coen-Cagli and Alexandre Pouget. Origin of information-limitine noise correlations. Proceedings of the National Academy of Sciences, 112(50):E6933- E6982, 2015.
Eric Kee and Hany Farid. perceptual metric for photo retouching. Proceedings of the National Academy of Sciences, (08(50):19907-199)) 2011.
David Knill and Whitman Richards. Perception as Bayesian inference. Cambridge University Press, 1996.
Kenneth Knoblauch and Laurence T Maloney. Mlds: Maximum likelihood difference scaling in r. Journal of Statistical Software, 25:1-26, 2008.
Kenneth Knoblauch, Laurence T Maloney, Kenneth Knoblauch, and Laurence T Maloney. Maximum likelihood conjoint measurement. Modeling psychophysical data in R, pp. 229-256, 2012.
Andrey Kolmogoroff. Grundbegriffe der wahhscheinniccheiisscchnngg Springer-Vertee Berlin Heidelberg, 1933.
Tuomas Kynkäänniemi, Tero Karras, Miia Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In The Eleventh International Conference on Learning Representations, 2023.
Valero Laparra and Jesús Malo. Visual aftereffects and sensory nonlinearities from single statistical framework. Frontiers in human neuroscience, 9:557, 2015.
S Laughlin. Matching coding to scenes enhance efficiency. In Physical and Biological Processing of Images: Proceedings of an International Symposium Organised by the Rank Prize Funds, London, England, 27-29 September, 1982, pp. 42-52. Springer, 1983.
Jan R Magnus. The moments of products of quadratic forms in normal variables. Statistica Neer- landica, 32(4))201-210, 1978. URL https: wwwp://wwww/.../m janmagnus n1 nt/papers/JRMO0
pdf
Jesús Malo and Juan Gutiérrez. VI non-linear properties emerge from local-to-global non-linear ica. Network: Computation in Neural Systems, 17(1):85-102, 2006.
Laurence T Maloney and Joong Nam Yang. Maximum likelihood difference scaling. Journal of Vision, 38))5-5,, 2003.
Tyler S Manning. Benjamin N Naecker, Iona R McLean, Bas Rokers, Jonathan W Pillow, and Emily A Cooper. A general framework for inferring bayesian ideal observer models from psychophysical data. eneuro, 10(1), 2023.
William T Newsome, Kenneth H Britten, and Anthony Movshon. Neuronal correlates of perceptual decision. Nature, (42(772)) 1989.
Gouki Okazawa, Satohiro Tajima, and Hidchiko Komatsu. Image statistics underlying natural texture selectivity of neurons macaque v4. Proceedings of the National Academy of Sciences, 112(4): E351-E360, 2015.
Bruno Olshausen and David J Field. How close are we to understanding v1? Neural computation, 1768):165511699, 2005.
Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40:49-70, 2000.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
11

---

Published as a conference paper at ICLR 2024
Alan A Stocker and Eero P Simoncelli. Noise characteristics and prior expectations in human visual speed perception. Nature neuroscience, 9(4):788 78788-585222 2006.
Louis Thurstone. law of comparative judgment. Psychological review, 34(4):273, 1927.
Jonathan Vacher and Thiband Briand. The Portilla-Simoncelii Texture Model: towards Understanding the Early Visual Cortex. Image Processing On Line, 11:170-211, 2021. https://doii org/
10. 20.1522000...222..333.
Jonathan Vacher, Andrew Isaac Meso, Laurent U Perrinet, and Gabriel Peyré. Bayesian modeling of motion perception using dynamical stochastic textures. Neural computation, 30(12):3355-339) 2018.
Jonathan Vacher, Aida Davila, Adam Kohn, and Ruben Coen-Cagli. Texture interpolation for probing visual perception. Advances in neural information processing systems, 13:22146-22153 2020.
Jonathan D Victor, Mary M Conte, and Charles Chubb. Textures as probes of visual processing.
Annual review of vision science, 3:275-296, 2017.
Tassilo der Twer and Donald IA MacLeod. Optimal nonlinear codes for the perception of natural colours. Network: Computation in Neural Systems, 12(3):395, 2001.
Martin Wainwright. Visual adaptation as optimal information transmission. Vision research, 39(23): 3960-3974, 1999.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from eτror visibility to structural similarity. IEEE transactions on image processing. 13(4):600-612, 2004.
Xue-Xin Wei and Alan Stocker. Mutual information, fisher information, and efficient coding. Neural computation, 28(2):305-326, 2016.
Xin Wei and Alan A Stocker. Lawful relation between perceptual bias and discriminability. Proceedings of the National Academy of Sciences, 1024--21024441024,, 2017.
Peter Whittle. The analysis of multiple stationary time series. Journal of the Royal Statistical Society: Series B (Methodological), 15(1):225-13, 1953.
Stuart Yarrow, Edward Challis, and Peggy Seriès. Fisher and shannon information in finite neural populations. Neural computation, 2447):1740-1780, 2012.
Hang Zhang, Xiangjuan Ren, and Laurence T Maloney. The bounded rationality of probability distortion. Proceedings of the National Academy of Sciences, 117(36):22024 22034, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
A FISHER INFORMATION OF LOG-NORMAL AND VON-MISES RANDOM VARIABLES
A random variable Z follows log-normal distribution (Z ~ LN bz)) if it has the following density defined for all = > 0 by
In
In(2)
82
Pz 20, bz) zbz (2)
2
exp
20
ln(2)b2
where is the mode and bz is the octave bandwidth of the density. A random variable θ follows by distribution (0 (M σ6)) if has the following density defined for all 9 [0, æ]
1
cos (2(0 0o))
Pe (0)) = TBoo exp
40
12

---

Published as a conference paper at ICLR 2024
Proposition 5. Let Z ~ (2 b2). The Fisher Informations carried by Z about respectively Zo and Bg are
b9 ln( and I(bz)= = In(2)
1+ [nn2)
I(zo)
(2
Proposition 6. Let @ V.M σe). The Fisher Informations carried by O about respectively Oo and De are
(1/4gg I(0)) = og Bo(1/102)
1 (1-
Bi(1/a0))
Bi(1408)
and I(ge) = 40 Bo(+/20))
40 +
Bo(2/108)
B FISHER INFORMATION OF PARAMETRIC GAUSSIAN VECTOR
Proof of Proposition
We start by reminding the following calculus results Petersen al. 2008
(1 V log (lA)) = (A-
iii dA -
The log of Px is
log(Px(s(z(s))
- log(2r) log((>(s))) 5( m(s)) m(s)).
1
1
Then,
llo (1)s)
1 (x (E(s) - (x)((s)))
(x, 8) = TT(E(())
Os
2
(x m(s)) E(()))
Then,
Ø Olg((x)1s) (1,,8) = Ci +
Ös
where
1 Tr(E(s))
C 4
C2 4 - ((s)) ( (x6
1
C3 ((=-)))"
1
C4 Tr (X))))))))))
(r (E(s) (x m(s)),
2
C3 Tr (E(s) -()) -(s)
C. - - (1)) -()))))))))))/ (1)((=/)///)))))))//)))))))))))) -(8)) (I(s) (x m(s)).
First, we observe that B(Ci) = C₁ and that (C3) (Co) (odd central moments).
Then, following (ii), we have
Hence,
(a- (T))) (E(s) (x m()))) -E -(((1-1))))((()))((0000 ((-(())
-EE ((V (VE(s) (x (x-ss))
13

---

Published as a conference paper at ICLR 2024
[Note: think about the covariance of (E((s)[(ss) (x (u(s)) Therefore, the expectation of C₁ is
Magnus 1978
E(C)) - (rs((s))
The expectation of C2 is the second order moment quadratic form (see Magnus (1978). Thus,
E(C2) = Tr(((s) 2
We can now deal with the expectation of the last term C3.
B(Ca) E (1(<
-E ((x-~(s)) ((s= ('()) - 2(s))
E
= = ())) ((x ((se--(s)((-pppp) (000)((=((()))))) 2(s) ( 2 ~(=)
Summing all the expectations leads to the result.
C MEAN AND COVARIANCE OF A WASSERSTEIN BARYCENTER
Proposition 7 (Wasserstein Barycenter of Gaussian Distributions). Let Xo N En) and X, N(H,, Σn) be nwo Gaussian random variables. The Wasserstein interpolation between P x. and IP x, defined as
P. argmin
(1 (1=s)W2(Px,,P)) P)2 + sW2
PEP(2xo:Px.)
where x Px,) is the set of probability densities with marginals Px and Px, and where
Wx(Px.Py)2 X~Px.Y-Py
inf
(IXX-113)
is the Wasserstein distance between Fx and Py. The probability distribution P, is Gaussian with mean pw (s) (1 s)ux,, + spx, and covariance
(w(s)= = ExO 1/2 (a- + (1122 Ex, xxxxx) )
1/2) 2
1/2
Proof. See CChnn et al. 2018
Corollary 1. Let X, (Nw((s) Ew(s)) where ((uw()), Ew are defined in Proposition then the Fisher information is given by
I(s) (uxi mxx) -xxO)
TE ( (ux: Mxx)
2/ Ir 1 22 (s) - (s)s
(9)
2
where (8)) 2> + Ex, Qo - Q1) Q + Q2 - 22 with
Qo = 1/2 1-1/220 2 Dx,
23 2ix00 and Q = V (2 2xx 2:1/2
1/2
Proof. Combine Proposition 4 and Proposition
D PERCEPTUAL SCALE AND FISHER INFORMATION
Proof of Proposition 2 One has (m, s) = P P M (s)) which implies that
log MIS) (m, 8) = (s()) (P (m, (b()).
Øs
as
Therefore, by squaring this last equation and taking its expectation with respect to M we obtain Equation (5) Then, taking the square root and integrating lead to the result.
14

---

Published as a conference paper at ICLR 2024
E EXPERIMENTAL DETAILS
For the GRF textures we used the following parameters (see distributions Appendix A):
Spatial freq. mode (2o) : [0.5, 0.75, 1.0, 1.25, 1.5, 2.0, 2.25, 2.5, 2.75, 3.0, 3.25, 3.5], Spatial freq. bandwidth (b2) [0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.25], Orientation bandwidth (oe) [1.5, 2.5, 3.5, 4.5, 5.5, 6.3, 7.5, 9.5, 10.5, 11.5, 12.5, 13.5].
The code to generate these stimuli is available here?
For the naturalistic interpolation of textures we the natural textures available here The interpo- lation parameters used are given in the main text (see Vacher et al. (2020) or Proposition 7 for the interpolation weight definition).
F
FISHER INFORMATION CALCULATION
Pixel assumption We consider that pixel are sample from a univariate Gaussian distribution. To com- pute the Fisher information, we estimate the mean and standard deviation and then use Proposition 4
Images assumption We consider the image to stationary GRF. To compute the Fisher informa- tion, we estimate the power spectrum and then use Proposition
Wavelet assumption We are using the steerable pyramid Portilla & Simoncelli (2000). Then, as in the case of VGG19 features, consider that the vector of wavelet activations each pixel location a sample of multivariate Gaussian distribution. So we compute the empirical mean and covariance square root and use Proposition E
G INDIVIDUAL PERCEPTUAL SCALES
In the figures below, show the perceptual scale measured each individual participant. Beware that the participants are not the same from one pair textures to another despite being label as "Participant oX".
Partieipant 01
Purticipart
Purticipant
Participant
Particionnt 06
1.0
1.C
1.0
1.0
0.5
0.5+
0.5
0.5
Trmman theoey
bumman theoey
Inmmo thoory
Inrmnn thoory
Inma: thoury
0.0
0.0 0.0
0.0 0.0
0.3 0.0
0.0
0.0
0.5
1.0
0.5
1.0
0.5
L.
0.3
1.0
0.0
0.5
1.0
Participantt 101
Participant 022
Participant 08
Farticipant 04
Particiocat 06
0.5
0.5
0.0
0.0
1.0
0.5
0.5
1.0
Participant 01
Participant 02
Participant
Participant 04
Partisiacnt 66
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0 0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
Figure 7: Gaussian textures. Measured perceptual scale for each participant. Top row: spatial frequency. Middle row: spatial frequency bandwidth. Bottom row: orientation bandwidth.
'hupss//
hhups://
15

---

Published as a conference paper at ICLR 2024
Partieipant 01
Partielpart
Purticipant
Participont
Partidirent
1.0
1.0
1.0
13
0.5
0.5
0.5
Tummen throey
humuun theoey
bormon theory
Termon Eboory
tumne thoury
0.0
0.0 0.0
0.0 0.0
0.0
0.0 0.0
0.0
0.5
1.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
Partielpaan 01
Partielpart 02
Purtielpant
Participant 04
Particignnt 05
1.0
0.5
1.0
0.5
0.5
Porticipant 01
Participanz 02
Participant
Participant 01
partictacnt 06
1.0
1.0
1.0
0.5
U.S-
0.5
0.5
0.5
0.0
0.0
0.0
0.3 0.0
0.0
0.0
0.5
1.0
0.0
Partielpant 01
Particlpart
Purticlpant
Participant
Particionnt 05
1.0
1.
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Participant 61
Participant 02
Participant 03
Farticipant 01
Participaat 06
0.5
0.5
0.5
Figure 8: Naturalistic textures, early sensitivity. Measured perceptual scale for each participant. Top-to- tootoottooommmooooooooo row: pairOl to pair05. Error bars represent 99.5% bootstrapped confidence intervals.
16

---

Published as a conference paper at ICLR 2024
Partieipant
Partielpart
Purticipant
Participont
Partidirent
1.0
1.0
1.0
13
10
0.5
0.5
0.3
Pimmon throey
humuun theoey
Termon thoory
Termon Eboory
Tnmmer thoury
0.0
0.0 0.0
0.0
0.0
0.0 0.0
0.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
Partielpant 01
Partielpart 01
Purtielpant
Participant 04
Partieigont 06
1.0
0.5
0.5
0.5
1.0
0.5
0.5
Porticipant 01
Participanzt 02
Participant
Participant 01
Particiant 06
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.3
0.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
Partielpant 01
Partielpart
Purticlpant
Participant
Particiount
1.0
1.
1.0
13
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Participant 61
Participant 02
Participant 03
Farticipant 01
Participaat 06
0.5
0.5
0.0
0.5
0.5
0.5
Figure 9: Naturalistic textures, late sensitivity. Measured perceptual scale for each participant. Top-to-bon tootoottoooommmoooooooooooooottt row: pair06 to pair10. Error bars represent 99.5% bootstrapped confidence intervals.
Participant 01
Participars 02
Participant
Participant 04
Particinent 05
1.0
0.5
brnman: throry
humsan theoey
Inzmon theory
Inmon thxory
huma thoory
0.5
0.5
1.0
0.0
0.5
0.5
0.5
Participent 01
Participon.l 112
Participant
Participant 04
Parkisi DG
1.0
1.0
1.0
0.5
0.5-
0.5
0.5
thuoery
theory
Lhoorry
Lhoory
liuuary
0.0
0.0
0.0
0.3
0.0
Figure 10: Naturalistic textures, conflicting prediction. Measured perceptual scale for each participant. Top and bottom row: pairl1 and 12ir12. Error bars represent 99.5% bootstrapped confidence intervals.
H ADDITIONAL EXPERIMENTE POWER SPECTRUM NORMALIZATION
One conclusion of our first set of experiments on the set of texture pairs we have chosen is that the perceptual scale is mainly explained by the change in the power spectrum along the path from
17

---

Published as a conference paper at ICLR 2024
texture the other. We decided normalize those textures by the average power spectrum between both textures of a pair and to run again a set of experiment. This manipulation deteriorate the perceptual discriminability between both textures as the manipulation changes the activation of VGG19 for each textures and for that reason we were not able to derive new predictions of the perceptual scale using the new VGG19 features. Instead we show in the result Figures 12 and 13 the initial prediction (whatever that means) also shown in Figure 2 of the main text. Surprisingly, some of the empirical results somewhat align with the predictions (Figure 12 paii r 02 to pair 00 and pair06 and pair08 to pairr 1110 10. We have no clue whether this observation is totally spurious if it underlies that the perceptual scale is explained by the higher-order statistics of the textures captured by VGG19, Indeed, how would it be possible if we have not even computed the prediction corresponding to the presented image ? Well, VGG19 trained on image-Net with some normalization, it is possible that the different layers of VGGI9 are performing normalization similar to the one we applied to the textures. Therefore, the predictions given by the original synthesized textures would be more correct that the one obtained with the power spectrum normalized textures. Unfortunately, the power spectrum manipulation also affects the level of noise estimated in the MLDS experiment (larger error bars in Figure [2 than in Figure 45 therefore the results are less reliable at the population level, Though, the individual results for the early and late sensitivity conditions shown in Figure 15 14 tend show that for some pair many participants have similar perceptual scale (despite larger enror bars) e.g. for pairOl participant 03/04; for pair02: participant 01 to 03 and participant 05, etc. This is not the case with the conflicting prediction condition (yellow, Figure 13
observe in Figure IT In addition,
In conclusion, those preliminary results leave us with lot of questions. A lot of work remain be done to properly understand what features are able to explain the observed perceptual scale.
Figure 11: Texture samples normalized by their average power spectrum. Red corresponds to the early sensitivity group. Blue corresponds to the late sensitivity group. Yellow corresponds to conflicting prediction across VGG-19 layers.
18

---

Published as a conference paper at ICLR 2024
pairO1
poir12
peir03
pair04
paird5
1.0
0.5
bucnan sheory
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
pair0G
poin07
poir08
pair09
pair10
1.0
0.5
Incaii sheory
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Figure 12: Measured and predicted (VGG19 averaged) perceptual scales for the early (top row) and late (bottom row) sensitivity pairs. Error bars represent 99.5% bootstrapped confidence intervals.
pairl1
pair
0.5
timorry
0.0
0.0
0.5
1.0
0.0
0.5
1.0
Figure 13: Measured and predicted (VGG19 averaged) perceptual scales for the conflicting prediction pairs. Error bars represent 99.5% bootstrapped confidence intervals.
19

---

Published as a conference paper at ICLR 2024
Partieipant 01
Partielpart
Purticipant
Participont
Partidiennt
1.0
1.
1.0
13
0.5
0.5-
0.5
0.5
Trmmon throey
humuun theoey
bormon theory
Iermon theory
farmen theory
0.0
0.0 0.0
0.0 0.0
0.0
0.0 0.0
0.0
0.5
1.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
Partielpaan 01
Partielpart 02
Purtielpant
Participant 04
Particigent 06
1.0
0.5
1.0
0.5
0.5
Participant 01
Participanz 02
Participant
Participant 04
Partictaent 66
1.0
1.0
1.0
0.5-
U.S-
0.5
0.5
0.5
0.0
0.0
0.0
0.3 0.0
0.0
0.0
0.5
1.0
0.0
0.5
0.5
Particlpant 01
Purticipart
Purticlpont
Participant
Particioant 06
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Participant 101
Participant 02
Participant 03
Farticipant 01
Particiaaat C6
0.5
0.5
Figure 14: Naturalistic textures, early sensitivity. Measured perceptual scale for each participant. Top-to- tottoottoommmoooooooooooooooottt row: pairOl to pair05. Error bars represent 99.5% bootstrapped confidence intervals.
20

---

Published as a conference paper at ICLR 2024
Partieipant 01
Partielpart
Purticipant
Participant
Partidirant
1.0
1.0
1.0
13
1.
0.5
0.5-
0.5
0.5
Promonn throey
homuun theoey
Termon thoory
Termon Eboory
Tarmee theury
0.0
0.0 0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Partielpann 01
Partielpart 012
Purtielpant
Participant 04
Partieigant 06
1.0
0.5
0.5
1.0
0.5
0.5
Porticcpantt 01
Participanzt 02
Participant
Participant 01
particiaent 06
1.0
1.0
1.0
0.5
U.S-
0.5
0.5
0.5
0.0
0.0
0.0
0.3
0.0
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
Particlpant 01
Partielpant
Purticlpant
Participont
Particioant
1.0
1.
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Participant 101
Participant 02
Participant 03
Farticipant 01
Participeat CG
0.5
0.5
0.10
0.5
0.5
Figure 15: Naturalistic textures, late sensitivity. Measured perceptual scale for each participant. Top-to-bot tottt-bottoom row: pair06 to p0ir Error bars represent 99.5% bootstrapped confidence intervals.
Participant 01
Partieipart D2
Purticipant
Participant 04
Partichant D6
0.5
0.5
hamman sheoey
bastoan theory
Ausana theary
kumaa theury
0.0
0.0
0.5
0.5
0.5
Participant 61
Participant 02
Participant
Participant 04
Partisiacnt CS
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
throrr
Hioorr
Eivory
lirery
0.0
0.0
0.0
0.0
0.0
0.5
1.0
8.0
0.0
0.5
Naturalistic textures, confficting prediction. Measured perceptual scale for each participant. Top and bottom row: pairll I and pair12. Error bars represent 99.5% bootstrapped confidence intervals.
Figure 16:
21


Reviews:
[{"review_id": "wd0L81rW2q", "reviewer": "Reviewer_gWW2", "summary": "The paper concerns itself with perceptual measures, in particular the perceptual distance between images showing what to me look like \"noise\" images each with a dominant spatial frequency. A theoiry is developed, and then tested experimentally.", "questions": "What does the parameter \"s\" mean?\n\nI do not understand the experiment. Three stimulli were presented with s1 < s2 < s3.  I suppose this means the parameter s somehow orders the stimulli, but I don't know how.  Then participants are required to pair either (s1,s2) or (s2,s3). Why remove the (s1,s3) option? (It's removal may bias results.)\n\nOverall, I get the impression the the authors have conducted perceptually experiments (on a very low number of participants) and that this paper is probably better suited to one of the perceptual psychology forums.  But, I found it hard to read, so I could very easily be wrong.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I very much approve of the style of the experiment.\nAnd I applaud all efforts to measure human-based distances.", "weaknesses": "I am not fully convinced by the model - the departure from human measures is significant.\n\nThe stimulli are very limited - gray scale images of textures that look to me like band-limited noise.  I am not at all sure how I should generlise any reults in this paper to general images. That is, it is not clear the model is general. \n\nI suppose what I would really like to see would be something akin to a just-noticable difference, and then to build a measure in frequency space based on jnds - by analogy to jnd in colour space.\n\n--\n\nNot all equations are not numbered, which is a presentation error because it makes discussion hard.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83CRHq9rTj", "reviewer": "Reviewer_AMLy", "summary": "This paper shows that the assumption that an observer has an internal representation of univariate parameters such as spatial frequency or orientation while stimuli are high-dimensional does not lead to contradictory predictions when following a theoretical framework. The perceptual scale is found to correspond to the transduction function in this framework and is related to the Fisher information of the generative model underlying perception. The research suggests that the stimulus power spectrum largely influences the perceptual scale. Furthermore, the study proposes that measuring the perceptual scale can help estimate the perceptual geometry of images, going beyond simple distance measurements to understand the path between images.", "questions": "This looks like careful and sophisticated work at first glance. I did not notice major defects in the paper, to my knowledge. However, the paper is difficult to follow and would benefit from careful editing.\nSince I do not have a solid background in this area, I cannot confidently evaluate the significance here.\nIt may be better if the authors can write their manuscript from the point of view of a general researcher in ICLR.\n\nAdditional question: Can the authors explain more on how measuring the perceptual scale helps estimate the perceptual geometry of images? This is claimed in the Abstract but rarely mentioned in the main body.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- A theoretical analysis on the perceptual scale in the case of GRFs was performed.\n- Different scaling experiments involving GRF and naturalistic textures were conducted.", "weaknesses": "- The theoretical analysis is performed for the case of GRFs, which does not apply to naturalistic textures (Note that Gaussian textures are a very limited class of images). Most of the Propositions are special cases of previous work, so what is the theoretical contribution of this work?\n- The different scaling experiments only involve a small set of pairs and 5 naive participants. There may be insufficient data for detailed analysis, and actually results were presented in Sec. 3 without any analysis.\n- The paper appears to be hastily written with many typos (e.g., page 5: e have -> we have; Fig. 2: the colors are wrong, and the caption's description conflicts with the main body, Page 8: Proposition 2 and Proposition 2; ...) and symbols are not always well-defined or explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E4pkG5yw0c", "reviewer": "Reviewer_wfqs", "summary": "The paper seems to measure the perceptual scale of spatial frequency, orientation, and synthetic texture interpolation.", "questions": "1. The goals of the paper should be more precise.\n\n2. The comparison to previous methods should be performed, and performed in a comprehensive way.\n\n3. The authors may want to test their models on natural photographic texture images, besides the synthetic and simplistic ones (as shown in Figs. 1 and 2).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem is of fundamental importance.\n\n2. The reviewer likes reading the introduction, especially the literature review.", "weaknesses": "1. The reviewer has read the paper multiple times but fails to comprehend the main contributions. \n\n2. The authors claim to have a convergence theorem, but what are the implications and practical relevance? Specifically, a Gaussian field is assumed, but high-dimensional natural images are highly non-Gaussian.\n\n3. The function $\\psi$ in Eq. (2) as the perceptual scale is tested in a very constrained scenario, without comparing to competing methods. For example, a recent computational model of the contrast sensitivity function considers spatio-temporal frequency, eccentricity, luminance, and area [C1].\n\n4. Moreover, the experimental results regarding texture interpolation are performed in a discriminative setting, not from a generative perspective (i.e., texture synthesis).\n\n\n[C1] stelaCSF: A unified model of contrast sensitivity as the function of Spatio-Temporal frequency, Eccentricity, Luminance and Area,\n SIGGRAPH 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tWMlD0xAYd", "reviewer": "Reviewer_qN8s", "summary": "The authors derive nonlinear relations between certain univariate image features and the perceptual scale (human response) to these features using Fisher information. \nWhile the response for frequency and orientation of Gaussian Random Fields proves to be quite consistent with actual human responses derived from similarity judgements, the situation is more complicated in naturalistic textures.\nFor Gaussian fields the correspondence is good because (1) these images are easy to characterize with those parameters and hence, the Fisher information of those parameters captures all the information of the multidimensional objects, and (2) the discriminability is actually related to the image information. However, for the responses to interpolation between naturalistic textures, it is not clear which features to consider to compute the Fisher information. In this case, the authors try different features (pixel values, wavelet responses, VGG19 responses, and the power spectrum), and the best agreement is found using the power spectrum.", "questions": "MAJOR QUESTIONS:\n\n* Please give examples interpolating between more general textures (e.g. a brick wall, and a flower field, or a pile of fruits -see examples of natural textures in Portilla & Simoncelli IJCV 2000-). Do you get similar results (theoretical predictions and human responses) in those cases?\n\n* The description of the method is confusing. For example, how do you get the nonlinear response in Fig. 1? I guess first you use the expressions in appendix A, then, you use \\Psi in Eq. 6 and then you integrate, right?. This is not clear in the text. Similarly, how do you get the predictions in Fig. 2?. For the VGG response you assume they are Gaussian vectors, then you apply Proposition 4, and then you integrate Eq. 6?.  \n\n* Appendices give explicit expressions for the Fisher information for the frequency and orientation of Gaussian fields and of Gaussian vectors for the activations of VGG-19 (if they were Gaussian), but how do you compute J for the pixel representation? (just apply an FFT and then use that estimation of the spectrum and the formula in Preposition 3?). \nHow do you do it for wavelets?... What is the wavelet decomposition you used? There are many of them!\n\n* The normalization of the response axes imply that all modifications of the stimuli lead to the same perceptual distortion. Simple visual inspection of the pairs in Fig. 2 shows that this is not correct. If the authors do not propose a way to give an absolute scaling in these different dimensions they should not claim that they are giving a metric of the image space.\nWhat they just provide is a nonlinear (up to a scalar) relation between displacements in certain directions and variations in the inner representation, but this is not a metric.\n\n* Following the above comment, the proposed method would be unable to give a measure to predict the Mean Opinion Score (MOS) on distortion in databases such as TID [Ponomarenko et al. 13] or KADID [Lin et al. 19]. If this is not the case, the authors should mention how to infer this MOS. \n\n* The title is too vague, it does not reflect the content of the paper, and actually overstresses \"distances\" and \"metrics\" not quite addressed in the work. What about changing the title by something like: \"Nonlinear image representations in humans from Fisher information\".\n\nMINOR ISSUES:\n\n* In the abstract authors say \"we demonstrate that it is related to the Fisher information of the generative model that underlies perception\", while it should say \"we demonstrate that it is related to the Fisher information of the generative model that underlies the stimuli\"\n\n* In the first paragraph of page 6 the authors say \"We will see that in both cases...\" ... Where do the authors show this? (this is related to the confusing description of the method stated above).\n\n* Typos:  \nfirst paragraph of page 8 \"VGG-19 to another (bottom-right of Fig. 2)\" ---> \"VGG-19 to another (bottom-left of Fig. 2)\"\nsecond paragraph page 8 \"frequency mode (Proposition 2 and Proposition 2)\" ---> ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors address an interesting issue (the nonlinearity of the human image representation), and they show that Fisher information may be a driving factor of discriminability in simple textures defined by band-pass power spectra. \nThe provided expressions of Fisher information have technical interest.", "weaknesses": "It is not clear how to generalize their approach to more complicated textures, where it is not obvious the features to describe them and how to compute Fisher information, J, from those. \n\nIn the more complicated scenario the authors only try a limited (non conclusive) set of features on some so-called naturalistic textures which are not very general either. All this reduces the strength of the experiments and the possible conclusions. Moreover, in some of the cases (e.g. pixel and wavelet features) they do not explain how to compute J.\n    \nGiven the normalization of the response axes, the derived nonlinear functions cannot be used as a metric to compute differences between textures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)