Title: Information Geometry of the Retinal Representation Manifold
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural coding

theoretical neuroscience

stochastic methods

neural networks
Abstract: The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations of naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created a stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also a variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli to study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing an examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are information-limiting rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less in the population than for single cells and that as a function of firing rate, Fisher information varies less than sensitivity. We conclude that under natural scenes, population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.

Information Geometry of the Retinal Representation Manifold

Abstract

The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations of naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created a stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also a variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli to study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing an examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are information-limiting rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less in the population than for single cells and that as a function of firing rate, Fisher information varies less than sensitivity. We conclude that under natural scenes, population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.

1 Introduction

Neural populations represent information with their collective activity. In the field of sensory neuroscience, whereas there is much focus on characterizing the sensitivity of neurons to different stimuli, the true function of a sensory system is to discriminate between stimuli. A quantitative description of discriminability in a neural population requires not only a precise knowledge of sensitivity to stimuli, but also the structure of noise correlations in the population.

Neural representation manifolds constitute a modern approach to studying neural populations in a geometric framework, and recent machine learning literature has seen extensive research on the geometry of representation manifolds of neural networks trained for artificial tasks [1, 2, 3, 4, 5]. In separate work, Fisher information has been widely utilized in neuroscience for measuring decoding fidelity, particularly in theoretical studies exploring the effect of noise correlations [6, 7, 8]. In this study, we propose a framework for studying the Fisher information geometry of representation manifolds for natural visual scenes in a population of visual neurons, in particular the retina. In addition, for the first time, we apply such a manifold analysis to the neural representation of natural stimuli for real data.

The discriminability of biological visual systems has long been studied for low-dimensional and discrete artificial stimuli [9, 10, 11]. However, naturalistic visual stimuli in the real world are inherently high-dimensional and discriminability is generally heterogeneous across different types of changes in a stimulus. Recent advances in convolutional neural networks (CNNs) have enabled the accurate prediction of retinal responses to naturalistic stimuli, as well as ethological computations [12, 13, 14]. Nevertheless, previous work has predominantly focused on deterministic trial-averaged neural codes, which is not adequate for computing discriminability. Thus an encoding model of stochastic retinal representations for natural scenes is required, which constitutes one of the primary contributions of this work.

We present here a highly accurate model of both sensitivity and stochasticity for the retinal ganglion cell (RGC) population, thus creating the first accurate model of discriminability in a neural population for its natural inputs. From an analysis of the Fisher information geometry of the retinal representation of stimuli, we find that the most discriminable direction for the RGC population varies greatly across stimuli and is correlated with the mean neural response. We further address a long standing question as to the effect of noise correlations on information. We find that the most discriminative mode in the representation space often aligns with the most stochastic direction, implying that noise correlations in the retina are detrimental to information coding. Finally, we analyze how discriminability depends on firing rate of individual cells and in the population, and discuss the implications in relation to principles of information maximization.

2 Theory

In this section, we describe basic theory of information geometry [15] and its application to sensory neuroscience. Let x denote the stimulus vector and y denote the spike count vector whose dimension is equal to the number of neurons encoding the stimulus. Each stimulus induces a conditional probability distribution of sensory neural responses P ( y | x ) and mean firing rates µ ( x ) := E [ y | x ] . Along any decoding vector a , the sensitivity is defined as ||∇ x ⟨ µ ( x ) , a ⟩ || , while the stochasticity is defined as the standard deviation of ⟨ y , a ⟩ conditioned on x (Figure 1). In our case in the retina, µ ( x ) is modeled by a noiseless CNN and P ( y | x ) is modeled by the stochastic counterpart.

The representation manifold is defined as the manifold of mean neural responses induced by stimuli: { µ ( x ) } . We also define a statistical manifold of conditional response distributions { P ( y | x ) } parameterized by stimuli x , which yield a natural coordinate system. To equip the statistical manifold with a Riemannian metric, we consider the Kullback-Leibler divergence between two nearby distributions:

where I ( x ) is the Fisher information matrix with respect to the stimulus, and the KL divergence is linearized up to the second order. The Fisher information metric g stim | x ( e i , e j ) = I ( x ) ij naturally characterizes the local discriminability of stimuli near x .

When we assume that neural responses follow multivariate Gaussian distributions y | x ∼ N ( µ ( x ) , Σ ( x )) and the covariance matrix is locally independent of the stimulus, d Σ ( x ) d x = 0 ,

Stimulus space

Representation manifold

X

TpM

SensoryNetwork

Differentiation

Noise

the Fisher information metric can be expressed as

where d µ ( x ) d x is the Jacobian matrix. Eq. 2 is commonly termed linear Fisher information which bounds the performance of unbiased linear decoders, which may be more biologically plausible than nonlinear decoders [16, 17, 18]. The latter assumption is a good approximation in our case because a neural network consisting only of ReLU nonlinearities is locally equivalent to an effective linear network [19] and the covariance matrix of a linear network is a constant matrix. This metric expression is interpretable in that discriminability is proportional to sensitivity and inversely proportional to noise. The length of a geodesic connecting two stimuli under this information metric measures the discriminability between them, when represented by the neural population.

Equivalently, the mean response µ ( x ) on the representation manifold can also serve as a coordinate system. Then the metric is given by g resp | x ( e i , e j ) = Σ -1 ( x ) ij . The metric matrix transforms between the stimulus space and the representation space as a tensor [1].

We define here several important quantities for analysis. The eigenvector of I ( x ) associated with the largest eigenvalue is the most discriminable direction in the stimulus tangent space, which we term the most discriminable input (MDI) at stimulus x . The square root of the eigenvalue is the corresponding discriminability [20]. The MDI in the stimulus space can be pushed forward to the most discriminative response mode (MDR) in the representation tangent space via the Jacobian matrix: MDR ∝ d µ ( x ) d x · MDI. Similarly, the most sensitive direction in the stimulus (input) tangent space

(MSI) can be obtained as the principal eigenvector of d µ ( x ) d x T · d µ ( x ) d x and the associated most sensitive response mode (MSR) obeys MSR ∝ d µ ( x ) d x · MSI. Additionally, the most stochastic (noisy) response mode (MNR) is defined as the principal eigenvector of the noise covariance matrix Σ ( x ) , which is no more than principal component analysis. For a ReLU neural network with small stochasticity, the stimulus dependence of the aforementioned tangent space features is solely derived from the stimulus dependence of the effective linear network [19].

One practical challenge of the above analysis with natural visual stimuli is the extremely high dimensionality of the stimulus space. Fortunately, we only need to compute the Fisher information matrix in a subspace defined by the recorded cell population. First we define the instantaneous receptive field for each cell as the direction of the greatest sensitivity to the stimulus at one stimulus point, which is the gradient of that cell's response, d µ i ( x ) d x [13]. Then we only need to compute Fisher information in the subspace spanned by instantaneous receptive fields, as justified by the following theorem.

Theorem : The top N most discriminable directions are linear combinations of the instantaneous receptive fields, where N is the number of output neurons.

Proof: Let IR := span ( { d µ i ( x ) d x } ) represent the subspace spanned by instantaneous receptive fields. Suppose that the most discriminable direction v 1 is not in IR , then we can decompose v 1 = v o 1 + α v p 1 , where v o 1 is the orthogonal component in IR ⊥ and α v 1 p is the parallel component

Convolutional

Convolutional

Convolutional

One-hot

Binomial

Stimulus

layer

layer

layer

layer

Noise

spike count

2

0

Ckernels

8kernels

8kernels

time

time

N(0,g2)

N(0, g2)

N(0, 9°)

which contradicts with the fact that v 1 is the most discriminable direction. Therefore, v 1 is in IR .

Replacing IR with IR ∩ { v 1 } ⊥ , with the same analysis we can prove that the second most discriminable direction v 2 is also in IR . Subsequently, the whole theorem can be proven.

3 Methods

The spiking activity of a population of tiger salamander retinal ganglion cells was recorded in response to a sequence of jittering natural images. We created a stochastic encoding model capturing P ( y | x ) with the CNN architecture that included independent noise in the stimulus and in each unit of the CNN. The CNN parameters were trained using backpropagation to fit the mean firing rates µ ( x ) to create a deterministic model, and then these parameters were frozen. Noise parameters included one parameter for each layer to set the standard deviation of the independent noise, and one parameter for binomial noise in each ganglion cell. These were then optimized by maximizing the log-likelihood of the empirical data and exploring the parameter space to match a few second-order statistics of the neural responses. Once the parameter optimization was completed, we computed the Fisher information metric with the model numerically for each stimulus and analyzed the most discriminable directions.

3.1 Stochastic model

The model architecture is shown in Figure 2. The model takes a spatiotemporal visual stimulus as an input and its output is a set of spike counts, one for each neuron in each time bin. The deterministic part of the model including its hyperparameters and training method is adopted from Ref. [12, 13, 14] which is a three-layer CNN chosen after an extensive architecture search. This CNN is the SOTA retinal encoding model for natural scenes in that (1) the prediction accuracy of RGC mean responses outperforms other models; (2) the activity of internal units is correlated with retinal interneuron responses recorded separately and not used in training; and (3) the model trained only on natural scenes can reproduce a wide range of phenomena induced by artificial stimuli. Thus, the deterministic part of our model has a clear correspondence with the real retina in that its circuitry is mechanistically interpretable and it captures many retinal computations. Every convolutional filter in our model was implemented by linearly stacking a sequence of 3 × 3 small filters, which outperforms the traditional method [21]. A parametric tanh nonlinearity was attached to the last convolutional layer for the

purpose of enforcing the refractory period constraint:

where a, b are cell-dependent model parameters, and r max represents the cell-dependent maximal firing rate.

The essential module that enables us to study retinal population coding is the one-hot layer. Model parameters are fit with the one-hot layer, which bijectively maps a few output units of the final convolutional layer to recorded neurons by the end of the training. The one-hot layer is then removed after parameter optimization so that we can analyze the representation of the full population, typically consisting of thousands of units. The rationale behind this method is the mosaic organization of each retinal cell type, in which all ganglion cells of a given type tile the retina with their dendrites [11]. Formally, the one-hot layer is defined as

where the input is the output of the last convolutional layer, k is the index of recorded neurons, i, j (and p, q below) are location indices, w k is the linear combination weight vector, and C ( k ) is the neuron-to-channel map. Each w k will converge to a one-hot array after model training with our one-hot loss function, which is adapted from the semantic loss in Ref. [22]

and which reflects the locations of recorded neurons. Each output channel represents a ganglion cell type and C ( k ) was determined using hierarchical clustering of output channels (see the Supplementary Material for details).

Gaussian white noise was added to the stimulus to simulate stochasticity in photoreceptors, and was also added to pre-threshold signals in the first two layers to simulate stochasticity in bipolar cells and amacrine cells. We found that Gaussian noise outperforms Poisson noise in these layers, which can presumably be attributed to the noise generation mechanism in the retina and the central limit theorem. The stochasticity of ganglion cells was modeled as a parametric binomial probability mass function:

where n denotes the spike count in a time bin, r is the rate parameter having a one-to-one correspondence with the mean firing rate µ , k is the variability parameter to be optimized, and N is the cell-dependent upper bound of spike counts in one time bin reflecting the refractory period of that cell, which one can directly determine from data. In practice, for given k and N we can interpolate the r -µ mapping and use it to determine the rate parameter producing the desired mean firing rate, i.e., µ ( r ) = E p ( n ; r,k,N ) ( n ) . Although we have explored a variety of probability mass functions to fit the sub-Poisson statistics of retinal ganglion cells [23, 24], the binomial noise Eq. 7 performs the best among them (see the Supplementary Material) and also is the most natural choice considering the refractory period constraint. We stress that the binomial noise on ganglion cells is independent noise, and thus will not qualitatively affect the geometry and the functional implication of noise correlations.

3.2 Optimization

Since the second order statistics are functions of the entire dataset and cannot be decomposed into terms depending on the individual stimulus, we optimized deterministic parameters and stochastic

parameters separately. CNN parameters and one-hot parameters { w k } were optimized together with the standard Poisson loss function and L one-hot to fit mean firing rates smoothed using a 10 ms Gaussian filter. The optimization was performed using ADAM [25] via Pytorch [26] on NVIDIA TITAN Xp, GeForce GTX TITAN X, GeForce RTX 3090, TITAN RTX, TITAN V, and GeForce RTX 2080 Ti GPUs. The network was regularized with an L2 weight penalty at each layer and an L1 penalty on the model output. Then tanh parameters of the final nonlinearity were fitted with the least square regression. Then we optimized stochastic parameters while freezing all deterministic parameters. Variability parameters k were optimized by maximizing the log-likelihood of recorded spike counts based on empirical firing rates. Standard deviations of Gaussian noise { g 0 , g 1 , g 2 } in each layer were optimized through grid searches in order to best fit empirical noise correlations and stimulus correlations by minimizing the smoothed and weighted average of mean squared errors.

3.3 Computation of Fisher information metric

For each stimulus, the noise covariance matrix Σ was estimated by repeatedly simulating the stochastic model thousands of times. We then computed the eigendecomposition of Σ obtaining local information of the representation manifold: Σ = ∑ i λ i v i v i T . We applied the Gram-Schmidt orthogonalization to instantaneous receptive fields to obtain a basis of IR , which is a subspace of the stimulus tangent space. Then following Eq. 2, the Fisher information matrix with respect to the stimulus can be computed as

where z is the stimulus coordinate under the IR basis. In practice, we summed only over hundreds of the most stochastic modes that can explain 85% to 90% of the variance, as the sensitivity and stochasticity of the other modes are both close to zero indicating that those modes are orthogonal to the tangent space of the representation manifold. The gradient with respect to the stimulus was computed using the noiseless model.

4 Results

4.1 Model evaluation

Early stochastic encoding models of the retina were primarily developed for artificial stimuli [27, 28, 29], and their results cannot be directly generalized to natural scenes due to the stimulusdependent nature of noise in the retina. Here our fitted model was evaluated by comparing with neural data various commonly used second-order statistics including correlation measures and single-cell variability measures (Figure 3). Surprisingly, the model prediction matches the neural data remarkably well despite that the model only has a few noise parameters. This result suggests that because noise in the retina propagates through the same network pathways as the signal, they are both captured by the CNN structure. This indicates a strikingly simple origin for noise correlations between all ganglion cell pairs: independent noise that starts in individual upstream cells, and then propagates through the same network that creates mean stimulus sensitivity. Statistics for preparations with longer test sequences tend to be more reliable and consequently our model performs better on these. For the first order statistics (mean firing rates), the Pearson correlation coefficient between model predictions and data ranges from 70% to 80% , which is similar to Ref. [12, 13]. Such an accurate model of both sensitivity and stochasticity guarantees that the computation of Fisher information matrix is reliable (Eq. 2).

4.2 The most discriminable directions

Some examples of the most discriminable stimulus direction (MDI) and the most discriminative response mode (MDR) are shown in Figure 4a (see the Supplementary Material for more examples). Often the top eigenvalues were similar, and so to avoid over-interpreting the importance of individual eigenmodes, we found the sparsest direction (with the least L1 norm) in the space spanned by the top 5 MDIs for the purpose of visualization. Only spatial components are presented here as we found that temporal components for different stimuli are both very similar to each other and to those of instantaneous receptive fields [13]. We found that the MDI is strongly stimulus dependent, and is

(a)

(b)

Total correlation

Stimuluscorrelation

Noisecorrelation

0.6

0.45

0.08

0.4

0.30

0.04

0.2

0.15

0.00

0.0

0.00

0.0

0.2

0.4

0.6

0.00

0.15

0.30

0.45

0.00

0.04

0.08

Data

Data

Data

trial-to-trial correlation

Data

Model

0.6

0.4

0.2

0.0

factor

0.9

0.6

Fano t

0.3

0.0

1 2 34 5 6 7 8910111213141516171819202122232425262728293031323334

cells

typically localized in a region where the sensitivity is high, an effect that arises from the locality of convolutional connections. The MDI tends to appear around the central region (red box in Figure 4a) because pixels within this area affect more output units than pixels near the edges or corners.

One general observation is that the MDI varies greatly across stimuli. From a mathematical point of view, this is to say that certain features of the tangent space depend on the base point. With respect to neural function, this variability potentially indicates the presence of adaptation to the stimulus. We found that the most discriminable direction is related to the base point through the representation space such that the MDR and the mean response are correlated (Figure 4b). This correlation can be attributed to the fact that neurons with intermediate firing rates exhibit the highest single-cell sensitivity and discriminability (Figure 6a). As a result,the MDI sometimes captures the most salient feature of the stimulus (the first row of Figure 4a), although this is not always the case (the second row of Figure 4a). Similar stimuli and mean responses likely produce distinct MDIs especially when their mean response pattern has multiple spatial components that can each yield a localized MDR (the third and fourth rows of Figure 4a). Moreover, the correlation between MDR and mean response provides a mechanism for higher brain regions to estimate the most informative future response changes of retinal ganglion cells based on the current response and thereby adjust the corresponding sensitivity, which could have benefits for the property of predictive coding [30]. Additionally, we found that

(a)

(b)

(c)

Stimulus

MDI

Response

MDR

Stimulus+MDI

Cosinesimilaritybetweentop5MDRsandmeanresponse

shuffled

ordered

0.15

Frequency

0.10

0.05

0.00

0.0

0.3

0.6

0.9

Cosine

different parts of the stimulus generates the MDR for different cell types (Figure 4c), indicating that different cell types signal different stimulus regions in the image as conveying the most information.

4.3 Noise correlations in the retina limit information coding

The role of noise correlations in population coding has long been debated. Multiple scenarios have been advanced where noise correlations are detrimental to information coding [31, 10]. However, others suggest that noise correlations under certain circumstances may not limit information coding and can even be beneficial [32, 6, 9]. More specifically, it has been proposed that in response space, noise correlations decrease information when the differences in the signal are aligned with the noise direction, and increase information when they are orthogonal [33] (Figure 5a).

Here we tested these two hypotheses for the retina by computing cosine similarities between MDRs and MNRs. It turns out that the most discriminative mode is quite aligned with the most stochastic mode and orthogonal to modes with small variability (Figure 5b), implying that noise correlations in the retina are information-limiting correlations. We also performed a more direct test of this conclusion by analyzing trial-shuffled responses, which have independent noise. We found that removing noise correlations in this way created responses that were significantly more discriminative than the original model outputs with correlated noise for every population size (Figure 5c). To understand the mechanism of this reduction in information, we analyzed the eigenspectrum of the Fisher information matrix (Figure 5d). Both the sensitivity and the stochasticity increase with the discriminability of MDR, and their ratio reaches a maximum at the noisiest mode. Therefore, we propose that the fundamental reason for such information-limiting correlation is that noise propagates through the same network as the signal, causing the stochasticity ellipsoid to align closely with, and be rounder than, the sensitivity ellipsoid. One trivial and extreme example of this arrangement is when all noise originates at the level of the stimulus, e.g. from photoreceptors, then the resulting noise correlations will obviously be detrimental.

(a)

(b)

(c)

(d)

CosinesimilaritybetweenMDRandMNR

0.45

+

1st MDR

+

2ndMDR

3rd MDR

△I = 0

△I>0

△<O

+

4th MDR

0.30

5th MDR

Cosine

S1

S1

Neuron 2

Neuron2

2

S1

Neuron2

2

0.15

S2

S2

S2

0

8

16

24

Neuron1

Neuron1

Neuron1

MNRindex

Spectrum of Fisher information matrix

2.0

correlatednoise

independentnoise

1.5

1.2

Discriminability

1.5

Stochasticity

Sensitivity

0.8

1.0

1.0

0.4

0.5

0.5

0.0

0.0

0

100

200

300

0

60

120

180

#cells

MDRindex

4.4 Firing rate dependency

The firing rate dependencies of stochasticity, sensitivity, and discriminability for individual neurons and the whole population are shown in Figure 6. For individual neurons, both the sensitivity and the stochasticity peak around intermediate firing rates and decay at high firing rates (a consequence of the refractory period). Thus, the single-cell discriminability also saturates around the intermediate firing rate. In contrast, population coding does not exhibit such a decay at high rates, implying that neurons can encode information complementarily, which is a benefit of population coding. In other words, even when a fraction of neurons get saturated by the stimulus and become uninformative at high firing rates, other neurons can still provide discriminative responses to stimulus changes. Although the population sensitivity and stochasticity significantly increase with the mean firing rate, the firing rate dependency of the population discriminability remains relatively flat, which is potentially beneficial for information transmission assuming that stimuli are uniformly distributed on the natural scene manifold [35, 36].

5 Discussion

In summary, we established a novel information-geometric framework to understand the high dimensional representation manifold of the retinal population for natural scenes, one that can also be

(a)

(b)

Individual neurons

Population

mean

60

2.4

Stochasticity

Stochasticity

0.3

0.8

0.0

0.0

4

mean

0.75-

3

Sensitivity

Sensitivity

0.50

0.25

1

0

2.0

mean

Discriminability

1.2

Discriminability

1.5

0.8

1.0

0.4

0.5

0.0

0.0

0

80

160

240

0

25

50

75

Firing rate (Hz)

Mean firing rate (Hz)

generalized to other neural systems. However, there still exist remaining questions and further work that can be pursued within this framework.

First, our analysis primarily focused on examining the local metric tensor, which characterizes the effects of infinitesimal stimulus changes. It would be interesting to extend the analysis to finite stimulus changes to explore more global geometry of the representation manifold. For example, one exciting direction is to find geodesics between two stimuli under the information metric and observe how one stimulus transitions into another with the least amount of discriminable changes for the retinal population [37].

Second, we found that noise correlations in the retina are information-limiting correlations. This conclusion is surprisingly robust against many hyperparameter selections, even when second order statistics are not fitted well. Therefore, it remains unclear whether this is a general result for all feedforward networks with independent noise applied to their inputs and intermediate layers. If this is indeed the case, noise correlations in other brain areas that do not limit information coding must arise from different mechanisms.

Finally, the Fisher information metric in this work was computed in the full stimulus space. However, most points in the stimulus space represent meaningless noise images that no animal has ever encountered. Therefore, an important avenue for future research would involve incorporating the structure of natural stimuli and computing the metric with stimuli constrained to a natural-scene manifold.

References

Michael Hauser and Asok Ray. Principles of riemannian geometry in neural networks. Advances in neural information processing systems , 30, 2017.

Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications , 11(1):746, 2020.

Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences , 119(43):e2200800119, 2022.

Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 315-323, 2018.

Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations , 2021.

Rubén Moreno-Bote, Jeffrey Beck, Ingmar Kanitscheider, Xaq Pitkow, Peter Latham, and Alexandre Pouget. Information-limiting correlations. Nature neuroscience , 17(10):1410-1417, 2014.

Larry F Abbott and Peter Dayan. The effect of correlated variability on the accuracy of a population code. Neural computation , 11(1):91-101, 1999.

Bruno B Averbeck and Daeyeol Lee. Effects of noise correlations on information encoding and decoding. Journal of neurophysiology , 95(6):3633-3644, 2006.

Oleg I Rumyantsev, Jérôme A Lecoq, Oscar Hernandez, Yanping Zhang, Joan Savall, Radosław Chrapkiewicz, Jane Li, Hongkui Zeng, Surya Ganguli, and Mark J Schnitzer. Fundamental bounds on the fidelity of sensory cortical coding. Nature , 580(7801):100-105, 2020.

Rava Azeredo da Silveira and Fred Rieke. The geometry of information coding in correlated neural populations. Annual Review of Neuroscience , 44:403-424, 2021.

David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience , 14(10):1317-1322, 2011.

Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. Advances in neural information processing systems , 29, 2016.

Niru Maheswaranathan, Lane T McIntosh, Hidenori Tanaka, Satchel Grant, David B Kastner, Joshua B Melander, Aran Nayebi, Luke E Brezovec, Julia H Wang, Surya Ganguli, et al. Interpreting the retinal neural code for natural scenes: From computations to neurons. Neuron , 111(17):2742-2755, 2023.

Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems , 32, 2019.

Shun-ichi Amari. Information geometry and its applications , volume 194. Springer, 2016.

Jeffrey Beck, Vikranth R Bejjanki, and Alexandre Pouget. Insights from a simple expression for linear fisher information in a recurrently connected population of spiking neurons. Neural computation , 23(6):1484-1502, 2011.

Ingmar Kanitscheider, Ruben Coen-Cagli, Adam Kohn, and Alexandre Pouget. Measuring fisher information accurately in correlated neural populations. PLoS computational biology , 11(6):e1004218, 2015.

Adam Kohn, Ruben Coen-Cagli, Ingmar Kanitscheider, and Alexandre Pouget. Correlations and neuronal population information. Annual review of neuroscience , 39:237-256, 2016.

Haozhe Shan and Haim Sompolinsky. Minimum perturbation theory of deep perceptual learning. Physical Review E , 106(6):064406, 2022.

Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Eigen-distortions of hierarchical representations. Advances in neural information processing systems , 30, 2017.

Xuehao Ding, Dongsoo Lee, Satchel Grant, Heike Stein, Lane McIntosh, Niru Maheswaranathan, and Stephen Baccus. A mechanistically interpretable model of the retinal neural code for natural scenes with multiscale adaptive dynamics. In 2021 55th Asilomar Conference on Signals, Systems, and Computers , pages 287-291. IEEE, 2021.

Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning , pages 5502-5511. PMLR, 2018.

Michael J Berry, David K Warland, and Markus Meister. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences , 94(10):5411-5416, 1997.

Xaq Pitkow and Markus Meister. Decorrelation and efficient coding by retinal ganglion cells. Nature neuroscience , 15(4):628-635, 2012.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32:8026-8037, 2019.

EJ Chichilnisky. A simple white noise analysis of neuronal light responses. Network: computation in neural systems , 12(2):199, 2001.

Justin Keat, Pamela Reinagel, R Clay Reid, and Markus Meister. Predicting every spike: a model for the responses of visual neurons. Neuron , 30(3):803-817, 2001.

Jonathan W Pillow, Liam Paninski, Valerie J Uzzell, Eero P Simoncelli, and EJ Chichilnisky. Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. Journal of Neuroscience , 25(47):11003-11013, 2005.

Laurence Aitchison and Máté Lengyel. With or without you: predictive coding and bayesian inference in the brain. Current opinion in neurobiology , 46:219-227, 2017.

Ingmar Kanitscheider, Ruben Coen-Cagli, and Alexandre Pouget. Origin of information-limiting noise correlations. Proceedings of the National Academy of Sciences , 112(50):E6973-E6982, 2015.

Jon Cafaro and Fred Rieke. Noise correlations improve response fidelity and stimulus encoding. Nature , 468(7326):964-967, 2010.

Bruno B Averbeck, Peter E Latham, and Alexandre Pouget. Neural correlations, population coding and computation. Nature reviews neuroscience , 7(5):358-366, 2006.

Elad Schneidman, Michael J Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in a neural population. Nature , 440(7087):10071012, 2006.

Simon Laughlin. A simple coding procedure enhances a neuron's information capacity. Zeitschrift für Naturforschung c , 36(9-10):910-912, 1981.

Xue-Xin Wei and Alan A Stocker. Mutual information, fisher information, and efficient coding. Neural computation , 28(2):305-326, 2016.

Olivier J Hénaff and Eero P Simoncelli. Geodesics of learned representations. In 4th International Conference on Learning Representations, ICLR 2016 , 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a fitted surrogate neural network model to approximate the Fisher information metric induced on image space by the retinal ganglion cell population code. They use this analysis to argue that noise correlations in the retina are information-limiting.\n\nSTRENGTHS:\n1. The question of whether noise correlations in early sensory areas are information-limiting is a classic and broadly interesting issue in theoretical and systems neuroscience. This paper offers a novel approach to the problem of Fisher information estimation. \n\n2. I think the observation that the most discriminable stimulus depends on the base point (Lines 196-208) is an intriguing finding, though perhaps not so surprising. Making a convincing link between this variation and neural adaptation would be an interesting topic for future study (I fully acknowledge that a detailed characterization of this phenomenon is likely beyond the scope of the present manuscript).\n\nWEAKNESSES:\n1. I have several concerns regarding the robustness of the paper's conclusions to the architecture and goodness of fit of the surrogate model. One key strength of some past work on information limiting correlations---I have in mind Rumyantsev et al, cited as [8] in the submitted manuscript---is the demonstration that the quantities of interest can be accurately resolved given a number of measurements comparable to the number of experimental recordings. It is not clear to me whether the same should be true here. Can the authors provide evidence that their approach gives accurate estimates of the directions of maximal variation, and that the surrogate model is not overfit?\n\n2. In a similar vein, the authors argue reasonably convincingly that it is reasonable to neglect stimulus-dependence of the noise covariance (i.e., $d\\Sigma/dx \\simeq 0$) for a ReLU network model, but do not give direct evidence that this is a reasonable assumption for the retinal population code. \n\n3. In several places, data supporting the authors' analysis decisions are not shown, and their choices are not always clearly described. For example, in the truncated approximation (8) for the Fisher information matrix, can you specify (at least in the SI) the precise criterion used to select the \"hundreds of the most stochastic modes\" included (Lines 170-171)?\n\n**A pedantic concern:** The paper and supplementary material contain several small violations of the anonymity requirements. My score for the paper does not take this concern into account. \n\n- Lines 257-259: \"Acknowledgements: This work was supported by grants from the NEI, R01EY022933, R01EY025087 and P30EY026877 (SAB).\"\n\n- In the supplementary ZIP, under code, the LICENSE file contains the line \"Copyright (c) 2023 Baccus Lab.\"\n\nQUESTIONS:\n- Line 33: In addition to Wang & Ponce 2021, earlier work by Shao et al., \"The Riemannian geometry of deep generative models\" (2018) should be cited.\n\n- Lines 73-81: It could be useful to mention that (8) has been termed the \"linear Fisher information\" in past works, and to cite Beck et al., \"Insights from a Simple Expression for Linear Fisher Information in a Recurrently Connected Population of Spiking Neurons\" (NECO 2011) and Kanitscheider et al., \"Measuring Fisher Information Accurately in Correlated Neural Populations\" (PCBI 2015). \n\n- Line 84: Why is [1] cited rather than a general text on Riemannian manifolds (or, indeed, [14])?\n\n- Line 86: Using the acronym \"MDS\" for \"most discriminable stimulus\" conflicts with the standard use of \"MDS\" to mean \"multidimensional scaling.\" Please consider using an alternative acronym, e.g., \"MDI\" for \"most discriminable input.\"\n\n- Lines 152-153: Though the binomial noise model yields the best fit, could it still be worthwhile to reproduce Figure 5 for the alternative noise models, as a sort of robustness check? \n\n- Line 157: Please provide more detailed information on compute resources than \"on NVIDIA GPUs.\"\n\n- Lines 185-186: \"Statistics for preparations with longer test sequences tend to be more reliable and consequently our model performs better on these.\" Data to support this claim is not shown, correct? It would be useful to show more clearly the effects of this heteroskedasticity. \n\n- Figure 4: Please state in the caption whether these measures are computed on held-out stimuli.\n\n- Lines 206-208: Could you make this apparent link to ideas of hierarchical predictive coding more precise? \n\n- Figure 6a-b: It would be useful to remind the reader of the definitions of stochasticity, sensitivity, and discriminability in the caption. \n\n- Figure 6a: In the three sub-panels of Panel (a), there are so many dots overlaid on top of each other that it is hard to tell how single neurons are distributed within the blobs of data. Showing 2D histograms might be more informative. It would also be useful to plot stochasticity, sensitivity, and discriminability against one another rather than only against firing rate (depending on the result, this could be deferred to the supplement). \n\n- Figure 6b: The linear fit to the discriminability-mean firing rate relationship is not very convincing because of the substantial spread in the data. \n\n- Lines 244-247: I agree that it would be interesting to compute geodesics, but I'm less optimistic that this could be done in practice due to the numerical challenges associated with solving the geodesic equation in high dimensions. One previous attempt by Hénaff and Simoncelli (\"Geodesics of learned representations,\" ICLR 2016) used an optimization-based approach, with somewhat mixed success. Can you elaborate on why you think this is a realistic possibility?\n\nLIMITATIONS:\nThe authors provide some discussion of the limitations of their work, but I think a more comprehensive assessment of the possible failings of each step of their approach would enhance the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a CNN-based model fit to recorded responses from salamander retinal ganglion cells to explore the Fisher information matrix $I$ of neural firing. They report that top eigenvectors of $I$ vary markedly with the stimulus being shown and that most discriminative response modes often align with top eigenvectors of $I$. They further argue, on the basis of their analysis, that noise correlations in the retina are likely to be information-limiting because signal and noise are propagated via feedforward mechanisms through the same channels.\n\nThis is an interesting paper that employs information geometry methods to provide some insights into retinal coding. However, the results are somewhat preliminary, and it is unclear how closely these insights are tied to a particular model architecture. The presentation of some aspects of model training is also somewhat confusing.\n\nSTRENGTHS:\n- Good fits to experimental data.\n- Attempts to link models to known physiology. Well-grounded in current neuroscientific theories and questions.\n- While Fisher information has been widely used to investigate neural population coding, the use of a model fit to natural image data to estimate Fisher information over a wider range of stimuli is innovative.\n\nWEAKNESSES:\n- Several modeling choices seem somewhat _ad hoc_, and it's unclear how much the results depend on them. For instance, numbers of hidden layers, types of nonlinearities, and structure of injected noise might possibly play a role, but it's unclear. The authors would likely argue that their good fits to summary statistics of data obviate these questions, since the role of the model is simply to interpolate between measured data for purposes of the manifold analysis, but that's to be shown. \n- Section 3.2 details a somewhat surprising hodgepodge of optimization strategies for different components of the model, suggesting the results depend fairly sensitively on how training is done.\n- Results are intriguing but seem somewhat preliminary. For instance, the analysis of MDS and MDR in Figure 4 is interesting, but I'm missing the bigger picture about what to take away from this.\n\nQUESTIONS:\n- Is there a reason the authors have chosen the non-standard binomial parameterization in Equation 7? Of course it is binomial with sufficient statistic $n$ and natural parameter $k\\log \\frac{r}{1-r}$, which would normally be written $\\log \\frac{p}{1-p}$, but calling $k$ a \"variability parameter\" seems somewhat misleading, as this is still a binomial distribution with a single parameter. It is stated that $N$ is cell-dependent. What about $r$ and $k$?\n\n- I found the exposition of the one-hot layer in Section 3.1 very difficult to follow. I understand that the goal is to mimic retinal organization, but I'm still unclear how this matches up with the math in Eqs. 5 and 6. For instance, I don't know what `input` means in (5). Are $i$ and $j$ indices in the tensor of the last layer, e.g., `W[C(k), i, j]`? And $k$ indexes neurons? Are distinct $(i, j)$ supposed to correspond to distinct RF locations? Similarly, in (6), what do $(p, q)$ correspond to conceptually? It took me a while staring at the formula to understand why a 1-hot $\\mathbf{W}$ is optimal for a given $p$ and $q$, and then the sum just takes care of all possible assignments? And I'm still not clear on how this one-hot assignment for each $k$ prevents output units with different $k$ from using the same $(C, i, j)$. I think a conceptual figure would be a real help here.\n\n- Minor: Line 166: This $w$ is not the same as the one in (6), correct?\n\nLIMITATIONS:\n- As stated in the paper, the calculations are predicated on the assumption that the noise correlations $\\boldsymbol{\\Sigma}$ are locally independent of the stimulus. This may be optimistic in some cases.\n- As noted above, the results are conditioned on a particular choice of model architecture, which may or may not affect the results.\n- Since the model is based on fits to a limited number of ganglion cells in salamander, it may or may not generalize well to the entire retinal population or other species.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This study uses an information geometry approach to studying visual coding in retinal populations.\n- Using local-linear analyses (eigenmode + spectra) at different conditional responses of a CNN + fitted noise model, the authors observe \n- The authors analyze the most sensitive coding directions of the population, as well as the co-alignment of the noise, and find that noise in the retina is information limiting.\n- This is an interesting and conceptually clear/simple paper that adds to our understanding of neural coding in the retina.\n\nUPDATE: Sep 1, 2023. I have read the rebuttal, it addressed my Qs and I already adjusted my confidence accordingly.\n\nSTRENGTHS:\n- I appreciate the clear writing style.\n- The model and eigenvector analyses are conceptually simple to follow.\n- The finding that noise correlations are information limiting is interesting.\n\nWEAKNESSES:\n- I am slightly concerned with the lack of effective null models against which to compare here. Given the size of the dimensionality. Either a positive or negative control. For example, is it conceivable that a different model with different noise eigenbasis, but with the same fit performance?\n\nQUESTIONS:\n- Similar Jacobian eigendecomposition analyses were conducted in Berardino et al. (\"Eigen-distortions of hierarchical representations\", Neurips 2017) to analyze model perceptual alignment with human visual perception. (In this paper they assume isotropic noise in the response domain, but in follow-up analyses in their dissertation, I believe they have more complex noise models.) Importantly, they not only consider the top eigenvector, but also the null space (eigenvectors with eigenvalue=0) as a means to probe model perception. I am wondering if such an analysis, systematically examining the null space of the model, would supplement your results with unique or complementary findings.\n- Is there a difference in noise geometry for natural vs synthetic stimuli? Would these be captured by the model? I think a useful tool to quantify this would be recent work by Duong et al. (\"Dissimilarity metric spaces for stochastic neural networks\"; ICLR 2023), analyzing covariance orientation and scale for different classes of stimuli.\n- Could there be a normative explanation for the noise and coding sensitivity co-alignment?\n- Fig 5A should probably explicitly cite the Averbeck and Pouget review in the caption, saying that it is modified from there. Unless I am mistaken, it does look very similar.\n- Minor: The abstract is a little long.\n\nLIMITATIONS:\nThere was arguably some discussion of limitations but nothing explicit.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new framework to understand stimulus discriminability for models of the visual pathway. They derive a Riemannian metric on the representation manifold and from there, different measures such as most discriminative/sensitive stimulus directions and their counterparts in the response space. The authors apply the method to a CNN model of the retina, fitted on experimental RGC data from tiger salamander. They find that discriminative stimulus directions are often aligned with stochastic modes and that populational codes can help in high firing rate regimes.\n\nSTRENGTHS:\nThe authors present a framework which is quite general and can be applied to different models. This could even help to design better experiments and advance the geometrical understanding of neural representations.\n\nWEAKNESSES:\nWhile the theoretical framework is quite strong, the model presentation and the conclusions are not clear in some places. Especially the drawn conclustions are quite strong, and further investigation (for example (noise) model dependency) should be carried out. \nMore specifically I have the following remarks:\n- Can the authors reiterate on the deduction of eq. 1 and how this defines a Riemannian metric on $\\{ P(y|x)\\}$?\n- The model could be described more clearly.  \n- The optimization of the model is not completley clear to me and some clarification is needed:\n  - how is the Poisson loss defined?\n  - what is the exact (hierarchical) procedure?\n  - maybe a short pseudo code would help here. \n\n- The results seem to be highly dependent on the chosen model, and its noise assumptions. Firstly, it is not clear if they still hold if the model (or its noise) is changed. Secondly, the influence of the different noise assumptions could be tested. Thirdly, it is not clear how $g_i$ are exactly chosen (see optimization).\n\n- While the model seems to be spatiotemporal, the authors purely focus on the analysis of the spatial component. What role does the temporal part play and can it be analyzed in the same framework?\n\nQUESTIONS:\n- How does the model compare to a Bayesian network, in which noise would be (a priori) independent (but systematically optimized) for each processing unit? \n- How does the model perform on other performance measures? Even simple metrics such as reconstruction error, mean firing rate etc. could help to judge the results.\n- How does the model compare to a simple baseline model (for ex. LNP)?\n- Are the MDS robust across different models?\n- Is there a way (or even meaningful) to compare the MDS to the RF of individual cells?\n- How does the MDR (in Fig 4a) for individual cell type channels look like? Are they interesting and can be linked to RF?\n- How do MDRs compare to other dimensionality reduction methods? For example, does it make sense to do PCA (or more fancy methods) on the neural data, push it forward to the stimulus space and compare to MDR?\n\nLIMITATIONS:\nYes, the authors have addressed some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}