Information Geometry of the Retinal Representation Manifold
Authors: Xuehao Ding
Dongsoo Lee
Joshua Brendan Melander
George Sivulka
Surya Ganguli
Stephen Baccus
Venue: NeurIPS 2023 poster (2023)
Keywords: neural coding
theoretical neuroscience
stochastic methods
neural networks

Abstract: The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations of naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created a stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also a variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli to study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing an examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are information-limiting rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less in the population than for single cells and that as a function of firing rate, Fisher information varies less than sensitivity. We conclude that under natural scenes, population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.

Information Geometry of the Retinal Representation Manifold
Xuehao Ding', Dongsoo Lee², Joshua B. Melander', George Sivulka', Surya Ganguli", and Stephen Baccus"
Department of Applied Physics, Stanford University
2,3 Neurosciences Phd Program, Stanford University
DDppartment of Electrical Engineering, Stanford University
'Department of Neurobiology, Stanford University [xhding, dsnl, melander, gsivulka, sganguli, baccus) @ stanford.edu
Abstract
The ability for the brain to discriminate among visual stimuli constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are coormation-limttiii rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less the population than for single cells and that as function of firing rate, Fisher information varies less than sensitivity. population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.
conclude that under natural scenes,
Introduction
Neural populations represent information with their collective activity. In the field of sensory neuroscience, whereas there is much focus on characterizing the sensitivity of neurons to different stimuli, the true function of a sensory system is to discriminate between stimuli. A quantitative description of discriminabilityy in neural population requires not only a precise knowledge of sensitivity to stimuli, but also the structure of noise correlations in the population.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Neural representation manifolds constitute modern approach to studying neural populations in a geometric framework, and recent machine learning literature has seen extensive research on the
geometry of representation manifolds of neural networks trained for artificial tasks [ separate work, Fisher information has been widely utilized in neuroscience for measuring decoding fidelity, particularly in theoretical studies exploring the effect of noise correlations 6 7 [] In this study, we propose a framework for studying the Fisher information geometry of representation manifolds for natural visual scenes in a population of visual neurons, in particular the retina. In addition, for the first time, we apply such a manifold analysis to the neural representation of natural stimuli for real data.
2
3
S
In
The discriminability of biological visual systems has long been studied for low-dimessional and discrete artificial stimuli (9 [0] IT] However, naturalistic visual stimuli in the real world are inherently high-dimensional and discriminability generally heterogeneous across different types of changes in stimulus. Recent advances in convolutional neural networks (CNNs) have enabled the accurate prediction of retinal responses to naturalistic stimuli, as well as ethological computations Nevertheless, previous work has predominantly focused on deterministic trial-average neural codes, which is not adequate for computing discriminability. Thus an encoding model of stochastic retinal representations for natural scenes is required, which constitutes one of the primary contributions of this work.
12 13 [4]
We present here highly accurate model of both sensitivity and stochasticity for the retinal ganglion cell (RGC) population. thus creating the first accurate model of discriminability in a neural population for its natural inputs. From an analysis of the Fisher information geometry of the retinal representation of stimuli, we find that the most discriminable direction for the RGC population varies greatly across stimuli and is correlated with the mean neural response. We further address a long standing question as to the effect of noise correlations on information. We find that the most discriminative mode the representation space often aligns with the most stochastic direction, implying that noise correlations in the retina are detrimental to information coding. Finally, we analyze how discriminability depends on firing rate of individual cells and in the population, and discuss the implications in relation to principles of information maximization.
2 Theory
In this section, we describe basic theory of information geometry 15] and its application to sensory neuroscience. Let æ denote the stimulus vector and y denote the spike count vector whose dimension is equal to the number of neurons encoding the stimulus. Each stimulus induces a conditional probability distribution of sensory neural responses P(y]æ>) and mean firing rates p(x)) == Ely ( ax)) Along any decoding vector a. the sensitivity is defined as ||V (I(()), a) ||. while the stochasticity defined as the standard deviation of (y. a) conditioned on (Figure I In our case in the retina, (()) is modeled by a noiseless CNN and P(y ) is modeled by the stochastic counterpart.
The representation manifold is defined as the manifold of mean neural responses induced by stimuli: (x(z))) We also define a statistical manifold of conditional response distributions {P(y) })) parameterized by stimuli æ, which yield natural coordinate system. To equip the statistical manifold with a Riemannian metric, we consider the Kullback-Leibler divergence between two nearby distributions:
d x dx) = 2 DKr, (Py x)
Ply|= + dx)]
2 / P(y|z) In
d
1 + P(y az)
dy
Σ / P(y/az)) Ol P(y]æ) θ In ((y)
dydaa,dar
Ox;
Ox,
= da I(x:) dæ,
where I (xx) the Fisher information matrix with respect the stimulus, and the KL divergence is linearized up to the second order. The Fisher information metric g lz eạ) [(x);) naturally characterizes the local discriminability of stimuli near æ.
When we assume that neural responses follow multivariate Gaussian distributions y x ~ N (m(x), E(x)) and the covariance matrix is locally independent of the stimulus, <X(a) 0.

---

Stimulus space
Representation manifold
P
Sensory Network
Differentiation Noise
Figure Schematic of basic concepts sensory system. The sensory network maps from the stimulus space to the representation manifold. The differentiation operator maps the stimulus tangent space to the representation tangent space. With noise in the network, each stimulus corresponds to a cloud of points in the representation space. Generally, the sensitivity (the blue and the red arrows) and the stochasticity (the green oval) are both heterogeneous.
the Fisher information metric can be expressed as
I(x) = du((x) dx
du((r)) dæ
(x)
(2)
where du(a))
dar is the Jacobian matrix. Eq. 2 is commonly termed linear Fisher information which bounds the performance of unbiased linear decoders, which may be more biologically plausible than nonlinear decoders [16] 18] The latter assumption is a good approximation in our case because neural network consisting only of ReLU nonlinearities is locally equivalent to effective linear network [9 and the covariance matrix of a linear network is constant matrix. This metric expression interpretable in that discriminability is proportional to sensitivity and inversely proportional to noise. The length of a geodesic connecting two stimuli under this information metric measures the discriminability between them, when represented by the neural population.
Equivalently, the mean response ((x) on the representation manifold can also serve as a coordinate system. Then the metric is given by ge e,) = Σ (x)()) The metric matrix transforms between the stimulus space and the representation space tensor [
We detine here several important quantities for analysis. The eigenvector of I [(x) associated with the largest eigenvalue is the most discriminable direction in the stimulus tangent space, which we term the most discriminable input (MDI) at stimulus æ. The square root of the eigenvalue is the corresponding discriminability 20 The MDI in the stimulus space can be pushed forward to the most discriminative response mode (MDR) the representation tangent space via Jacobian matrix:
MDR X
dp((m) MDI. Similarly, the most sensitive direction in the stimulus (input) tangent space dæ
(MSI) can be obtained the principal eigenvector of du(æ) T du((a) and the associated most sensitive response mode (MSR) obeys MSR
Aa MSI. Additionally, the most stochastic (noisy) response mode (MNR) is defined the principal eigenvector of the noise covariance matrix Ex)) which is no more than principal component analysis. For a ReLU neural network with small stochasticity, the stimulus dependence of the aforementioned tangent space features is solely derived from the stimulus dependence of the effective linear network [19].
du((²)
One practical challenge of the above analysis with natural visual stimuli is the extremely high dimensionality of the stimulus space. Fortunately, we only need to compute the Fisher information matrix in subspace defined by the recorded cell population. First we define the instantancous receptive field for each cell as the direction of the greatest sensitivity to the stimulus at one stimulus point, which is the gradient of that cell's response, (=)( information in the subspace spanned by instantancous receptive fields, as justified by the following theorem.
[13]. Then only need to compute Fisher
dam
Theorem: The top N most discriminable directions are linear combinations of the instantaneous receptive fields, where N is the number of output neurons.
Let IR == span( du)(#)) }} represent the subspace spanned by instantaneous receptive fields. Suppose that the most diseriminable direction 01 is not 7R, then we can decompose V1 = vi + avf, where of is the orthogonal component in I IR- and ov, 2 is the parallel component
Proof:
3

---

Convolutional layer
Convolutional layer
Convolutional layer
One-hot layer
Binomial Noise
Stimulus
spike count
kernels
kermels
keernels
time
C
time
gf)
NT
(C(O, s0>
Figure 2: Model architecture. The first layer consists of eight 15 x 15 spatiotemporal convolutional filters. The second layer consists of eight 11 x 11 filters. The final layer consists of 9 filters, where C' represents the number of cell types. Batch normalization is applied after each convolutional filter. The nonlinearities in the first two layers are ReLUs, while in the last layer, the nonlinearity consists of the SoftPlus function followed by a parametric tanh function. Gaussian white noise is added to the stimulus and the first two convolutional layers, and independent binomial noise applied to the final nonlinearity. The one-hot layer selects the location of the unit in the final convolutional layer to match that of the recorded neuron. Further details are explained in the main text.
in IR, |1lll |,ll|| 1. |a] 1. Then following Eq. 2 have g atim (v1, v1) = =a a stim (uL. vz)
(3)
< < (of. uf),
which contradicts with the fact that V2 is the most discriminable direction. Therefore, 81 is in TR.
Replacing IR with IRN { with the same analysis we can prove that the second most discrim- inable direction U2 is also in IR. Subsequently, the whole theorem can be proven.
3 Methods
The spiking activity of a population of tiger salamander retinal ganglion cells was recorded response to sequence of jittering natural images. We created stochastic encoding model capturing P( ax) with the CNN architecture that included independent noise in the stimulus and in each unit of the CNN. The CNN parameters were trained using backpropagation fit the mean firing rates (xx)) to create deterministic model, and then these parameters were frozen. Noise parameters included one parameter for each layer set the standard deviation of the independent noise, and one parameter for binomial noise in each ganglion cell. These were then optimized by maximizing the likelioeldoood of the empirical data and exploring the parameter space to match a few second-order statisties of the neural responses. Once the parameter optimization was completed, we computed the Fisher information metric with the model numerically for each stimulus and analyzed the most discriminable directions.
3.1 Stochastic model
The model architecture is shown in Figure The model takes spatiotemporal visual stimulus an input and its output is a set of spike counts, one for each neuron each time bin. The deterministic part the model including its hyperparameters and training method is adopted from Ref. (12 [3] 14 which is a three-layer CNN chosen after an extensive architecture search. This CNN is the SOTA retinal encoding model for natural scenes in that (1) the prediction accuracy of RGC mean responses outperforms other models; (2) the activity of internal units is correlated with retinal interneuron responses recorded separately and used training; and (3) the model trained only on natural scenes can reproduce wide range of phenomena induced by artificial stimuli. Thus, the deterministic part our model has a clear correspondence with the real retina in that its circuitry is mechanistically interpretable and it captures many retinal computations. Every convolutional filter in our model was implemented by linearly stacking a sequence of x 3 small filters, which outperforms the traditional method 211. A parametric tanh nonlinearity was attached to the last convolutional layer for the

---

purpose of enforcing the refractory period constraint:
f(z; Tmaxx 6) Tmaz
[tanh(a(z - b)) + +],
(4)
2
where b are cell-dependent model parameters, and mmaz represents the cell-dependent maximal firing rate.
The essential module that enables us to study retinal population coding is the one-hot layer. Model parameters are fit with the one-hot layer, which bijectively maps a few output units of the final convolutional layer to recorded neurons by the end of the training. The one-hot layer is then removed after parameter optimization
that we can analyze the representation of the full population, typically consisting of thousands of units. The rationale behind this method is the mosaic organization of each retinal cell type, in which all ganglion cells of given type tile the retina with their dendrites [T]] Formally, the one-hot layer is defined as
out(k:
= Σ
w
i,),, VA
ij
Σ
u k
1, A
(5)
w 2 0, A j
where the input is the output of the last convolutional layer, k is the index of recorded neurons, j.j (and p. q below) location indices, w is the linear combination weight vector, and C(k) is the couron-to-channd map. Each w will converge one-hot array after model training with our one-hot loss function, which is adapted from the semantic loss in Ref. (2
Lome hot (w** x
Σ
II 1
+ 6pg.1 - 1)1,
PQ
(6)
(1 0
i,
=
8 pq.is
otherwise
and which reflects the locations of recorded neurons. Each output channel represents ganglion cell type and C(k) was determined using hierarchical clustering of output channels (see the Supplementary Material for details).
Gaussian white noise was added to the stimulus to simulate stochasticity in photoreceptors, and was also added to pre-threshold signals in the first two layers to simulate stochasticity in bipolar cells and amacrine cells. We found that Gaussian noise outperforms Poisson noise these layers, which presumably be attributed the noise generation mechanism in the retina and the central limit theorem. The stochasticity of ganglion cells was modeled as a parametric binomial probability mass function:
T(K N +1) (n) & r(k: (N n) + 1) I(k nelk-n- neenn+11
pon (1 r)k-(N--), n 0, N (7)
where n denotes the spike count time bin, the rate parameter having a one-to-one corre- spondence with the mean firing rate k is the variability parameter to be optimized, and N is the cell-dependent upper bound of spike counts in one time bin reflecting the refractory period of that cell, which one can directly determine from data. In practice, for given i and N we can interpolate the r 1 mapping and use it determine the rate parameter producing the desired mean firing rate, i.e., p(r) = &p (n). Although we have explored variety of probability mass functions to fit the sub-Poisson statisties of retinal ganglion cells [23 24), the binomial noise Eq. performs the best among them (see the Supplementary Material) and also is the most natural choice considering the refractory period constraint. We stress that the binomial noise on ganglion cells is independent noise, and thus will not qualitatively affect the geometry and the functional implication of noise correlations.
3.2 Optimization
Since the second order statistics are functions of the entire dataset and cannot be decomposed into terms depending the individual stimulus, optimized deterministic parameters and stochastic
5

---

parameters separately. CNN parameters and one-hot parameters {w*} were optimized together with the standard Poisson loss function and Lone- fit mean firing rates smoothed using a 10 ms Gaussian filter. The optimization was performed using ADAM (25 via Pytorch [26] on NVIDIA TITAN Xp. GeForce GTX TITAN X, GeForee RTX 3090, TITAN RTX, TITAN V, and GeForce RTX 2080 Ti GPUs The network was regularized with an L.2 weight penalty at each layer and an LI penalty on the model output. Then tanh parameters of the final nonlinearity were fitted with the least square regression. Then we optimized stochastic parameters while freezing all deterministic parameters. Variability parameters k were optimized by maximizing the log-likelihood of recorded spike counts based empirical firing rates. Standard deviations of Gaussian noise {9u, 91 92 | in each layer were optimized through grid searches in order to best fit empirical noise correlations and stimulus correlations by minimizing the smoothed and weighted average of mean squared errors.
3.3 Computation of Fisher information metric
For each stimulus, the noise covariance matrix E was estimated by repeatedly simulating the stochastic model thousands of times. We then computed the eigendecomposition of Σ obtaining local information of the representation manifold: Σ = - A. We applied the Gram-Schmid orthogonalization to instantaneous receptive fields to obtain basis of IR, which is subspace of the stimulus tangent space. Then following Eq. 2] the Fisher information matrix with respect to the stimulus can be computed as
I -= (u(2), x,1 (u(2), vz)
dz
dz
where z is the stimulus coordinate under the IR basis. In practice, we summed only over hundreds of the most stochastic modes that can explain 85% to 90% of the variance, as the sensitivity and stochasticity of the other modes are both close to zero indicating that those modes orthogonal to the tangent space of the representation manifold. The gradient with respect to the stimulus was computed using the noiseless model.
4
Results
4.1
Model evaluation
Early stochastic encoding models of the retina were primarily developed for artificial stimuli 27 28 [2]] and their results cannot be directly generalized to natural scenes due to the stimulus- dependent nature of noise in the retina. Here our fitted model was evaluated by comparing with neural data various commonly used second-order statistics including correlation measures and single-cell variability measures (FigureB). Surprisingly, the model prediction matches the neural data remarkably well despite that the model only has a few noise parameters. This result suggests that because noise in the retina propagates through the same network pathways the signal, they are both captured by the CNN structure. This indicates a strikingly simple origin for noise correlations between all ganglion cell pairs: independent noise that starts in individual upstream cells, and then propagates through the same network that creates mean stimulus sensitivity. Statistics for preparations with longer test sequences tend to be more reliable and consequently our model performs better on these. For the first order statistics (mean firing rates), the Pearson correlation coefficient between model predictions and data ranges from 70% to 80%, which is similar to Ref. 222 13. Such an accurate model of both sensitivity and stochasticity guarantees that the computation of Fisher information matrix is reliable (Eq. 2
4.2
The most discriminable directions
Some examples of the most discriminable stimulus direction (MDI) and the most discriminative response mode (MDR) are shown in Figure (see the Supplementary Material for more examples). Often the top eigenvalues were similar, and so to avoid over-interpreting the importance of individual eigenmodes, we found the sparsest direction (with the least LI norm) in the space spanned by the top 5 MDIs for the purpose of visualization. Only spatial components are presented here as we found that temporal components for different stimuli are both very similar to each other and to those of instantancous receptive fields [13). We found that the MDI is strongly stimulus dependent, and is

---

(a)
Total correlation
Stimulus correlation
Noise correlation
0.6
0.45
0.4
Model
0.30
0.2
0.15
0.0
0.00
0.0
0.2
0.4
0.6
0.00
0.15
0.30
0.45
0.00
0.04
0.08
Data
Data
Data
(b)
Fano factor trial-to-trial correlation
Model
0.4
0.2
0.6
0.3
2
10 101312231140 141551111111111
29 3333433
cells
Figure The model captures a variety of second-order statistics of neural responses (see the Supplementary Material for definitions). (a) Comparison of total pairwise correlation, stimulus correlation, and noise correlation between neural data and model prediction. Black lines represent perfect agreement. Each circle corresponds a pair of neurons an experimental preparation coded by its color and the circle radius is proportional to the square root of the total length of the test set. (b) Comparison of single-cell variability measures including the Fano factor and trial-to-trial correlation between neural data and model prediction. Neuron indices share the same color code with (a).
typically localized in a region where the sensitivity is high, an effect that arises from the locality of convolutional connections. The MDI tends to appear around the central region (red box Figure 4a because pixels within this area affect more output units than pixels near the edges corners.
One general observation that the MDI varies greatly across stimuli. From mathematical point of view, this is to say that certain features of the tangent space depend on the base point. With respect to neural function, this variability potentially indicates the presence of adaptation the stimulus. We found that the most discriminable direction is related to the base point through the representation space such that the MDR and the mean response are correlated (Figure 4b). This correlation can be attributed to the fact that neurons with intermediate firing rates exhibit the highest single-cell sensitivity and discriminability (Figure 6a As the MDI sometimes captures the most salient feature of the stimulus (the first row of Figure 4a). although this is not always the case (the second row of Figure 4ap. Similar stimuli and mean responses likely produce distinct MDIs especially when their mean response pattern has multiple spatial components that can each yield localized MDR (the third and fourth rows of Figure 4a). Moreover, the correlation between MDR and mean response provides mechanism for higher brain regions estimate the most informative future response changes of retinal ganglion cells based on the current response and thereby adjust the corresponding sensitivity, which could have benefits for the property of predictive coding [30]. Additionally, we found that
7

---

(a)
(b)
Stimulus
MDI
Response
MDR
Stimulus + MDI
Cosine similarity between top MDRs mean response
ordered
Erequency
0.05
0.00
5.0
0.3
Cosine
(c)
Figure 4: Most discriminable directions. (a) Examples of the most discriminable directions for different stimuli in the set. The first column shows representative frame of the spatiotemporal stimulus. The second column shows the spatial component of the spatiotemporal MDI obtained from the singular value decomposition. Red boxes correspond the unit array the final layer. The third column shows the mean response averaged over cell-type channels. The fourth column shows the absolute values of MDR averaged over channels. The last column shows the superposition of the MDI and stimulus. (b) Histogram of the cosine similarity between the mean response vector and its projection onto the space spanned by the top 5 MDRs. Similarities between shuffled pairs are also shown for baseline comparison. (c) One example of the stimulus (left panel, the central 18 x 18 array shown) and the corresponding MDR for different cell type channels.
different parts the stimulus generates the MDR for different cell types (Figure 4c). indicating that different cell types signal different stimulus regions in the image conveying the most information.
4.3 Noise correlations in the retina limit information coding
The role of noise correlations in population coding has long been debated. Multiple scenarios have been advanced where noise correlations are detrimental information coding [3] However, others suggest that noise correlations under certain circumstances may not limit information coding and can even be beneficial [32 6 9
More specifically, has been proposed that in response space, noise correlations decrease information when the differences in the signal are aligned with the noise direction, and increase information when they are orthogonal 33
(Figure 55a
Here we tested these two hypotheses for the retina by computing cosine similarities between MDRs and MNRs. mode and orthogonal to modes with small variability (Figure 5b). implying that noise correlations in the retina information-limiting correlations. We also performed conclusion by analyzing trial-shuffled responses, which have independent noise. We found that removing noise correlations in this way created responses that were significantly more discriminative than the original model outputs with correlated noise for every population size (Figure 5c). To understand the mechanism of this reduction in information, we analyzed the eigenspectrum of the Fisher information matrix (Figure 5d]. Both the sensitivity and the stochasticity increase with the discriminability MDR, and their ratio reaches maximum the noisiest mode. Therefore, we propose that the fundamental reason for such information corimaatioo-iiiiingg correlation is that noise propagates through the same network as the signal, causing the stochasticity ellipsoid to align closely with, and be rounder than, the sensitivity ellipsoid. One trivial and extreme example of this arrangement is when all noise originates the level of the stimulus, e.g. from photoreceptors, then the resulting noise correlations will obviously be detrimental.
that the most discriminative mode quite aligned with the most stochastic
direct of this

---

(b)
Cosine similarity between MDR and MNRR
1st NDR
(a)
0
- 5th MDR
Neuron
Neuron
Neuron
0:15
0
16 MNR index
(c)
(d)
Spectrum of Fisher information matrix
2.0
correlated noise
independent noise
Discriminabiitty
1.5
Stochasticity
0.8
1.0
0.5
100
200
300
120
180
cells
index
Figure 5: Noise correlations in the retina are detrimental to information coding, (a) Compared to independent noise (left), noise correlations can either increase information (middle) decrease information (right) 333. (b) Cosine similarities between the top 5 MDRs and the top 30 MNRs averaged over stimuul.. (c) Discriminability MDI for different numbers of output units. The plot for trial-shuffled responses with independent noise is shown for comparison. Output units are randomly selected from the central 500μm x 500pm region [34] (d) Sensitivity and stochasticity of the top 200 MDRs averaged over stimuli.
Firing rate dependency
The firing rate dependencies of stochasticity, sensitivity, and discriminability for individual neurons and the whole population shown in Figure For individual neurons, both the sensitivity and the stochasticity peak around intermediate firing rates and decay high firing rates (a consequence of the refractory period). Thus, the single-cell discriminability also saturates around the intermediate firing rate. In contrast, population coding does not exhibit such a decay at high rates, implying that neurons can encode information complementarily, which is benefit of population coding. In other words, even when fraction of neurons get saturated by the stimulus and become uninformative at high firing rates, other neurons can still provide discriminative responses to stimulus changes. Although the population sensitivity and stochasticity significantly increase with the mean firing rate, the firing rate dependency of the population discriminability remains relatively flat, which potentially beneficial for information transmission assuming that stimuli are uniformly distributed on the natural scene manifold 335| 36).
Discussion
In summary, we established a novel information -eommerri framework to understand the high di- mensional representation manifold of the retinal population for natural scenes, one that can also be

---

(8)
(b)
Individual neurons
Population
mean
Stochaticity
Stochanticity
0.0
mest
0.75
Sensitvvity
Sensitivity
0.50
0.25
mean
1.2
Discriminability
Discriminability
1.5
0.8
0.4
0.5
0.0
0.0
160 Firing rate (H2)
80
Mean firing rate (H2)
Figure 6: Firing rate dependency. (a) Log-scale 2D histograms showing the firing rate dependencies of stochasticity, sensitivity, and discriminability for individual neurons whose definitions are given in the theory section. Mean values were plotted with thick black lines. (b) Scatter plots showing the population-averagel firing rate dependencies of stochasticity, sensitivity, and discriminability for first MDR. The linear regression of the populational discriminability is shown.
generalized other neural systems. However, there still exist remaining questions and further work that can be pursued within this framework.
First, our analysis primarily focused on examining the local metric tensor, which characterizes the effects infinitesimal stimulus changes. would be interesting to extend the analysis to finite stimulus changes explore more global geometry of the representation manifold. For example, one exciting direction to find geodesics between two stimuli under the information metric and observe how one stimulus transitions into another with the least amount of discriminable changes for the retinal population
Second, we found that noise correlations in the retina are information-limittin correlations. This conclusion is surprisingly robust against many hyperparameter selections, even when second order statistics are not fitted well. Therefore, remains unclear whether this is general result for all feedforward networks with independent noise applied to their inputs and intermediate layers. If this is indeed the case, noise correlations in other brain areas that do not limit information coding must arise from different mechanisms.
Finally, the Fisher information metric in this work was computed in the full stimulus space. However, most points the stimulus space represent meaningless noise images that no animal ever encountered. Therefore, an important avenue for future research would involve incorporating the structure of natural stimuli and computing the metric with stimuli constrained to natural-scene manifold.
10

---

Acknowledgements
This work was supported by grants from the NEI, R01EY022933, RO1EY025087 and P30EY026877
(SAB).
References
[1] Michael Hauser and Asok Ray. Principles of riemannian geometry in neural networks. Advances in neural information processing systems, 30, 2017.
[2] Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds deep neural networks. Nature communications, 11(1):746, 2020.
[3] Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43))2220080011 2022.
[4] Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315-323, 2018.
[5] Binxu Wang and Carlos R Ponce. A geometric analysis deep generative image models and its applications. In International Conference Learning Representations, 2021.
[6] Rubén Moreno-Bote, Jeffrey Beck, Ingmar Kanitscheider, Xaq Pitkow, Peter Latham, and Alexandre Pouget. Information-limitngg correlations. Nature neuroscience, 17(10)1410-14117 2014.
[7] Larry F Abbott and Peter Dayan. The effect of correlated variability on the accuracy of population code. Neural computation, 11(1):91-10,, 1999.
[8] Bruno B Averbeck and Dacycol Lec. Effects of noise correlations on information encoding and decoding. Journal of newrophysiology, 95(6)33633-3644, 2006.
[9] Oleg Rumyantsev, Jérôme A Lecoq, Oscar Hernande7, Yanping Zhang, Joan Savall, Radoslaw Chrapkiewicz, Jane Li, Hongkui Zeng, Surya Ganguli, and Mark J Schnitzer. Fundamental
bounds on the fidelity of sensory cortical coding. Nature, 180078011:100-105, 2020.
[10] Rava Azeredo da Silveira and Fred Ricke. The geometry of information coding in correlated neural populations. Annual Review of Neuroscience, 44:403-424, 2021.
[11] David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, 14(10):1317-132, 2011.
[12] Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models the retinal response to natural scenes. Advances in neural information processing systems, 29, 2016.
[13] Niru Maheswaranathan, Lane T McIntosh, Hidenori Tanaka, Satchel Grant, David B Kastner, Joshua В Melander, Aran Nayebi, Luke E Brezovec, Julia H Wang, Surya Ganguli, et al. Interpreting the retinal neural code for natural scenes: From computations neurons. Neuron, 111(17):742-2755. 2023.
[14] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baceus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
[15] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
[16] Jeffrey Beck, Vikranth R Bejjanki, and Alexandre Pouget. Insights from simple expression for linear fisher information recurrently connected population of spiking neurons. Newral computation, 23(6): 1584-1 2011.

---

[17] Ingmar Kanitscheider, Ruben Coen-Cagli, Adam Kohn, and Alexandre Pouget. Measuring fisher information accurately in correlated neural populations. PLoS computational biology. 1 116))=1004218, 2015.
[18] Adam Kohn, Ruben Coen- Cagli, Ingmar Kanitscheider, and Alexandre Pouget. Correlations and neuronal population information. Annual review of neuroscience, 39:237-23 226,, 2016.
[19] Haozhe Shan and Haim Sompolinsky. Minimum perturbation theory of deep perceptual learning. Physical Review E, 106(6):06440,, 2022.
[20] Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Eigen-distortic of hierarchical representations. Advances in neural information processing systems, 30, 2017.
[21] Xuehao Ding, Dongsoo Lee, Satchel Grant, Heike Stein, Lane Mcintosh, Niru Mah- eswaranathan, and Stephen Baccus, A mechanistically interpretable model of the retinal neural code for natural scenes with multiscale adaptive dynamics. In 2021 55th Asilomar Conference Signals, Systems, and Computers, pages 287-291. IEFE, 2021.
[22] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pages 5502-5511, PMLR, 2018.
[23] Michael J Berry, David K Warland, and Markus Meister. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences, (110):5511154111 1997.
[24] Xaq Pitkow and Markus Meister. Decorrelation and efficient coding by retinal ganglion cells. Nature neuroscience, 15(4):628-63) 2012.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 2014.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019.
[27] EJ Chichilnisky. A simple white noise analysis of neuronal light responses. Network: computa- tion in neural systems, 12(2):199, 2001.
[28] Justin Keat, Pamela Reinagel, R Clay Reid, and Markus Meister. Predicting every spike: a model for the responses of visual neurons. Neuron, 30(3):803-817,, 2001.
[29] Jonathan W Pillow, Liam Paninski, Valerie J Uzzell, Eero P Simoncelli, and EJ Chichilnisky. Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. Journal of Neuroscience, 25(47): 1003-1 1013, 2005.
[30] Laurence Aitchison and Máté Lengyel. With or without you: predictive coding and bayesian inference in the brain. Current opinion in neurobiology, 46:219-227, 2017.
[31] Ingmar Kanitscheider, Ruben Coen-Cagli, and Alexandre Pouget. Origin of information-limitin noise correlations. Proceedings of the National Academy of Sciences, -1953-3966673--6999 2015.
[32] Jon Cafaro and Fred Rieke. Noise correlations improve response fidelity and stimulus encoding. Nature, 468(7326)9944-96 9677777 2010.
[33] Bruno B Averbeck, Peter E Latham, and Alexandre Pouget. Neural correlations, population coding and computation. Nature reviews neuroscience, 7(5):358-366, 2006.
[34] Elad Schneidman, Michael Berry, Ronen Segev, and William Bialek. Weak pairwise correla- tions imply strongly correlated network states in neural population. Nature, 44007087):1007- 1012, 2006.
[35] Simon Laughlin. A simple coding procedure enhances a neuron's information capacity. Zeitschrift für Naturforschung c. 3699-100991--912.. 1981.
12

---

[36] Xue-Xin Xin Wei and Alan A Stocker. Mutual information, fisher information, and efficient coding. Neural computation, 28(2):00--326, 2016.
[37] Olivier J Hénaff and Eero Simoncelli. Geodesics of learned representations. In 4th Interna- lional Conference on Learning Representations, ICLR 2016, 2016.
13


Reviews:
[{"review_id": "a2JMBAQEoc", "reviewer": "Reviewer_GBMw", "summary": "The authors present a new framework to understand stimulus discriminability for models of the visual pathway. They derive a Riemannian metric on the representation manifold and from there, different measures such as most discriminative/sensitive stimulus directions and their counterparts in the response space. The authors apply the method to a CNN model of the retina, fitted on experimental RGC data from tiger salamander. They find that discriminative stimulus directions are often aligned with stochastic modes and that populational codes can help in high firing rate regimes. ", "questions": "- How does the model compare to a Bayesian network, in which noise would be (a priori) independent (but systematically optimized) for each processing unit? \n- How does the model perform on other performance measures? Even simple metrics such as reconstruction error, mean firing rate etc. could help to judge the results.\n- How does the model compare to a simple baseline model (for ex. LNP)?\n- Are the MDS robust across different models?\n- Is there a way (or even meaningful) to compare the MDS to the RF of individual cells?\n- How does the MDR (in Fig 4a) for individual cell type channels look like? Are they interesting and can be linked to RF?\n- How do MDRs compare to other dimensionality reduction methods? For example, does it make sense to do PCA (or more fancy methods) on the neural data, push it forward to the stimulus space and compare to MDR?", "limitations": "Yes, the authors have addressed some limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors present a framework which is quite general and can be applied to different models. This could even help to design better experiments and advance the geometrical understanding of neural representations. ", "weaknesses": "While the theoretical framework is quite strong, the model presentation and the conclusions are not clear in some places. Especially the drawn conclustions are quite strong, and further investigation (for example (noise) model dependency) should be carried out. \nMore specifically I have the following remarks:\n- Can the authors reiterate on the deduction of eq. 1 and how this defines a Riemannian metric on $\\{ P(y|x)\\}$?\n- The model could be described more clearly.  \n- The optimization of the model is not completley clear to me and some clarification is needed:\n  - how is the Poisson loss defined?\n  - what is the exact (hierarchical) procedure?\n  - maybe a short pseudo code would help here. \n\n- The results seem to be highly dependent on the chosen model, and its noise assumptions. Firstly, it is not clear if they still hold if the model (or its noise) is changed. Secondly, the influence of the different noise assumptions could be tested. Thirdly, it is not clear how $g_i$ are exactly chosen (see optimization).\n\n- While the model seems to be spatiotemporal, the authors purely focus on the analysis of the spatial component. What role does the temporal part play and can it be analyzed in the same framework?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and helpful suggestions. \n\nWeaknesses:\n1. The LHS of Eq. 1 is equal to $-2\\int P(y|x)\\ln[1+\\frac{dP(y|x)}{P(y|x)}]dy$. We can add this line to help understand the derivation. The remaining step is just to apply a high-dimensional Taylor expansion to the logarithm function. Note that the first order term will vanish so only the second order term will be left. This is common knowledge in the information geometry (see citation [14)].\n2. The model architecture is mainly described in the caption of Fig. 2 which is adopted from the architecture in citations [11, 12, 13] and more emphasis in the paper is given to the stochastic part. We can modify this part to make the description more clear and also emphasize more that this work follows from earlier studies.  Importantly, the model is a highly successful model that (1) underwent extensive architecture search to fit the mean response of salamander retina to natural movies; (2) it generalizes to correctly predict > a decade's worth of experiments on artificial stimuli; and (3) its hidden units and computations match that of the biological retina. We show remarkably that this *same* model could also correctly predict stochastic second order correlations. \n3. Poisson loss is defined as loss(input,target)=input−target∗log(input)+log(target!). We used the regular Poisson loss in PyTorch. The detailed procedure of hierarchical clustering is summarized in the supplementary material. We computed the cosine similarity matrix across channels as the affinity matrix and applied the standard procedure with the function implemented in the scikit-learn package. We can define the Poisson loss and improve the description of hierarchical clustering in the final paper.\n4. For the first two points, please see the overall rebuttal above. For the third point, we grid-searched $g_i$ to optimize the mean squared errors of noise correlations and stimulus correlations (weighted average), and more details of this can be provided. \n5. It is true that the MDS is spatiotemporal. Analyses in Fig. 4b, Fig. 5, Fig. 6 are conducted with the full spatiotemporal MDS vector. We also computed the temporal components of MDSs but found that the temporal components for different stimuli are very similar, and are also very similar to the temporal components of instantaneous receptive fields. We did not find interesting results in terms of pure temporal components and therefore did not present them in figures considering the page limit, but can briefly describe these temporal components.\n\nQuestions:\n1. A Bayesian network is a probabilistic graph model, whereas our model is a mechanistically interpretable neural network whose internal units have correspondence with bipolar and amacrine cells as shown in citation [12] (Maheswaranathan et al., 2023). Rather than taking a different approach that would be disconnected with the biological implementation, in this manner it is important that the full sensitivity and stochasticity can be captured by taking a deterministic model and adding appropriate noise. The linearity and non-linearity in the model correspond to synaptic weighting / current integration and firing / vesicle release, respectively. We can emphasize more about the interpretability and mechanistic correspondence of our model in the final paper.\n2. We computed the Pearson correlation between model firing rates and recorded firing rates, which ranges from 70% to 80%, which is the state-of-the-art performance for natural scenes. This is similar to the firing rate performance reported in citations [11, 12] because we are using a similar model architecture, which we can report it in the final paper.\n3. The Linear-Nonlinear model and Generalized Linear Model (GLM), performs poorly under natural scenes compared to a 3-layer CNN model (Maheswaranathan, Niru, et al. Neuron 2023). Although GLMs have been shown to capture noise correlations under white noise stimuli (Pillow et al., 2008), the Linear-Nonlinear-Poisson model by definition cannot capture noise correlations between cells because noise is independent, which was tested explicitly in Meytlis et al. 2012. Such noise correlations are known to be important for decoding of stimuli under white noise and natural scenes (Ruda, Kiersten, et al. 2020).\n4. Our experiments used four animals and we trained one model for each preparation. As stated more fully in the general rebuttal, comparing the results of these models, we believe that our results and conclusions are robust. \n5. Since the receptive field is computed by averaging over stimuli, we think it is more meaningful to compare the MDS with instantaneous receptive fields (gradient of neural response with respect to stimulus), which is dependent on stimulus. According to the theorem we proved in the paper, the MDS lies in the subspace spanned by the instantaneous receptive fields of the output neurons. Indeed they often look similar, and sometimes the MDS looks like a superposition of two or three instantaneous receptive fields.\n6. One example of MDR across different cell type channels as well as the stimulus is shown in Fig. A1 in the attached pdf. One can see that different parts of the stimulus generates the MDR for different cell types, indicating that different cell types signal different stimulus regions in the image as conveying the most information. We will add this figure to the paper and discuss briefly the cell-type dependency.\n7. The most noisy response directions (MNR) are simply found by PCA with the result shown in Fig. 5b that MNR and MDR are correlated. We have not tried more sophisticated dimensionality reduction methods, which would be appropriate in the in the future to better understand the full structure of the neural manifold, but the relationship between MNR and MDR is highly likely to hold. We can emphasize the equivalence between PCA and the identification of the MNR in the final paper to help understand our method.\n"}, {"review_id": "azEtRROXT5", "reviewer": "Reviewer_d8Ka", "summary": "- This study uses an information geometry approach to studying visual coding in retinal populations.\n- Using local-linear analyses (eigenmode + spectra) at different conditional responses of a CNN + fitted noise model, the authors observe \n- The authors analyze the most sensitive coding directions of the population, as well as the co-alignment of the noise, and find that noise in the retina is information limiting.\n- This is an interesting and conceptually clear/simple paper that adds to our understanding of neural coding in the retina.\n\nUPDATE: Sep 1, 2023. I have read the rebuttal, it addressed my Qs and I already adjusted my confidence accordingly.", "questions": "- Similar Jacobian eigendecomposition analyses were conducted in Berardino et al. (\"Eigen-distortions of hierarchical representations\", Neurips 2017) to analyze model perceptual alignment with human visual perception. (In this paper they assume isotropic noise in the response domain, but in follow-up analyses in their dissertation, I believe they have more complex noise models.) Importantly, they not only consider the top eigenvector, but also the null space (eigenvectors with eigenvalue=0) as a means to probe model perception. I am wondering if such an analysis, systematically examining the null space of the model, would supplement your results with unique or complementary findings.\n- Is there a difference in noise geometry for natural vs synthetic stimuli? Would these be captured by the model? I think a useful tool to quantify this would be recent work by Duong et al. (\"Dissimilarity metric spaces for stochastic neural networks\"; ICLR 2023), analyzing covariance orientation and scale for different classes of stimuli.\n- Could there be a normative explanation for the noise and coding sensitivity co-alignment?\n- Fig 5A should probably explicitly cite the Averbeck and Pouget review in the caption, saying that it is modified from there. Unless I am mistaken, it does look very similar.\n- Minor: The abstract is a little long.", "limitations": "There was arguably some discussion of limitations but nothing explicit.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- I appreciate the clear writing style.\n- The model and eigenvector analyses are conceptually simple to follow.\n- The finding that noise correlations are information limiting is interesting.", "weaknesses": "- I am slightly concerned with the lack of effective null models against which to compare here. Given the size of the dimensionality. Either a positive or negative control. For example, is it conceivable that a different model with different noise eigenbasis, but with the same fit performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and helpful suggestions.\n\nWeaknesses:\nFor the main result about the effect of noise correlation in the retina, we had already performed a null / control analysis that could be added to the final paper. As shown in Fig. A3 in the attached pdf, we shuffled trials of the model response to set all noise correlations to zero, thus creating a null model with independent noise and compared it with the original model. We found that the neural response with independent noise has significantly higher discriminability than the response with correlated noise, which further confirms our finding of how noise correlations reduce information under natural scenes.  We also note that our neural architecture is building on previous work that accounted for the mean deterministic but not variable stochastic responses of the salamander retina to natural movies (see our cited [11,12,13]).  There alternative architectures were considered, including the Generalized Linear Model (GLM), which was shown to perform poorly in modeling the mean response compared to the 3-layer CNN model (Maheswaranathan, Niru, et al. Neuron 2023).  Therefore we are extending a SOTA architecture and extending its applicability from mean responses to stochastic second order correlations in responses to natural movies, which in and of itself is a major contribution that has never been achieved before in the retina using a feedforward network whose internal hidden units and  computations match that of the biological retina.  \n\nQuestions:\n1. Thank you for alerting us to this very relevant citation. We have also computed the least discriminable stimulus direction, and compared to the MDS which is more localized, they look more scattered and indeed difficult to discriminate for human eyes (Fig. A2 in the pdf). Since the least discriminable directions are less relevant to our main conclusions, they are not presented in the paper but we could add them in revision.\n2. We believe that there is a large difference in the geometry for natural and synthetic stimuli. We also trained a model using checkerboard white noise stimuli and the resulting noise parameters and noise covariance geometry looks very different from that for natural scene, implying that noise in the retina highly depends on the overall stimulus statistics. Since the scope of our paper focuses on the much more ethologically important regime of natural scenes, we didn't present our results for white noise, but we can briefly describe describe such stimulus dependency.\n3. Suppose that all Gaussian noise arises at the level of the stimulus, then obviously the noise and the signal directions will align. Deviation between noise and signal directions can only occur when the dominant noise source arises deeper in the network. In our case, a large fraction of noise source often arises earlier, implying that signal and noise will still be more aligned. Although we do not have a normative explanation as to whether the allocation of noise earlier in the network arises due to some advantage related to information or that it occurs due to mechanistic constraints (e.g.  photoreceptors having noise in the biochemical transduction cascade), we will improve our explanation of such co-alignment in the final paper. And as stated in the discussion section, one interesting future direction is to better understand under what circumstances such alignment would happen.\n4. Yes, Fig. 5A  is modified from the Averbeck and Pouget review, cited in the text, and we will also add the citation in the caption in the final version of the paper.\n"}, {"review_id": "KKnFSzjuo9", "reviewer": "Reviewer_SiWR", "summary": "This paper uses a CNN-based model fit to recorded responses from salamander retinal ganglion cells to explore the Fisher information matrix $I$ of neural firing. They report that top eigenvectors of $I$ vary markedly with the stimulus being shown and that most discriminative response modes often align with top eigenvectors of $I$. They further argue, on the basis of their analysis, that noise correlations in the retina are likely to be information-limiting because signal and noise are propagated via feedforward mechanisms through the same channels.\n\nThis is an interesting paper that employs information geometry methods to provide some insights into retinal coding. However, the results are somewhat preliminary, and it is unclear how closely these insights are tied to a particular model architecture. The presentation of some aspects of model training is also somewhat confusing.", "questions": "- Is there a reason the authors have chosen the non-standard binomial parameterization in Equation 7? Of course it is binomial with sufficient statistic $n$ and natural parameter $k\\log \\frac{r}{1-r}$, which would normally be written $\\log \\frac{p}{1-p}$, but calling $k$ a \"variability parameter\" seems somewhat misleading, as this is still a binomial distribution with a single parameter. It is stated that $N$ is cell-dependent. What about $r$ and $k$?\n\n- I found the exposition of the one-hot layer in Section 3.1 very difficult to follow. I understand that the goal is to mimic retinal organization, but I'm still unclear how this matches up with the math in Eqs. 5 and 6. For instance, I don't know what `input` means in (5). Are $i$ and $j$ indices in the tensor of the last layer, e.g., `W[C(k), i, j]`? And $k$ indexes neurons? Are distinct $(i, j)$ supposed to correspond to distinct RF locations? Similarly, in (6), what do $(p, q)$ correspond to conceptually? It took me a while staring at the formula to understand why a 1-hot $\\mathbf{W}$ is optimal for a given $p$ and $q$, and then the sum just takes care of all possible assignments? And I'm still not clear on how this one-hot assignment for each $k$ prevents output units with different $k$ from using the same $(C, i, j)$. I think a conceptual figure would be a real help here.\n\n- Minor: Line 166: This $w$ is not the same as the one in (6), correct?", "limitations": "- As stated in the paper, the calculations are predicated on the assumption that the noise correlations $\\boldsymbol{\\Sigma}$ are locally independent of the stimulus. This may be optimistic in some cases.\n- As noted above, the results are conditioned on a particular choice of model architecture, which may or may not affect the results.\n- Since the model is based on fits to a limited number of ganglion cells in salamander, it may or may not generalize well to the entire retinal population or other species.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Good fits to experimental data.\n- Attempts to link models to known physiology. Well-grounded in current neuroscientific theories and questions.\n- While Fisher information has been widely used to investigate neural population coding, the use of a model fit to natural image data to estimate Fisher information over a wider range of stimuli is innovative.", "weaknesses": "- Several modeling choices seem somewhat _ad hoc_, and it's unclear how much the results depend on them. For instance, numbers of hidden layers, types of nonlinearities, and structure of injected noise might possibly play a role, but it's unclear. The authors would likely argue that their good fits to summary statistics of data obviate these questions, since the role of the model is simply to interpolate between measured data for purposes of the manifold analysis, but that's to be shown. \n- Section 3.2 details a somewhat surprising hodgepodge of optimization strategies for different components of the model, suggesting the results depend fairly sensitively on how training is done.\n- Results are intriguing but seem somewhat preliminary. For instance, the analysis of MDS and MDR in Figure 4 is interesting, but I'm missing the bigger picture about what to take away from this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and helpful critiques.\n\nWeaknesses:\n\n1, 2.\nPlease see the overall author rebuttal as to choices of model architecture and optimization. Most importantly, we note that our neural architecture is building on previous work that accounted for the mean deterministic but not variable stochastic responses of the salamander retina to natural movies (see our cited [11,12,13]). In that work an extensive architecture search was performed which varied number of layers, number of channels per layer, etc... to find the simplest model that can capture mean ganglion cell responses.  We are building on this extensive architecture search by taking the SOTA model and showing, remarkably, that without any further modification, other than the injection of optimized independent noise in each layer, we can accurately capture many many pairwise noise correlations in the retinal ganglion cell output.  This in and of itself is a major contribution to computational neuroscience: nobody has ever accurately captured such retinal noise correlations, for natural scenes, using a feedforward neural circuit model, whose internal structure matches that of the biological retina.\n\n3.\nWe agree that many MDSs and MDRs are difficult to interpret visually. Thus, our main conclusions lie in Fig 5,6, particularly related to the ongoing question about the role of noise correlations as described in the overall author rebuttal. In addition, our recent research finds that when constrained on a natural scene manifold, MDS will become more visually interpretable.\n\nQuestions:\n\n1.\nThe rationale for choosing the non-standard binomial distribution is the refractory period of retinal ganglion cells which induces a maximal spike count in a time bin. So $N$ is the cell-dependent parameter that we can directly determine from the observed refractory period in our experimental recording. After fixing $N$, $k$ is the parameter controlling the variability of the distribution (also cell-dependent), and $k$ can only be determined through model optimization. $r$ is not a parameter to be optimized, it rather controls the firing rate which is updated according to the current CNN output. We can improve our description about the binomial noise in the final paper to make it more clear.\n\n2.\nThe input to the one-hot layer is the output of the last convolutional layer, which is a 3d tensor where the first index is the channel index and the other two indices $i,j$ refer to the location. $k$ is the index of recorded neurons. So Eq.5 says that each output unit is a linear combination of units in different locations in a channel, and such combination weights will eventually converge to a one-hot vector after model training. \n\nIn Eq. 6, $p,q$ are also location indices like $i,j$. To intuitively understand why a one-hot $w^k$ can minimize the loss function, the key observation is that, suppose $w^k$ is a one-hot vector, then $|\\prod_{ij}(w^k_{ij}+\\delta_{pq,ij}-1)|=1$ for $(p,q)$ such that $w^k_{pq}=1$. While if $w^k$ is far from a one-hot vector, then such product will produce an extremely small number for any $(p,q)$. We will describe this more clearly in the final paper.\n\n3.\nCorrect. We will use a different notation to denote the eigenvalue in the final paper.\n\nLimitations:\n\n1. $\\frac{d\\Sigma}{dx}$ is expected zero for any point in stimulus space that yields mean hidden neuron activities where every ReLU neuron's activation is a reasonable distance away from a zero-crossing, measured in units of the standard deviation of noise injected into the ReLU.   Therefore, for the model, this statement is true over most of stimulus space except for those stimuli which place  the mean response of one or more ReLU's close to their zero crossing.  In essence, whenever every ReLU's mean response is just greater than its own noise input's standard deviation away from its zero crossing, small changes in stimulus do not change the linear response of the overall model, and therefore do not change the noise correlations $\\Sigma$.  Now given our model is an excellent model of the retina itself, this statement should be a good approximation for the retina.  One could test this by providing two nearby stimuli to the retina and directly measuring noise correlations and showing they do not change much.  But this new biological experiment is beyond the scope of this paper.  We hope our direct prediction of measured noise correlations suffices.\n\n2. \nSee the overall author rebuttal as to the motivation of model architecture. In particular we are building on an extensive prior architecture search and our results are robust and reproducible across 4 different model fits to 4 different biological retinas. \n\n3.\nIt is a well-known anatomical principal that ganglion cells tile the retina, so at each location there is only one cell of a given type. This mosaic organization also occurs in the salamander retina (Fig. 2 in Kastner & Baccus, 2011) implying that single recorded neurons can be generalized by the one-hot layer to the population of the same cell type. It is acknowledged that all cell types may not have been recorded from. As to generalization to different species, the primary properties at play here, strong nonlinearity under natural scenes, the presence of noise correlations of a similar magnitude, similar nonlinear phenomenology and a mosaic ganglion cell organization have all been observed across different species of vertebrate retina.  \n"}, {"review_id": "OcFzx8enos", "reviewer": "Reviewer_pZyp", "summary": "This paper uses a fitted surrogate neural network model to approximate the Fisher information metric induced on image space by the retinal ganglion cell population code. They use this analysis to argue that noise correlations in the retina are information-limiting. ", "questions": "- Line 33: In addition to Wang & Ponce 2021, earlier work by Shao et al., \"The Riemannian geometry of deep generative models\" (2018) should be cited.\n\n- Lines 73-81: It could be useful to mention that (8) has been termed the \"linear Fisher information\" in past works, and to cite Beck et al., \"Insights from a Simple Expression for Linear Fisher Information in a Recurrently Connected Population of Spiking Neurons\" (NECO 2011) and Kanitscheider et al., \"Measuring Fisher Information Accurately in Correlated Neural Populations\" (PCBI 2015). \n\n- Line 84: Why is [1] cited rather than a general text on Riemannian manifolds (or, indeed, [14])?\n\n- Line 86: Using the acronym \"MDS\" for \"most discriminable stimulus\" conflicts with the standard use of \"MDS\" to mean \"multidimensional scaling.\" Please consider using an alternative acronym, e.g., \"MDI\" for \"most discriminable input.\"\n\n- Lines 152-153: Though the binomial noise model yields the best fit, could it still be worthwhile to reproduce Figure 5 for the alternative noise models, as a sort of robustness check? \n\n- Line 157: Please provide more detailed information on compute resources than \"on NVIDIA GPUs.\"\n\n- Lines 185-186: \"Statistics for preparations with longer test sequences tend to be more reliable and consequently our model performs better on these.\" Data to support this claim is not shown, correct? It would be useful to show more clearly the effects of this heteroskedasticity. \n\n- Figure 4: Please state in the caption whether these measures are computed on held-out stimuli.\n\n- Lines 206-208: Could you make this apparent link to ideas of hierarchical predictive coding more precise? \n\n- Figure 6a-b: It would be useful to remind the reader of the definitions of stochasticity, sensitivity, and discriminability in the caption. \n\n- Figure 6a: In the three sub-panels of Panel (a), there are so many dots overlaid on top of each other that it is hard to tell how single neurons are distributed within the blobs of data. Showing 2D histograms might be more informative. It would also be useful to plot stochasticity, sensitivity, and discriminability against one another rather than only against firing rate (depending on the result, this could be deferred to the supplement). \n\n- Figure 6b: The linear fit to the discriminability-mean firing rate relationship is not very convincing because of the substantial spread in the data. \n\n- Lines 244-247: I agree that it would be interesting to compute geodesics, but I'm less optimistic that this could be done in practice due to the numerical challenges associated with solving the geodesic equation in high dimensions. One previous attempt by Hénaff and Simoncelli (\"Geodesics of learned representations,\" ICLR 2016) used an optimization-based approach, with somewhat mixed success. Can you elaborate on why you think this is a realistic possibility?", "limitations": "The authors provide some discussion of the limitations of their work, but I think a more comprehensive assessment of the possible failings of each step of their approach would enhance the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The question of whether noise correlations in early sensory areas are information-limiting is a classic and broadly interesting issue in theoretical and systems neuroscience. This paper offers a novel approach to the problem of Fisher information estimation. \n\n2. I think the observation that the most discriminable stimulus depends on the base point (Lines 196-208) is an intriguing finding, though perhaps not so surprising. Making a convincing link between this variation and neural adaptation would be an interesting topic for future study (I fully acknowledge that a detailed characterization of this phenomenon is likely beyond the scope of the present manuscript). ", "weaknesses": "1. I have several concerns regarding the robustness of the paper's conclusions to the architecture and goodness of fit of the surrogate model. One key strength of some past work on information limiting correlations---I have in mind Rumyantsev et al, cited as [8] in the submitted manuscript---is the demonstration that the quantities of interest can be accurately resolved given a number of measurements comparable to the number of experimental recordings. It is not clear to me whether the same should be true here. Can the authors provide evidence that their approach gives accurate estimates of the directions of maximal variation, and that the surrogate model is not overfit?\n\n2. In a similar vein, the authors argue reasonably convincingly that it is reasonable to neglect stimulus-dependence of the noise covariance (i.e., $d\\Sigma/dx \\simeq 0$) for a ReLU network model, but do not give direct evidence that this is a reasonable assumption for the retinal population code. \n\n3. In several places, data supporting the authors' analysis decisions are not shown, and their choices are not always clearly described. For example, in the truncated approximation (8) for the Fisher information matrix, can you specify (at least in the SI) the precise criterion used to select the \"hundreds of the most stochastic modes\" included (Lines 170-171)?\n\n**A pedantic concern:** The paper and supplementary material contain several small violations of the anonymity requirements. My score for the paper does not take this concern into account. \n\n- Lines 257-259: \"Acknowledgements: This work was supported by grants from the NEI, R01EY022933, R01EY025087 and P30EY026877 (SAB).\"\n\n- In the supplementary ZIP, under code, the LICENSE file contains the line \"Copyright (c) 2023 Baccus Lab.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and helpful suggestions.\n\nWeaknesses:\n\n1. We are analytically computing the Fisher information given an excellent model of the retina. Rumyantsev et al. in contrast had to estimate Fisher information directly from data without a model. Therefore they had to worry about whether they had enough trials and neurons. We don't have to worry about this because our model is accurate to two essential properties under natural scenes, the sensitivity as captured by an accurate model of the neural code, and an accurate fit of noise correlations. Given these measures, everything afterwards in computing Fisher information is analytic.\n\n2. $\\frac{d\\Sigma}{dx}$ is expected zero for any point in stimulus space that yields mean hidden neuron activities where every ReLU neuron's activation is a reasonable distance away from a zero-crossing, measured in units of the standard deviation of noise injected into the ReLU.   Therefore, for the model, this statement is true over most of stimulus space except for those stimuli which place  the mean response of one or more ReLU's close to their zero crossing.  In essence, whenever every ReLU's mean response is just greater than its own noise input's standard deviation away from its zero crossing, small changes in stimulus do not change the linear response of the overall model, and therefore do not change the noise correlations $\\Sigma$.  Now given our model is an excellent model of the retina itself, this statement should be a good approximation for the retina.  One could test this by providing two nearby stimuli to the retina and directly measuring noise correlations and showing they do not change much.  But this new biological experiment is beyond the scope of this paper.  We hope our direct prediction of measured noise correlations suffices.\n3. We usually consider the top 500 most stochastic modes (top 500 principal components) that can explain 85% to 90% of the total variance depending on the stimulus. Additionally, as shown in Fig. 5b, MDS mostly correlates with the top 30 most stochastic modes. Therefore, we claim that our summation converges for the purpose of computing MDS. We can state our criterion of this selection more explicitly in the final version.\n\nQuestions:\n\n1. Thank you for adding the new citation, we will cite it in the final version.\n2. We will mention the terminology and add the citations in the final version.\n3. We cited reference [1] as the source of the insight that a Riemannian metric can transform between stimulus space and representation space, whereas a general text typically doesn't concern neural networks or sensory systems.\n4. Thanks for bringing this up, we will use MDI instead in the final paper.\n5. Fig. 5 is about the geometry of noise correlations. That is to say, the result in Fig. 5 is a product of Gaussian noise added before the final layer. Noise added to the final layer, no matter it is binomial noise or other noise models, is independent noise, will not affect the conclusion in Fig. 5. To confirm this, we have plotted Fig. 5 even without the final independent noise, and the conclusion holds. We can clarify the role of the final independent noise.\n6. We will provide the detailed GPU information in the final paper.\n7. In the caption of Fig. 3, we stated that \"the circle radius is proportional to the square root of the total length of the test set\". And one can indeed see from Fig. 3 that larger circles are matched better.\n8. The stimuli belong to the test set.\n9. The correlation between the MDR and mean response provides a mechanism for a higher brain region to estimate the most informative response changes of retinal ganglion cells in the immediate future based on the current response. Using a strategy known as Bayesian Inference,  the higher brain by detecting the response, $R$, could potentially increase sensitivity to the most informative $\\Delta R$ in order to extract the most salient and informative features of the stimulus, analogous to attentional cueing. In addition, by subtracting out the mean response to emphasize the informative $\\Delta R$, the higher brain could use predictive coding, which yields a more efficient representation by encoding the prediction error. A combined optimal readout strategy may combine both elements, known as Bayesian Predictive Coding (Aitchison et al. 2017).\n10. Yes, we can remind readers that the definitions are in the theory section.\n11. We agree that the suggested plots directly comparing different quantities could be put in the supplementary material, since they are less relevant to our main conclusions but some readers might be interested.\n12. Yes, actually we should have set the y-axis to start from 0 since we are looking at the fluctuation relative to the absolute value. In that case we believe that the plot will look much more convincing. (See Fig. A4 in the attached pdf.)\n13. We also believe that it would be impractical to solve the high-dimensional differential equation to find the geodesic rigorously for our model. But there could exist approximation methods to help answer the question. The first step could be computing the geodesic distances of a finite set of paths. With these data it becomes possible to refine the path searching space if paths with small geodesic distances share any common features (dimensionality reduction methods could be used here). Applying some constraints to the path in the pixel space with Euclidean metric could accelerate the searching. Reducing the full model to some approximately equivalent small model would also help.\n"}]

Author rebuttal:
Thank you for pointing out the strengths of our work. As stated by two reviewers, a primary important contribution of our work is that we have solved a long-lasting debate about the role of noise correlations in the retina under natural visual stimuli using a novel information-geometric framework, one that can also be generalized to other neural systems. More generally, although numerous analyses have been conducted about discriminability for simple stimuli, and deterministic sensitivity under natural scenes have been modeled, our stochastic model is the first to be able to capture both sensitivity and stochasticity under natural stimuli for any system, thus enabling a broad set of questions about stimulus discriminability that were previously inaccessible. 

One shared concern of the reviewers is the rationale of our selections of the model, hyperparameters, and optimization method and whether the scientific results are robust against these selections. Although these are reasonable concerns, some reviewers may not be familiar with previous work that is the foundation of our current model, which consists of the deterministic part (the CNN and the one-hot layer) and the stochastic part (Gaussian and binomial noise). We will discuss these two parts separately.

The CNN architecture and corresponding hyperparameters are adopted from citations [11, 13] and (Maheswaranathan, et al. 2023), including the number of layers, the number of channels, nonlinearities, implementation of convolutional layers, etc.. This previous work has not only shown that the current model setting can achieve the state-of-the-art performance in fitting the RGC mean firing rates for natural scenes, but also that the model has a clear correspondence with the real retina given the phenomenology reproduced by the model and the correlation between the activity of model internal units and recorded interneuron responses that the model was never fit to. Therefore, this CNN is not simply a statistical approach for fitting experimental data, but rather a mechanistically interpretable retinal model that captures the computations, circuits and representations of the salamander retina.

For the stochastic part of the model, which is new to our current work, it is important to note that our optimization did not change the deterministic parameters described above, but only optimized the addition of noise parameters to capture as much as possible the full properties of second-order retinal stochasticity as a function of stimulus and response. We have attempted a variety of approaches in fitting these second order statistics, including Poisson noise in intermediate layers, using Gaussian noise in the final layer, neglecting the refractory period, only fitting noise correlations, and so on. We have concluded that the current stochastic setting and optimization methods can fit second order statistics most accurately and fit different experimental preparations robustly. We understand that our optimization method for different model components may look complicated, but this is because fitting multiple second order statistics simultaneously across a large natural stimulus set is non-trivial and challenging.

To test robustness, we have tested models trained for four experimental preparations with different CNN and noise parameters, different number of output channels and cell-to-channel mappings,  different types of nonlinearities and different implementations of the convolutional layer, and different noise parameter selection criterions. We claim that all results reported in the paper are robust against these variations.

Furthermore, the Fisher information matrix only depends on sensitivity and the noise covariance matrix (stochasticity). Sensitivity was computed with the noiseless model which has been validated extensively by previous works ([11, 13] and Maheswaranathan et al. 2023), whereas the covariance matrix is proportional to the noise correlation matrix, which is captured well by our model as shown in Fig. 3A. Therefore, we claim that our computation of Fisher information metric is reliable.

In the final paper, we will emphasize more about the relation between our work and previous literatures, the robustness of our results, and the challenge of fitting experimental data.



Meta-review: This paper performs a novel analysis of neural responses in the salamander retina.  They find that natural scene noise correlations limit information transmission which is opposite to a common speculation.   Reviewers found the initial paper did not clearly justify model design choices and properly relate the work to the prior literature, but upon rebuttal and discussion, reviewers all agreed that the paper was worthy of acceptance providing that these clarifications are added to the paper.
Decision: Accept (poster)