SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Early stochastic encoding models of the retina were primarily developed for artificial stimuli [27, 28, 29], and their results cannot be directly generalized to natural scenes due to the stimulus-dependent nature of noise in the retina. Here our fitted model was evaluated by comparing with neural data various commonly used second-order statistics including correlation measures and single-cell variability measures (Figure 3). Surprisingly, the model prediction matches the neural data remarkably well despite that the model only has a few noise parameters. This result suggests that because noise in the retina propagates through the same network pathways as the signal, they are both captured by the CNN structure. This indicates a strikingly simple origin for noise correlations between all ganglion cell pairs: independent noise that starts in individual upstream cells, and then propagates through the same network that creates mean stimulus sensitivity. Statistics for preparations with longer test sequences tend to be more reliable and consequently our model performs better on these. For the first order statistics (mean firing rates), the Pearson correlation coefficient between model predictions and data ranges from  $70\%$  to  $80\%$ , which is similar to Ref. [12, 13]. Such an accurate model of both sensitivity and stochasticity guarantees that the computation of Fisher information matrix is reliable (Eq. 2).
4.2 The most discriminable directions
Some examples of the most discriminable stimulus direction (MDI) and the most discriminative response mode (MDR) are shown in Figure 4a (see the Supplementary Material for more examples). Often the top eigenvalues were similar, and so to avoid over-interpreting the importance of individual eigenmodes, we found the sparsest direction (with the least L1 norm) in the space spanned by the top 5 MDIs for the purpose of visualization. Only spatial components are presented here as we found that temporal components for different stimuli are both very similar to each other and to those of instantaneous receptive fields [13]. We found that the MDI is strongly stimulus dependent, and is
Figure 3: The model captures a variety of second-order statistics of neural responses (see the Supplementary Material for definitions). (a) Comparison of total pairwise correlation, stimulus correlation, and noise correlation between neural data and model prediction. Black lines represent perfect agreement. Each circle corresponds to a pair of neurons in an experimental preparation coded by its color and the circle radius is proportional to the square root of the total length of the test set. (b) Comparison of single-cell variability measures including the Fano factor and trial-to-trial correlation between neural data and model prediction. Neuron indices share the same color code with (a).
typically localized in a region where the sensitivity is high, an effect that arises from the locality of convolutional connections. The MDI tends to appear around the central region (red box in Figure 4a) because pixels within this area affect more output units than pixels near the edges or corners.
One general observation is that the MDI varies greatly across stimuli. From a mathematical point of view, this is to say that certain features of the tangent space depend on the base point. With respect to neural function, this variability potentially indicates the presence of adaptation to the stimulus. We found that the most discriminable direction is related to the base point through the representation space such that the MDR and the mean response are correlated (Figure 4b). This correlation can be attributed to the fact that neurons with intermediate firing rates exhibit the highest single-cell sensitivity and discriminability (Figure 6a). As a result, the MDI sometimes captures the most salient feature of the stimulus (the first row of Figure 4a), although this is not always the case (the second row of Figure 4a). Similar stimuli and mean responses likely produce distinct MDIs especially when their mean response pattern has multiple spatial components that can each yield a localized MDR (the third and fourth rows of Figure 4a). Moreover, the correlation between MDR and mean response provides a mechanism for higher brain regions to estimate the most informative future response changes of retinal ganglion cells based on the current response and thereby adjust the corresponding sensitivity, which could have benefits for the property of predictive coding [30]. Additionally, we found that
(c)
Figure 4: Most discriminable directions. (a) Examples of the most discriminable directions for different stimuli in the test set. The first column shows a representative frame of the spatiotemporal stimulus. The second column shows the spatial component of the spatiotemporal MDI obtained from the singular value decomposition. Red boxes correspond to the unit array in the final layer. The third column shows the mean response averaged over cell-type channels. The fourth column shows the absolute values of MDR averaged over channels. The last column shows the superposition of the MDI and stimulus. (b) Histogram of the cosine similarity between the mean response vector and its projection onto the space spanned by the top 5 MDRs. Similarities between shuffled pairs are also shown for baseline comparison. (c) One example of the stimulus (left panel, the central  $18 \times 18$  array is shown) and the corresponding MDR for different cell type channels.
different parts of the stimulus generates the MDR for different cell types (Figure 4c), indicating that different cell types signal different stimulus regions in the image as conveying the most information.
4.3 Noise correlations in the retina limit information coding
The role of noise correlations in population coding has long been debated. Multiple scenarios have been advanced where noise correlations are detrimental to information coding [31, 10]. However, others suggest that noise correlations under certain circumstances may not limit information coding and can even be beneficial [32, 6, 9]. More specifically, it has been proposed that in response space, noise correlations decrease information when the differences in the signal are aligned with the noise direction, and increase information when they are orthogonal [33] (Figure 5a).
Here we tested these two hypotheses for the retina by computing cosine similarities between MDRs and MNRs. It turns out that the most discriminative mode is quite aligned with the most stochastic mode and orthogonal to modes with small variability (Figure 5b), implying that noise correlations in the retina are information-limiting correlations. We also performed a more direct test of this conclusion by analyzing trial-shuffled responses, which have independent noise. We found that removing noise correlations in this way created responses that were significantly more discriminative than the original model outputs with correlated noise for every population size (Figure 5c). To understand the mechanism of this reduction in information, we analyzed the eigenspectrum of the Fisher information matrix (Figure 5d). Both the sensitivity and the stochasticity increase with the discriminability of MDR, and their ratio reaches a maximum at the noisiest mode. Therefore, we propose that the fundamental reason for such information-limiting correlation is that noise propagates through the same network as the signal, causing the stochasticity ellipsoid to align closely with, and be rounder than, the sensitivity ellipsoid. One trivial and extreme example of this arrangement is when all noise originates at the level of the stimulus, e.g. from photoreceptors, then the resulting noise correlations will obviously be detrimental.
(d)
Figure 5: Noise correlations in the retina are detrimental to information coding. (a) Compared to independent noise (left), noise correlations can either increase information (middle) or decrease information (right) [33]. (b) Cosine similarities between the top 5 MDRs and the top 30 MNRs averaged over stimuli. (c) Discriminability of MDI for different numbers of output units. The plot for trial-shuffled responses with independent noise is shown for comparison. Output units are randomly selected from the central  $500\mu m \times 500\mu m$  region [34]. (d) Sensitivity and stochasticity of the top 200 MDRs averaged over stimuli.
4.4 Firing rate dependency
The firing rate dependencies of stochasticity, sensitivity, and discriminability for individual neurons and the whole population are shown in Figure 6. For individual neurons, both the sensitivity and the stochasticity peak around intermediate firing rates and decay at high firing rates (a consequence of the refractory period). Thus, the single-cell discriminability also saturates around the intermediate firing rate. In contrast, population coding does not exhibit such a decay at high rates, implying that neurons can encode information complementarily, which is a benefit of population coding. In other words, even when a fraction of neurons get saturated by the stimulus and become uninformative at high firing rates, other neurons can still provide discriminative responses to stimulus changes. Although the population sensitivity and stochasticity significantly increase with the mean firing rate, the firing rate dependency of the population discriminability remains relatively flat, which is potentially beneficial for information transmission assuming that stimuli are uniformly distributed on the natural scene manifold [35, 36].
In summary, we established a novel information-geometric framework to understand the high dimensional representation manifold of the retinal population for natural scenes, one that can also be
Figure 6: Firing rate dependency. (a) Log-scale 2D histograms showing the firing rate dependencies of stochasticity, sensitivity, and discriminability for individual neurons whose definitions are given in the theory section. Mean values were plotted with thick black lines. (b) Scatter plots showing the population-averaged firing rate dependencies of stochasticity, sensitivity, and discriminability for the first MDR. The linear regression of the populational discriminability is shown.
generalized to other neural systems. However, there still exist remaining questions and further work that can be pursued within this framework.
First, our analysis primarily focused on examining the local metric tensor, which characterizes the effects of infinitesimal stimulus changes. It would be interesting to extend the analysis to finite stimulus changes to explore more global geometry of the representation manifold. For example, one exciting direction is to find geodesics between two stimuli under the information metric and observe how one stimulus transitions into another with the least amount of discriminable changes for the retinal population [37].
Second, we found that noise correlations in the retina are information-limiting correlations. This conclusion is surprisingly robust against many hyperparameter selections, even when second order statistics are not fitted well. Therefore, it remains unclear whether this is a general result for all feedforward networks with independent noise applied to their inputs and intermediate layers. If this is indeed the case, noise correlations in other brain areas that do not limit information coding must arise from different mechanisms.
Finally, the Fisher information metric in this work was computed in the full stimulus space. However, most points in the stimulus space represent meaningless noise images that no animal has ever encountered. Therefore, an important avenue for future research would involve incorporating the structure of natural stimuli and computing the metric with stimuli constrained to a natural-scene manifold.
This work was supported by grants from the NEI, R01EY022933, R01EY025087 and P30EY026877 (SAB).
[1] Michael Hauser and Asok Ray. Principles of riemannian geometry in neural networks. Advances in neural information processing systems, 30, 2017.
[2] Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
[3] Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43):e2200800119, 2022.
[4] Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315-323, 2018.
[5] Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations, 2021.
[6] Rubén Moreno-Bote, Jeffrey Beck, Ingmar Kanitscheider, Xaq Pitkow, Peter Latham, and Alexandre Pouget. Information-limiting correlations. Nature neuroscience, 17(10):1410-1417, 2014.
[7] Larry F Abbott and Peter Dayan. The effect of correlated variability on the accuracy of a population code. Neural computation, 11(1):91-101, 1999.
[8] Bruno B Averbeck and Daeyeol Lee. Effects of noise correlations on information encoding and decoding. Journal of neurophysiology, 95(6):3633-3644, 2006.
[9] Oleg I Rumyantsev, Jérôme A Lecoq, Oscar Hernandez, Yanping Zhang, Joan Savall, Radosław Chrapkiewicz, Jane Li, Hongkui Zeng, Surya Ganguli, and Mark J Schnitzer. Fundamental bounds on the fidelity of sensory cortical coding. Nature, 580(7801):100-105, 2020.
[10] Rava Azeredo da Silveira and Fred Rieke. The geometry of information coding in correlated neural populations. Annual Review of Neuroscience, 44:403-424, 2021.
[11] David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, 14(10):1317-1322, 2011.
[12] Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. Advances in neural information processing systems, 29, 2016.
[13] Niru Maheswaranathan, Lane T McIntosh, Hidenori Tanaka, Satchel Grant, David B Kastner, Joshua B Melander, Aran Nayebi, Luke E Brezovec, Julia H Wang, Surya Ganguli, et al. Interpreting the retinal neural code for natural scenes: From computations to neurons. Neuron, 111(17):2742-2755, 2023.
[14] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
[15] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
[16] Jeffrey Beck, Vikranth R Bejjanki, and Alexandre Pouget. Insights from a simple expression for linear fisher information in a recurrently connected population of spiking neurons. Neural computation, 23(6):1484-1502, 2011.
[36] Xue-Xin Wei and Alan A Stocker. Mutual information, fisher information, and efficient coding. Neural computation, 28(2):305-326, 2016.
[37] Olivier J Henaff and Eero P Simoncelli. Geodesics of learned representations. In 4th International Conference on Learning Representations, ICLR 2016, 2016.