SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xuehao Ding $^{1}$ , Dongsoo Lee $^{2}$ , Joshua B. Melander $^{3}$ , George Sivulka $^{4}$ , Surya Ganguli $^{5}$ , and Stephen A. Baccus $^{6}$
$^{1,5}$ Department of Applied Physics, Stanford University
$^{2,3}$ Neurosciences PhD Program, Stanford University
$^{4}$ Department of Electrical Engineering, Stanford University
$^{6}$ Department of Neurobiology, Stanford University
{xhding, dsnl, melander, gsivulka, sganguli, baccus}@stanford.edu
The ability for the brain to discriminate among visual stimuli is constrained by their retinal representations. Previous studies of visual discriminability have been limited to either low-dimensional artificial stimuli or pure theoretical considerations without a realistic encoding model. Here we propose a novel framework for understanding stimulus discriminability achieved by retinal representations of naturalistic stimuli with the method of information geometry. To model the joint probability distribution of neural responses conditioned on the stimulus, we created a stochastic encoding model of a population of salamander retinal ganglion cells based on a three-layer convolutional neural network model. This model not only accurately captured the mean response to natural scenes but also a variety of second-order statistics. With the model and the proposed theory, we computed the Fisher information metric over stimuli to study the most discriminable stimulus directions. We found that the most discriminable stimulus varied substantially across stimuli, allowing an examination of the relationship between the most discriminable stimulus and the current stimulus. By examining responses generated by the most discriminable stimuli we further found that the most discriminative response mode is often aligned with the most stochastic mode. This finding carries the important implication that under natural scenes, retinal noise correlations are information-limiting rather than increasing information transmission as has been previously speculated. We additionally observed that sensitivity saturates less in the population than for single cells and that as a function of firing rate, Fisher information varies less than sensitivity. We conclude that under natural scenes, population coding benefits from complementary coding and helps to equalize the information carried by different firing rates, which may facilitate decoding of the stimulus under principles of information maximization.
Neural populations represent information with their collective activity. In the field of sensory neuroscience, whereas there is much focus on characterizing the sensitivity of neurons to different stimuli, the true function of a sensory system is to discriminate between stimuli. A quantitative description of discriminability in a neural population requires not only a precise knowledge of sensitivity to stimuli, but also the structure of noise correlations in the population.
Neural representation manifolds constitute a modern approach to studying neural populations in a geometric framework, and recent machine learning literature has seen extensive research on the geometry of representation manifolds of neural networks trained for artificial tasks [1, 2, 3, 4, 5]. In separate work, Fisher information has been widely utilized in neuroscience for measuring decoding fidelity, particularly in theoretical studies exploring the effect of noise correlations [6, 7, 8]. In this study, we propose a framework for studying the Fisher information geometry of representation manifolds for natural visual scenes in a population of visual neurons, in particular the retina. In addition, for the first time, we apply such a manifold analysis to the neural representation of natural stimuli for real data.
The discriminability of biological visual systems has long been studied for low-dimensional and discrete artificial stimuli [9, 10, 11]. However, naturalistic visual stimuli in the real world are inherently high-dimensional and discriminability is generally heterogeneous across different types of changes in a stimulus. Recent advances in convolutional neural networks (CNNs) have enabled the accurate prediction of retinal responses to naturalistic stimuli, as well as ethological computations [12, 13, 14]. Nevertheless, previous work has predominantly focused on deterministic trial-averaged neural codes, which is not adequate for computing discriminability. Thus an encoding model of stochastic retinal representations for natural scenes is required, which constitutes one of the primary contributions of this work.
We present here a highly accurate model of both sensitivity and stochasticity for the retinal ganglion cell (RGC) population, thus creating the first accurate model of discriminability in a neural population for its natural inputs. From an analysis of the Fisher information geometry of the retinal representation of stimuli, we find that the most discriminable direction for the RGC population varies greatly across stimuli and is correlated with the mean neural response. We further address a long standing question as to the effect of noise correlations on information. We find that the most discriminative mode in the representation space often aligns with the most stochastic direction, implying that noise correlations in the retina are detrimental to information coding. Finally, we analyze how discriminability depends on firing rate of individual cells and in the population, and discuss the implications in relation to principles of information maximization.
In this section, we describe basic theory of information geometry [15] and its application to sensory neuroscience. Let  $\pmb{x}$  denote the stimulus vector and  $\pmb{y}$  denote the spike count vector whose dimension is equal to the number of neurons encoding the stimulus. Each stimulus induces a conditional probability distribution of sensory neural responses  $P(\pmb{y}|\pmb{x})$  and mean firing rates  $\mu(\pmb{x}) \coloneqq E[\pmb{y}|\pmb{x}]$ . Along any decoding vector  $\pmb{a}$ , the sensitivity is defined as  $||\nabla_{\pmb{x}}\langle\mu(\pmb{x}),\pmb{a}\rangle||$ , while the stochasticity is defined as the standard deviation of  $\langle\pmb{y},\pmb{a}\rangle$  conditioned on  $\pmb{x}$  (Figure 1). In our case in the retina,  $\mu(\pmb{x})$  is modeled by a noiseless CNN and  $P(\pmb{y}|\pmb{x})$  is modeled by the stochastic counterpart.
The representation manifold is defined as the manifold of mean neural responses induced by stimuli:  $\{\pmb {\mu}(\pmb {x})\}$ . We also define a statistical manifold of conditional response distributions  $\{P(\pmb {y}|\pmb {x})\}$  parameterized by stimuli  $\pmb{x}$ , which yield a natural coordinate system. To equip the statistical manifold with a Riemannian metric, we consider the Kullback-Leibler divergence between two nearby distributions:
$$
\begin{array}{l} d s ^ {2} (\boldsymbol {x}, \boldsymbol {x} + d \boldsymbol {x}) := 2 \cdot D _ {K L} [ P (\boldsymbol {y} | \boldsymbol {x}) | | P (\boldsymbol {y} | \boldsymbol {x} + d \boldsymbol {x}) ] \\ \approx - 2 \int P (\boldsymbol {y} | \boldsymbol {x}) \ln \left[ 1 + \frac {d P (\boldsymbol {y} | \boldsymbol {x})}{P (\boldsymbol {y} | \boldsymbol {x})} \right] d \boldsymbol {y} \\ \approx \sum_ {i j} \int P (\boldsymbol {y} | \boldsymbol {x}) \frac {\partial \ln P (\boldsymbol {y} | \boldsymbol {x})}{\partial \boldsymbol {x} _ {i}} \frac {\partial \ln P (\boldsymbol {y} | \boldsymbol {x})}{\partial \boldsymbol {x} _ {j}} d \boldsymbol {y} d \boldsymbol {x} _ {i} d \boldsymbol {x} _ {j} \tag {1} \\ = d \boldsymbol {x} ^ {T} \cdot \boldsymbol {I} (\boldsymbol {x}) \cdot d \boldsymbol {x}, \\ \end{array}
$$
where  $\mathbf{I}(\mathbf{x})$  is the Fisher information matrix with respect to the stimulus, and the KL divergence is linearized up to the second order. The Fisher information metric  $g^{stim}|_{\mathbf{x}}(\mathbf{e}_i,\mathbf{e}_j) = \mathbf{I}(\mathbf{x})_{ij}$  naturally characterizes the local discriminability of stimuli near  $\mathbf{x}$ .
When we assume that neural responses follow multivariate Gaussian distributions  $\pmb{y}|\pmb{x} \sim N(\pmb{\mu}(\pmb{x}), \pmb{\Sigma}(\pmb{x}))$  and the covariance matrix is locally independent of the stimulus,  $\frac{d\pmb{\Sigma}(\pmb{x})}{dx} = \mathbf{0}$ ,
Figure 1: Schematic of basic concepts in a sensory system. The sensory network maps from the stimulus space to the representation manifold. The differentiation operator maps the stimulus tangent space to the representation tangent space. With noise in the network, each stimulus corresponds to a cloud of points in the representation space. Generally, the sensitivity (the blue and the red arrows) and the stochasticity (the green oval) are both heterogeneous.
the Fisher information metric can be expressed as
$$
\boldsymbol {I} (\boldsymbol {x}) = \frac {d \boldsymbol {\mu} (\boldsymbol {x})}{d \boldsymbol {x}} ^ {T} \cdot \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {x}) \cdot \frac {d \boldsymbol {\mu} (\boldsymbol {x})}{d \boldsymbol {x}}, \tag {2}
$$
where  $\frac{d\mu(x)}{dx}$  is the Jacobian matrix. Eq. 2 is commonly termed linear Fisher information which bounds the performance of unbiased linear decoders, which may be more biologically plausible than nonlinear decoders [16, 17, 18]. The latter assumption is a good approximation in our case because a neural network consisting only of ReLU nonlinearities is locally equivalent to an effective linear network [19] and the covariance matrix of a linear network is a constant matrix. This metric expression is interpretable in that discriminability is proportional to sensitivity and inversely proportional to noise. The length of a geodesic connecting two stimuli under this information metric measures the discriminability between them, when represented by the neural population.
Equivalently, the mean response  $\pmb{\mu}(\pmb{x})$  on the representation manifold can also serve as a coordinate system. Then the metric is given by  $g^{resp}|_{\pmb{x}}(\pmb{e}_i,\pmb{e}_j) = \pmb{\Sigma}^{-1}(\pmb{x})_{ij}$ . The metric matrix transforms between the stimulus space and the representation space as a tensor [1].
We define here several important quantities for analysis. The eigenvector of  $I(x)$  associated with the largest eigenvalue is the most discriminable direction in the stimulus tangent space, which we term the most discriminable input (MDI) at stimulus  $x$ . The square root of the eigenvalue is the corresponding discriminability [20]. The MDI in the stimulus space can be pushed forward to the most discriminative response mode (MDR) in the representation tangent space via the Jacobian matrix: MDR  $\propto \frac{d\mu(x)}{dx} \cdot \mathrm{MDI}$ . Similarly, the most sensitive direction in the stimulus (input) tangent space (MSI) can be obtained as the principal eigenvector of  $\frac{d\mu(x)^T}{dx} \cdot \frac{d\mu(x)}{dx}$  and the associated most sensitive response mode (MSR) obeys MSR  $\propto \frac{d\mu(x)}{dx} \cdot \mathrm{MSI}$ . Additionally, the most stochastic (noisy) response mode (MNR) is defined as the principal eigenvector of the noise covariance matrix  $\Sigma(x)$ , which is no more than principal component analysis. For a ReLU neural network with small stochasticity, the stimulus dependence of the aforementioned tangent space features is solely derived from the stimulus dependence of the effective linear network [19].
One practical challenge of the above analysis with natural visual stimuli is the extremely high dimensionality of the stimulus space. Fortunately, we only need to compute the Fisher information matrix in a subspace defined by the recorded cell population. First we define the instantaneous receptive field for each cell as the direction of the greatest sensitivity to the stimulus at one stimulus point, which is the gradient of that cell's response,  $\frac{d\mu_i(x)}{dx}$  [13]. Then we only need to compute Fisher information in the subspace spanned by instantaneous receptive fields, as justified by the following theorem.
Theorem: The top  $N$  most discriminable directions are linear combinations of the instantaneous receptive fields, where  $N$  is the number of output neurons.
Proof: Let  $IR \coloneqq \operatorname{span}(\{\frac{d\mu_i(\boldsymbol{x})}{dx}\})$  represent the subspace spanned by instantaneous receptive fields. Suppose that the most discriminable direction  $v_1$  is not in  $IR$ , then we can decompose  $v_1 = v_1^o + \alpha v_1^p$ , where  $v_1^o$  is the orthogonal component in  $IR^\perp$  and  $\alpha v_1^p$  is the parallel component
Figure 2: Model architecture. The first layer consists of eight  $15 \times 15$  spatiotemporal convolutional filters. The second layer consists of eight  $11 \times 11$  filters. The final layer consists of  $C9 \times 9$  filters, where  $C$  represents the number of cell types. Batch normalization is applied after each convolutional filter. The nonlinearities in the first two layers are ReLUs, while in the last layer, the nonlinearity consists of the SoftPlus function followed by a parametric tanh function. Gaussian white noise is added to the stimulus and the first two convolutional layers, and independent binomial noise is applied to the final nonlinearity. The one-hot layer selects the location of the unit in the final convolutional layer to match that of the recorded neuron. Further details are explained in the main text.
in  $IR, ||\pmb{v_1}|| = ||\pmb{v_1^p}|| = 1, |\alpha| < 1$ . Then following Eq. 2 we have
$$
\begin{array}{l} g ^ {s t i m} \left(\boldsymbol {v} _ {1}, \boldsymbol {v} _ {1}\right) = \alpha^ {2} g ^ {s t i m} \left(\boldsymbol {v} _ {1} ^ {\boldsymbol {p}}, \boldsymbol {v} _ {1} ^ {\boldsymbol {p}}\right) \\ <   g ^ {\text {s t i m}} \left(\boldsymbol {v} _ {1} ^ {\boldsymbol {p}}, \boldsymbol {v} _ {1} ^ {\boldsymbol {p}}\right), \tag {3} \\ \end{array}
$$
which contradicts with the fact that  $\pmb{v_{1}}$  is the most discriminable direction. Therefore,  $\pmb{v_{1}}$  is in  $IR$ .
Replacing  $IR$  with  $IR \cap \{v_1\}^\perp$ , with the same analysis we can prove that the second most discriminable direction  $v_2$  is also in  $IR$ . Subsequently, the whole theorem can be proven.
The spiking activity of a population of tiger salamander retinal ganglion cells was recorded in response to a sequence of jittering natural images. We created a stochastic encoding model capturing  $P(\pmb{y}|\pmb{x})$  with the CNN architecture that included independent noise in the stimulus and in each unit of the CNN. The CNN parameters were trained using backpropagation to fit the mean firing rates  $\mu(\pmb{x})$  to create a deterministic model, and then these parameters were frozen. Noise parameters included one parameter for each layer to set the standard deviation of the independent noise, and one parameter for binomial noise in each ganglion cell. These were then optimized by maximizing the log-likelihood of the empirical data and exploring the parameter space to match a few second-order statistics of the neural responses. Once the parameter optimization was completed, we computed the Fisher information metric with the model numerically for each stimulus and analyzed the most discriminable directions.
3.1 Stochastic model
The model architecture is shown in Figure 2. The model takes a spatiotemporal visual stimulus as an input and its output is a set of spike counts, one for each neuron in each time bin. The deterministic part of the model including its hyperparameters and training method is adopted from Ref. [12, 13, 14] which is a three-layer CNN chosen after an extensive architecture search. This CNN is the SOTA retinal encoding model for natural scenes in that (1) the prediction accuracy of RGC mean responses outperforms other models; (2) the activity of internal units is correlated with retinal interneuron responses recorded separately and not used in training; and (3) the model trained only on natural scenes can reproduce a wide range of phenomena induced by artificial stimuli. Thus, the deterministic part of our model has a clear correspondence with the real retina in that its circuitry is mechanistically interpretable and it captures many retinal computations. Every convolutional filter in our model was implemented by linearly stacking a sequence of  $3 \times 3$  small filters, which outperforms the traditional method [21]. A parametric tanh nonlinearity was attached to the last convolutional layer for the
purpose of enforcing the refractory period constraint:
$$
f (x; r _ {m a x}, a, b) = \frac {r _ {m a x}}{2} \cdot [ \tanh (a (x - b)) + 1 ], \tag {4}
$$
where  $a, b$  are cell-dependent model parameters, and  $r_{max}$  represents the cell-dependent maximal firing rate.
The essential module that enables us to study retinal population coding is the one-hot layer. Model parameters are fit with the one-hot layer, which bijectively maps a few output units of the final convolutional layer to recorded neurons by the end of the training. The one-hot layer is then removed after parameter optimization so that we can analyze the representation of the full population, typically consisting of thousands of units. The rationale behind this method is the mosaic organization of each retinal cell type, in which all ganglion cells of a given type tile the retina with their dendrites [11]. Formally, the one-hot layer is defined as
$$
\operatorname {o u t} (k) = \sum_ {i j} w _ {i j} ^ {k} \cdot \operatorname {i n p u t} (C (k), i, j), \forall k
$$
$$
\sum_ {i j} w _ {i j} ^ {k} = 1, \forall k \tag {5}
$$
$$
w _ {i j} ^ {k} \geq 0, \forall k, i, j
$$
where the input is the output of the last convolutional layer,  $k$  is the index of recorded neurons,  $i,j$  (and  $p,q$  below) are location indices,  $w^{k}$  is the linear combination weight vector, and  $C(k)$  is the neuron-to-channel map. Each  $w^{k}$  will converge to a one-hot array after model training with our one-hot loss function, which is adapted from the semantic loss in Ref. [22]
$$
\mathcal {L} _ {\text {o n e - h o t}} \left(w ^ {k}\right) \propto - \ln \sum_ {p q} \left| \prod_ {i j} \left(w _ {i j} ^ {k} + \delta_ {p q, i j} - 1\right) \right|, \tag {6}
$$
$$
\delta_ {p q, i j} = \left\{ \begin{array}{l l} 1 & p = i, q = j \\ 0 & \text {o t h e r w i s e} \end{array} \right.,
$$
and which reflects the locations of recorded neurons. Each output channel represents a ganglion cell type and  $C(k)$  was determined using hierarchical clustering of output channels (see the Supplementary Material for details).
Gaussian white noise was added to the stimulus to simulate stochasticity in photoreceptors, and was also added to pre-threshold signals in the first two layers to simulate stochasticity in bipolar cells and amacrine cells. We found that Gaussian noise outperforms Poisson noise in these layers, which can presumably be attributed to the noise generation mechanism in the retina and the central limit theorem. The stochasticity of ganglion cells was modeled as a parametric binomial probability mass function:
$$
p ^ {B} (n; r, k, N) \propto \frac {\Gamma (k \cdot N + 1)}{\Gamma (k \cdot (N - n) + 1) \cdot \Gamma (k \cdot n + 1)} r ^ {k \cdot n} \cdot (1 - r) ^ {k \cdot (N - n)}, \quad n = 0, 1, \dots , N \tag {7}
$$
where  $n$  denotes the spike count in a time bin,  $r$  is the rate parameter having a one-to-one correspondence with the mean firing rate  $\mu$ ,  $k$  is the variability parameter to be optimized, and  $N$  is the cell-dependent upper bound of spike counts in one time bin reflecting the refractory period of that cell, which one can directly determine from data. In practice, for given  $k$  and  $N$  we can interpolate the  $r - \mu$  mapping and use it to determine the rate parameter producing the desired mean firing rate, i.e.,  $\mu (r) = E_{p(n;r,k,N)}(n)$ . Although we have explored a variety of probability mass functions to fit the sub-Poisson statistics of retinal ganglion cells [23, 24], the binomial noise Eq. 7 performs the best among them (see the Supplementary Material) and also is the most natural choice considering the refractory period constraint. We stress that the binomial noise on ganglion cells is independent noise, and thus will not qualitatively affect the geometry and the functional implication of noise correlations.
Since the second order statistics are functions of the entire dataset and cannot be decomposed into terms depending on the individual stimulus, we optimized deterministic parameters and stochastic
parameters separately. CNN parameters and one-hot parameters  $\{w^k\}$  were optimized together with the standard Poisson loss function and  $\mathcal{L}_{\mathrm{one - hot}}$  to fit mean firing rates smoothed using a  $10~ms$  Gaussian filter. The optimization was performed using ADAM [25] via Pytorch [26] on NVIDIA Titan Xp, GeForce GTX TITAN X, GeForce RTX 3090, TITAN RTX, TITAN V, and GeForce RTX 2080 Ti GPUs. The network was regularized with an L2 weight penalty at each layer and an L1 penalty on the model output. Then tanh parameters of the final nonlinearity were fitted with the least square regression. Then we optimized stochastic parameters while freezing all deterministic parameters. Variability parameters  $k$  were optimized by maximizing the log-likelihood of recorded spike counts based on empirical firing rates. Standard deviations of Gaussian noise  $\{g_0,g_1,g_2\}$  in each layer were optimized through grid searches in order to best fit empirical noise correlations and stimulus correlations by minimizing the smoothed and weighted average of mean squared errors.
3.3 Computation of Fisher information metric
For each stimulus, the noise covariance matrix  $\pmb{\Sigma}$  was estimated by repeatedly simulating the stochastic model thousands of times. We then computed the eigendecomposition of  $\pmb{\Sigma}$  obtaining local information of the representation manifold:  $\pmb{\Sigma} = \sum_{i}\lambda_{i}\pmb{v}_{i}\pmb{v}_{i}^{T}$ . We applied the Gram-Schmidt orthogonalization to instantaneous receptive fields to obtain a basis of  $IR$ , which is a subspace of the stimulus tangent space. Then following Eq. 2, the Fisher information matrix with respect to the stimulus can be computed as
$$
\boldsymbol {I} = \sum_ {i} \frac {d \left\langle \boldsymbol {\mu} (\boldsymbol {z}) , \boldsymbol {v} _ {i} \right\rangle^ {T}}{d z} \lambda_ {i} ^ {- 1} \frac {d \left\langle \boldsymbol {\mu} (\boldsymbol {z}) , \boldsymbol {v} _ {i} \right\rangle}{d z}, \tag {8}
$$
where  $z$  is the stimulus coordinate under the  $IR$  basis. In practice, we summed only over hundreds of the most stochastic modes that can explain  $85\%$  to  $90\%$  of the variance, as the sensitivity and stochasticity of the other modes are both close to zero indicating that those modes are orthogonal to the tangent space of the representation manifold. The gradient with respect to the stimulus was computed using the noiseless model.
4.1 Model evaluation