SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suhas Shrinivasan $^{1,\dagger}$ , Konstantin-Klemens Lurz $^{1}$ , Kelli Restivo $^{2}$ , George H. Denfield $^{3}$ , Andreas S. Tolias $^{2,5}$ , Edgar Y. Walker $^{4,*}$ , Fabian H. Sinz $^{1,2*}$
$^{1}$  Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany
$^{2}$  Center for Neuroscience and Artificial Intelligence, Department of Neuroscience, Baylor College of Medicine, Houston, USA
<sup>3</sup> Department of Psychiatry, Columbia University, New York City, USA
$^{4}$  Department of Physiology and Biophysics, and Computational Neuroscience Center, University of Washington, Seattle, USA
$^{5}$  Department of Electrical and Computer Engineering, Rice University, Houston, USA
†Correspondence: suhas.shrinivasan@uni-goettingen.de, * Equal contribution
Prevailing theories of perception hypothesize that the brain implements perception via Bayesian inference in a generative model of the world. One prominent theory, the Neural Sampling Code (NSC), posits that neuronal responses to a stimulus represent samples from the posterior distribution over latent world state variables that cause the stimulus. Although theoretically elegant, NSC does not specify the exact form of the generative model or prescribe how to link the theory to recorded neuronal activity. Previous works assume simple generative models and test their qualitative agreement with neurophysiological data. Currently, there is no precise alignment of the normative theory with neuronal recordings, especially in response to natural stimuli, and a quantitative, experimental evaluation of models under NSC has been lacking. Here, we propose a novel formalization of NSC, that (a) allows us to directly fit NSC generative models to recorded neuronal activity in response to natural images, (b) formulate richer and more flexible generative models, and (c) employ standard metrics to quantitatively evaluate different generative models under NSC. Furthermore, we derive a stimulus-conditioned predictive model of neuronal responses from the trained generative model using our formalization that we compare to neural system identification models. We demonstrate our approach by fitting and comparing classical- and flexible deep learning-based generative models on population recordings from the macaque primary visual cortex (V1) to natural images, and show that the flexible models outperform classical models in both their generative- and predictive-model performance. Overall, our work is an important step towards a quantitative evaluation of NSC. It provides a framework that lets us learn the generative model directly from neuronal population recordings, paving the way for an experimentally-informed understanding of probabilistic computational principles underlying perception and behavior.
Our environment is riddled with sensory stimuli that are noisy, ambiguous, and often incomplete, necessitating organisms to handle uncertainty in their sensory observations. Bayesian models of
perception and behavior have thus grown in prominence, successfully accounting for an extensive array of tasks across perception [1, 2], cognition [3], sensory-motor learning [4] and decision making [5-8]. These models posit that the brain maintains a statistical generative model of the world, where sensory observations  $\mathbf{x}$  are generated from unknown, world-state variable  $\mathbf{z}$ . Upon encountering a stimulus  $\mathbf{x}$ , perception in the brain is conceptualized as probabilistically inferring the world-state variable  $\mathbf{z}$  that caused  $\mathbf{x}$ . In other words, to perceive  $\mathbf{x}$ , the brain would invert the generative model to compute the posterior distribution over  $\mathbf{z}$ :  $p(\mathbf{z}|\mathbf{x})$ . One can express the posterior via Bayes' rule as:  $p(\mathbf{z}|\mathbf{x}) \propto p(\mathbf{x}|\mathbf{z}) p(\mathbf{z})$ , where  $p(\mathbf{z})$  is the prior distribution of  $\mathbf{z}$  and  $p(\mathbf{x}|\mathbf{z})$  is the conditional distribution characterizing how well a given  $\mathbf{z}$  describes  $\mathbf{x}$ . While this has been an influential framework, the neuronal underpinnings of probabilistic inference remain challenging to conceptualize and test experimentally. To this end, the Neural Sampling Code (NSC) [9-16] is a prominent theory that offers a unique link between neuronal responses and probabilistic inference. Specifically, NSC posits that neuronal responses,  $\mathbf{r}$ , to a given stimulus,  $\mathbf{x}$ , can be thought of as samples drawn from the posterior distribution:  $\mathbf{r} \sim p(\mathbf{z}|\mathbf{x})$  (Figure 1).
Background and related work Prevailing literature on NSC uses simple and restrictive generative models and performs qualitative comparisons of model predictions with neurophysiological data to test the theory. Notably, existing NSC works use simple prior- and conditional distributions with pre-specified parameters. For example, a popular choice for the conditional distribution of images (stimuli) has been Gaussian with a likelihood function that linearly combines pre-specified filters. Hoyer and Hyvarinen [9] learn these filters via independent component analysis on natural images, whereas Haefner, Berkes, and Fiser [12] use oriented Gabor filters instead. Similarly, a popular choice for the prior is the exponential distribution with a pre-specified rate parameter [9]. These choices are inspired by (a) what is already known about sensory neurons, especially in the primary visual cortex (V1), and (b) the fact that it renders posterior computation mathematically simpler. In the examples above, the choice of filters reflects well-known findings that the receptive fields of V1 neurons resemble (Gabor-like) orientation filters [17-20], and the exponential prior is motivated by the principle of sparse coding [9, 20]. Importantly, these parameters and distributions — and thereby, the generative models — are not informed or learned explicitly from neurophysiological data. Rather, these works typically sample from the posterior of the assumed generative model in response to strongly parameterized stimuli (e.g., noisy oriented gratings). The models — and thereby the theory — are then evaluated based on how well the samples qualitatively capture specific neurophysiological phenomena such as the mean-variance relationship [9, 21, 22], task-induced noise correlation structures [12], and contextual modulation in V1 neurons [16].
In contrast, recent advances in deep learning-based neural system identification models have set new standards in providing expressive models that can faithfully predict neural population responses to natural stimuli [23-39], and offer experimentally verifiable insights at the single-neuron level [31, 40-42]. Additionally, advances in generative modeling, especially of images, have clearly demonstrated the effectiveness of deep, highly nonlinear, generative models such as auto-regressive models [43, 44], variational autoencoders [45-47], normalizing flows [48-51], and diffusion models [52-54]. Given the complexity of high-dimensional natural stimuli and real-world tasks, it is paramount that NSC be considered under a generative model that can match such complexity.
Our objective and contributions Here we ask: what exactly is the brain's generative model? More specifically, can we identify the brain's generative model from NSC population responses to natural stimuli? Although simple generative models and qualitative evaluations in the NSC literature have offered us great insight into the potential generative models of the brain and engendered support for the theory, there remains a conspicuous gap in the quantitative evaluation of NSC, particularly in response to natural stimuli. In this work, we bridge this gap by proposing a formalization of NSC that 1 allows us to directly fit NSC generative models to recorded neuronal activity in response to natural images, 2 formulate richer and more flexible generative models, and 3 employ standard metrics such as log-likelihood and single trial correlation to quantitatively evaluate different generative models under NSC. As opposed to specifying a generative model that ought to be maintained by the brain, our framework allows us to learn the generative model directly from neurophysiological data. Learning expressive generative models in a data-driven fashion additionally lets us take advantage of population recordings of large and ever-increasing scale in the field [55-57]. Furthermore, our formalization 4 lets us derive a stimulus-conditioned predictive model of neuronal responses from the trained generative model, which can be directly compared to state-of-the-art system identification
Figure 1: Conceptualizing NSC. A. Latent variable model of the world (stimulus):  $\mathbf{z}$  is the world state variable (intensity of oriented Gabor filter here; figure inspired from Orbán et al. [13]) and  $\mathbf{x}$  is the observed sensory stimulus (e.g., an image of a tiger). B. Responses  $\mathbf{r}$  under NSC: As the brain encounters a stimulus  $\mathbf{x}$ , it inverts its generative model, combining the likelihood  $p(\mathbf{x}|\mathbf{z})$  and prior  $p(\mathbf{z})$ , to obtain the posterior  $p(\mathbf{z}|\mathbf{x})$ , and  $\mathbf{r}$  are samples from the posterior. C. Neural response distribution under NSC: Each point corresponds to a single response pair of two NSC neurons under three distinct stimuli depicted by distinct colors. The distribution of neurons matches the distribution over the corresponding latent variables  $\mathbf{z}$ .
models. The predictive model has the ability to provide experimentally-verifiable, neuron-specific predictions from the normative theory.
We demonstrate our approach by fitting classical generative models from NSC literature and flexible deep learning-based generative models on macaque primary visual cortex (V1) population responses to natural images. We show that the flexible models outperform classical models in both their generative- and predictive-model performance. Overall, this work presents an important step towards a quantitative evaluation of NSC, paving the way for a data-driven approach in learning the generative model of the brain.
2 Fitting the Neural Sampling Code
2.1 Theory
An explicit formalization of NSC We begin by formalizing NSC as a latent variable probabilistic model  $\mathbf{z} \longrightarrow \mathbf{x} \longrightarrow \mathbf{r}$ , where  $\mathbf{z}$  represents the world state variable that underlies the observable stimulus  $\mathbf{x}$  (Figure 1A). Subsequently, the stimulus  $\mathbf{x}$  gives rise to the neuronal responses  $\mathbf{r}$  via the posterior  $p(\mathbf{z}|\mathbf{x})$  (Figure 1B). NSC posits that the neuronal responses  $\mathbf{r}$  elicited by stimulus  $\mathbf{x}$  can be interpreted as stochastic samples from the posterior distribution  $p(\mathbf{z}|\mathbf{x})$  (Figure 1B, C). However, the exact relation between  $\mathbf{z}$  and  $\mathbf{r}$  is often left unspecified. For instance, it is not clear what aspect of the neuronal response (e.g., firing rate, presence or absence of spikes, or membrane potential) should be treated as a sample. In fact, most previous works do not make a distinction between  $\mathbf{r}$  and  $\mathbf{z}$ , and simply equate an aspect of the neuronal response such as firing rate with the latent sample. Here, we make this assumption explicit and treat the neural response  $\mathbf{r}$  as a random variable that matches the latent random variable  $\mathbf{z}$  in stimulus-conditioned distributions:
$$
\mathbf {z} _ {\text {s a m p l e}} \sim p (\mathbf {z} | \mathbf {x}) \tag {1}
$$
$$
\mathbf {r} = \mathbf {z} _ {\text {s a m p l e}}, \tag {2}
$$
Equation 2 is a slight abuse of notation to state the equivalence in the stimulus-conditioned distributions of  $\mathbf{r}$  and  $\mathbf{z}$ , more formally stated as:
$$
p (\mathbf {r} | \mathbf {x}) \stackrel {\mathrm {d}} {=} p (\mathbf {z} | \mathbf {x}), \tag {3}
$$
where  $\stackrel{\mathrm{d}}{=}^{}$  denotes equality in distribution or density function. By marginalizing the stimulus, we find that the marginal distribution of  $\mathbf{r}$  must also match that of  $\mathbf{z}$ :
$$
p (\mathbf {r}) = \int p (\mathbf {r} | \mathbf {x}) p (\mathbf {x}) \mathrm {d} \mathbf {x} \stackrel {\mathrm {d}} {=} \int p (\mathbf {z} | \mathbf {x}) p (\mathbf {x}) \mathrm {d} \mathbf {x} = p (\mathbf {z}). \tag {4}
$$
Explicitly formalizing NSC with distinct  $\mathbf{r}$  and  $\mathbf{z}$  has two advantages. Firstly, the resulting formulation provides the crucial link between the generative model  $\mathbf{z} \rightarrow \mathbf{x}$  and observed responses  $\mathbf{r}$ , serving as the basis for learning the generative model from the responses. This also provides the basis for a neuron-specific model comparison between different NSC models as well as the possibility to make predictions for specific neurons that can be experimentally tested. Secondly, the explicit link highlights the possibility to explore more flexible mappings between  $\mathbf{z}$  and  $\mathbf{r}$ . For instance, one could assume that the latent variable  $\mathbf{z}$  is encoded in the membrane potential, but what we observe are spike counts, i.e.  $\mathbf{r} = f(\mathbf{z})$  for some stochastic mapping  $f$ . This relation can, in turn, become part of the model, which can then be fitted to real data and compared to alternative versions of the model. In this work, we choose to learn the generative models from the data under the simplest mapping of  $\mathbf{r} \equiv \mathbf{z}$  (Section 3.2). Please see Section 4 for a discussion on alternative mappings between  $\mathbf{r}$  and  $\mathbf{z}$ .
In NSC, we note that the latent variables are what underlie the stimulus (such as the intensity of an oriented filter in [1A]) and are not necessarily any task-relevant experimenter-defined variables (such as orientation in an orientation-discrimination task). This is in contrast to the alternative theory of probabilistic population codes [58, 59], where typically, the latent variable is explicitly defined to be the task-relevant experimenter-defined variables.
Learning the generative model under NSC One way to quantitatively test an NSC generative model  $p(\mathbf{z},\mathbf{x})$  is testing how well the response distribution  $p(\mathbf{r}|\mathbf{x})$  approximates the posterior  $p(\mathbf{z}|\mathbf{x})$  of the generative model, i.e., testing equation 3. However in reality  $p(\mathbf{z},\mathbf{x})$ , and consequently  $p(\mathbf{z}|\mathbf{x})$  is unknown to us. However, our formalization allows us to learn the generative model  $p(\mathbf{z},\mathbf{x})$  via learning the joint distribution  $p(\mathbf{r},\mathbf{x})$ . The equivalence between  $p(\mathbf{z},\mathbf{x})$  and  $p(\mathbf{r},\mathbf{x})$  follows from equations 2 and 4:
$$
\begin{array}{l} p (\mathbf {z}, \mathbf {x}) = p (\mathbf {z} | \mathbf {x}) p (\mathbf {x}) \stackrel {\mathrm {d}} {=} p (\mathbf {r} | \mathbf {x}) p (\mathbf {x}) = p (\mathbf {r}, \mathbf {x}) (5) \\ = p (\mathbf {x} | \mathbf {z}) p (\mathbf {z}) \stackrel {\mathrm {d}} {=} p (\mathbf {x} | \mathbf {r}) p (\mathbf {r}) = p (\mathbf {r}, \mathbf {x}) (6) \\ \end{array}
$$
The joint distribution can then simply be fitted to recorded stimulus-response pairs  $\{\mathbf{x}^{(i)},\mathbf{r}^{(i)}\}_{i = 1}^{N}$  by maximizing the likelihood:
$$
\theta^ {*} = \arg \max  _ {\theta} \sum_ {i = 1} ^ {N} \log \underbrace {p \left(\mathbf {x} ^ {(i)} , \mathbf {r} ^ {(i)} ; \theta\right)} _ {\text {J o i n t}} = \underset {\theta_ {L}, \theta_ {P}} {\arg \max } \sum_ {i = 1} ^ {N} \log \underbrace {p \left(\mathbf {x} ^ {(i)} \mid \mathbf {r} ^ {(i)} ; \theta_ {L}\right)} _ {\text {L i k e l i h o o d}} + \log \underbrace {p \left(\mathbf {r} ^ {(i)} ; \theta_ {P}\right)} _ {\text {P r i o r}} \tag {7}
$$
where  $\theta$  are the parameters of the generative model, that we split into  $\theta_{L}$  and  $\theta_{P}$  for parameters relevant to the likelihood and prior, respectively. Provided that  $\theta_{L}$  and  $\theta_{P}$  do not overlap, the generative models can be learned by learning the likelihood and prior separately.
Evaluating NSC on data Fitting generative models under NSC on recorded data allows us to compare the generative models quantitatively by evaluating their performances as log-likelihood on a held-out test set. Furthermore, once we have learned the generative model  $p(\mathbf{r},\mathbf{x};\theta^{*})$ , we can invert it to arrive at the posterior  $p(\mathbf{r}|\mathbf{x};\theta^{*})$ . This provides a neuronal encoding model  $p(\mathbf{r}|\mathbf{x})$  under specific assumptions of the NSC model, allowing us to predict neural responses to arbitrary new stimuli. The performance of this predictive model can serve as yet another metric for quantitative model comparison under NSC. Additionally, the posterior allows us to compare the generative models to the normative-theory-free system identification models. It is important to note that our quantification does not make any assumption about the kind of stimuli  $\mathbf{x}$ . Existing works on NSC use parametric stimuli from classical neuroscience experiments and perform a qualitative comparison between model- and real-neuronal responses to the same stimuli. Our formulation, on the other hand, allows us to compare different NSC models on natural images, the type of stimuli the visual system has evolved to process.
2.2 Models
Following previous work in NSC [9-16], here we focus on vision and develop generative models under NSC for natural image stimuli  $\mathbf{x}$  and spike counts  $\mathbf{r}$  recorded from the visual cortex (Figure 2A). Developing generative models entails developing models for the prior  $p(\mathbf{r})$  and the likelihood  $p(\mathbf{x}|\mathbf{r})$  (Figure 2B). Additionally, we also fit an approximate posterior  $q(\mathbf{r}|\mathbf{x})$ . We summarize our fitting methodology in an algorithm towards the end of the section (Algorithm 1).
B
Figure 2: A. Overview of the experimental setup for neuronal recordings (see Section 3.2 for a summary of the data description and Cadena et al. [29] for the complete details). B. Parameterized generative model for NSC:  $\theta_{P}$ : parameter of the prior  $p(\mathbf{r};\theta_P)$ ;  $\theta_L$ : parameter of the likelihood  $p(\mathbf{x}|\mathbf{z};\theta_L)$ ;  $\phi$ : parameters of the (approximate) posterior  $q(\mathbf{r}|\mathbf{x};\phi)$ . C. Flexible prior, likelihood, and posterior models. The prior follows our dequantization framework consisting of 3 components: (1) continuous prior  $p(\zeta ;\omega)$  (normalizing flow with Gaussian base distribution), (2) quantizer  $P(\mathbf{r}|\zeta)$  (floor function), and (3) variational dequantizer  $q(\zeta |\mathbf{r};\nu)$  (a conditional normalizing flow, Appendix B). The likelihood is an isotropic Gaussian distribution over  $\mathbf{x}$ , where the parameters are functions (MLP) of  $\mathbf{r}$ . The posterior is a Gamma distribution over  $\mathbf{r}$  whose parameters are functions of  $\mathbf{x}$ , modeled as a system identification-based convolutional neural-network model (Appendix C).
Prior Spike counts  $\mathbf{r}$  are discrete. Hence, we can neither directly fit standard literature models, such as an exponential [9] or a Laplace distribution [20] to the discrete variable  $\mathbf{r}$ , nor can we straightforwardly fit more common flexible density models such as normalizing flows [49, 60], as they are all continuous density models. A common approach to remedy this is to employ uniform dequantization, where the discrete quantity is converted into a continuous signal by adding uniform random noise [43, 61, 62]. We adopt the more general approach of variational dequantization where the noise distribution is learned instead of being fixed to be uniform [63-66]. In this method, prior distribution over discrete  $\mathbf{r}$  is captured by positing a generative model involving a continuous latent variable  $\zeta$  linked to the discrete response  $\mathbf{r}$  via a deterministic quantizer function:  $P(\mathbf{r}) = \int P(\mathbf{r}|\zeta)p(\zeta)d\zeta$ , where  $P(\mathbf{r}|\zeta)$  is the quantizer and  $p(\zeta)$  is the continuous prior.
Since the integral is usually intractable, the whole model is fit by optimizing the evidence lower bound (ELBO):
$$
\log P (\mathbf {r}) \geq \mathbb {E} _ {\zeta \sim q (\zeta | \mathbf {r}; \nu)} \left[ \overbrace {\log p (\zeta ; \omega)} ^ {\text {C o n t i n u o u s p r i o r}} \right] + \mathbb {H} \left(\overbrace {q (\zeta | \mathbf {r} ; \nu)} ^ {\text {D e q u a n t i z e r}}\right) \tag {8}
$$
where  $q(\zeta|\mathbf{r};\nu)$  is the approximate posterior distribution with parameters  $\nu, p(\zeta;\omega)$  is the continuous prior with parameters  $\omega$ , and  $\mathbb{H}(q(\zeta|\mathbf{r};\nu)) = -\mathbb{E}_{\zeta \sim q(\zeta|\mathbf{r};\nu)}[\log q(\zeta|\mathbf{r};\nu)]$  is the conditional entropy of the dequantizing distribution. Note that Equation (8) only provides a lower-bound to  $\log P(\mathbf{r})$ , and a tighter bound via importance-weighted sampling [66-68] (Appendix A).
In our work, we only consider factorized prior distributions, i.e., we treat neurons to be a priori independent  $\log P(\mathbf{r}) = \sum_{i}\log P(\mathbf{r}_{i})$ . This choice is informed by both the nature of the V1 neural data that showed limited correlation across all stimuli and the simplicity of the fit it provides. The same independence assumption was applied for the continuous prior distribution over the dequantized responses  $\zeta$ . Given this dequantizer framework, we explore three different NSC priors by varying the distribution over the continuous latent  $p(\zeta ;\nu)$ :
1. Exponential (Exp),  $\frac{1}{\lambda}\exp \frac{-\zeta}{\lambda} H(\zeta)$ , as found in the original NSC model by Hoyer & Hyvarinen [9].
2. Half-normal (HN),  $\frac{\sqrt{2}}{\sigma\sqrt{\pi}}\exp \left(-\frac{\zeta^2}{2\sigma^2}\right)H(\zeta)$ , where  $H(\zeta)$  is the heavyside function.
3. Normalizing flow (Flow):  $p(\zeta; \omega) = p_{\mathrm{base}}(T^{-1}(\zeta; \omega)) \cdot |\frac{\partial T^{-1}(\zeta; \omega)}{\partial \zeta}|$ , where we choose  $p_{\mathrm{base}}$  to be a standard normal, and  $T^{-1}$  represents the following series of invertible mappings with learnable parameters  $\omega$ : [affine, tanh, affine, tanh, affine, tanh, affine, softplus  $^{-1}$ ], where softplus  $^{-1}(y) = \log(e^{y} - 1)$ , affine  $y = ay + b$  with learnable parameters  $a$  and  $b$ . softplus  $^{-1}$  ensures that the support of  $\zeta$  is non-negative, since we are ultimately interested in modeling the distribution of (non-negative) spike counts (Figure 2C, "Prior" sub-panel).
For the dequantizer distribution,  $q(\zeta | \mathbf{r}; \nu)$ , we utilize a conditional normalizing flow-based flexible distribution, as in [66] (Appendix B).
Likelihood We model the likelihood as an isotropic Gaussian distribution:
$$
p \left(\mathbf {x} \mid \mathbf {r} ^ {(i)}\right) = \mathcal {N} \left(\mathbf {x} \mid \boldsymbol {\mu} ^ {(i)}, \boldsymbol {\sigma} ^ {2 (i)} \cdot \mathbf {I}\right), \tag {9}
$$
where the parameters mean,  $\pmb{\mu}^{(i)}\in \mathbb{R}^{|x|}$  and variance,  $\sigma^{(i)}\in \mathbb{R}_{>0}^{|\mathbf{x}|}$  are functions of response,  $\mathbf{r}^{(i)}$  and  $|\mathbf{x}|$  is the number of dimensions of  $\mathbf{x}$ . We consider (1) a linear function where  $\pmb{\mu} = w_{\pmb{\mu}}\mathbf{r}^{(i)} + b_{\pmb{\mu}}$  and  $\pmb{\sigma} = \exp^{w_{\sigma}\mathbf{r}^{(i)} + b_{\sigma}}(\mathbf{Lin})$  and (2) a nonlinear function  $\pmb{\mu} = w_{\pmb{\mu}}\mathrm{MLP}(\mathbf{r}^{(i)}) + b_{\pmb{\mu}}$  and  $\pmb{\sigma} = \exp^{w_{\sigma}\mathrm{MLP}(\mathbf{r}^{(i)}) + b_{\sigma}}$ , where  $\mathrm{MLP}(\cdot)$  is a neural network (MLP) (Figure 2C "Likelihood" sub-panel).