SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Posterior In many cases, the posterior distribution for a desired generative model is not analytically tractable and must be approximated, commonly using variational inference or Markov Chain Monte Carlo sampling [69-71]. Here, since we learn the generative model  $p(\mathbf{x}, \mathbf{r}; \theta^{*})$ , we can approximate the true posterior  $p(\mathbf{r}|\mathbf{x}; \theta^{*})$  by fitting a model posterior  $q(\mathbf{r}|\mathbf{x}; \phi)$  to samples from  $p(\mathbf{x}, \mathbf{r}; \theta^{*})$  directly via maximum log-likelihood:
$$
\phi^ {*} = \arg \max  _ {\phi} \sum_ {\mathbf {x} ^ {\prime}, \mathbf {r} ^ {\prime}} \log q \left(\mathbf {r} ^ {\prime} \mid \mathbf {x} ^ {\prime}; \phi\right), \tag {10}
$$
where  $\mathbf{x}^{\prime},\mathbf{r}^{\prime}\sim p(\mathbf{x},\mathbf{r};\boldsymbol{\theta}^{*})$ , are samples from the trained generative model. We model the posterior distribution of responses conditioned on images as a factorized Gamma distribution, following state-of-the-art (SOTA) work in system identification [72]:  $p(\mathbf{r}|\mathbf{x}^{(i)}) = \prod_{j = 1}^{S}p_{\Gamma}\left(\mathbf{r}_{j}|\boldsymbol{\alpha}^{(i)},\boldsymbol{\beta}^{(i)}\right)$ , where  $\mathbf{x}^{(i)}$  is the  $i$ th image,  $\mathbf{r}_{j}$  is the  $j$ th neuron out of  $|\mathbf{r}| = S$  total neurons, and the parameters concentration,  $\boldsymbol{\alpha}^{(i)}$  and rate,  $\boldsymbol{\beta}^{(i)}$  are functions of the image,  $\mathbf{x}^{(i)}$ . Since these functions map an image to response distribution parameters, we model them using a convolutional neural network model (Figure 2C "Posterior" sub-panel), following SOTA system identification work [25, 27, 73] (Appendix C).
Prior	Likelihood	Name
Exponential (λ = 1)	Linear	Exp1-Lin
Exponential	Linear	Exp-Lin
Half-Normal	Linear	HN-Lin
Normalizing Flow	Linear	Flow-Lin
Exponential	MLP	Exp-MLP
Half-Normal	MLP	HN-MLP
Normalizing Flow	MLP	Flow-MLP
Table 1: Generative models that we fit as being composed of priors and likelihoods.
Algorithm 1 Learning NSC models from data
Require:  $N$  pairs of stimuli and neuronal responses respectively,  $\{\mathbf{x}^{(i)},\mathbf{r}^{(i)}\}_{i}^{N}$
Learning generative model  $p(\mathbf{x},\mathbf{r};\theta_P,\theta_L)$
1:  $\theta_P^* \gets \arg \max_{\theta} \sum_{i=1}^{N} \log p(\mathbf{r}^{(i)}; \theta_P)$
2:  $\theta_L^*\gets \arg \max_\theta \sum_{i = 1}^N\log p\left(\mathbf{x}^{(i)}|\mathbf{r}^{(i)};\theta_L\right)$
Learning approx posterior model  $q(\mathbf{r}|\mathbf{x};\phi)$
3: Sample  $\{\mathbf{x}^{\prime (i)},\mathbf{r}^{\prime (i)}\}_{i}^{S}\sim p(\mathbf{x},\mathbf{r};\theta_{P}^{*},\theta_{L}^{*})$
4:  $\phi^{*}\gets \arg \max_{\phi}\sum_{i}^{S}\log q\left(\mathbf{r}^{\prime (i)}|\mathbf{x}^{\prime (i)};\phi\right)$
3.1 Synthetic data
We simulated 10,000 pairs of images and neuronal responses from the following three classical NSC models: ① a Hoyer & Hyvarinen model (HNH) with an exponential prior [9], ② an Olshausen & Field (ONF) model where the prior is a Laplace distribution [20], and ③ a full Gaussian model (Gauss) where the prior is an isotropic Gaussian with mean 0 and variance  $\sigma_r^2$ . All the three models share a common linear, isotropic Gaussian likelihood  $p(\mathbf{x}|\mathbf{r}) = \mathcal{N}(\mathbf{x}|A\mathbf{r},\sigma^2\mathbf{I})$ , where  $A$  is the factor loading matrix learned via standard independent component analysis model (ICA) with a complete basis on natural image patches [9,74]. Additionally, we sampled image-response pairs from ④ our flexible model with Flow prior (described in Section 2.2) and MLP-based likelihood (Section 2.2), where all parameters were randomly initialized. For any given generative model, we first sample neuronal responses from the prior via  $\mathbf{r}^{(i)} \sim p(\mathbf{r})$ , and then sample corresponding images via
$\mathbf{x}^{(i)} \sim p(\mathbf{x}|\mathbf{r}^{(i)})$ , where  $i \in 1, \dots, 10,000$ . We hold out a set of 1,000 pairs as the test set. We fitted all models on the datasets simulated from the classical as well as the flexible models via Equation (7) and computed joint log-likelihoods of the trained models on the held-out test set as  $\log p(\mathbf{x}^{(i)}, \mathbf{r}^{(i)}) = \log p(\mathbf{x}^{(i)}|\mathbf{r}^{(i)}) + \log p(\mathbf{r}^{(i)})$ . For the classical models, maximum likelihood estimates of the parameters were obtained analytically (Appendix F). We trained the flexible model using gradient descent.
We find that (1) the flexible model fits responses and images simulated under other NSC models well, i.e., learns  $p(\mathbf{r})$  and  $p(\mathbf{x}|\mathbf{r})$  and closely approximates the log-likelihood of the true models. Importantly, it outperforms the fit of other NSC models with mismatched generative distributions, consistently being the best model after the ground-truth model (first 3 columns in Figure 3). Furthermore, the flexible model is capable of generating complex image and response distributions that could not be easily captured by the classical generative models (column 4 in Figure 3). This demonstrates that our framework allows for NSC model fitting and that the flexible model has the ability to flexibly capture the data distribution across a wide range of generative models. Critically, a flexible model could fit complex generative models that cannot be modeled well by other classical models.
3.2 Neurophysiological data
Data description Next, we demonstrate applying our approach to real neuronal data. We used 32-channel laminar
NeuroNexus arrays (Figure 2A) to record population activity from the primary visual cortex (V1) of two awake male rhesus macaque monkeys (Macaca mulatta) [29] as they fixated on grayscale natural images sampled from the ImageNet dataset [75]. All the experiments concerned with the recordings adhered to the National Institutes of Health, United States guidelines, and received approval from the Institutional Animal Care and Use Committee. Each image was presented for  $120~\mathrm{ms}$ , and spike counts between  $40~\mathrm{ms}$  and  $160~\mathrm{ms}$  after the image onset were computed and used as the neuronal response  $\mathbf{r}$ . The image stimulus  $\mathbf{x}$  used for modeling is  $41\times 41$  px. For more details on the experiments and data collection, refer to Cadena et al. [29]. We collected data across 12 recording sessions, each having approximately 16,000 image-response pairs and at least 16 well-isolated single units. We split the dataset into approximately 10,000 pairs for training, 3,000 pairs for validation, and 3,000 for testing (for exact details on all sessions, see Appendix D). We do not aggregate data across sessions and fit models separately for each session since the images can differ from session to session, and not all neurons have seen every image.
Fitting the generative model Given a dataset of images and responses,  $\{\mathbf{x},\mathbf{r}\}_{i = 0}^{N}$ , we fit the likelihood  $p(\mathbf{x}|\mathbf{r};\theta_L)$  on the image-response pairs and the prior  $p(\mathbf{r};\theta_P)$  on the responses,  $\mathbf{r}$  as in Equation (7). We fit all of the generative models on each recording session as following procedure described in Table 1. Below, we describe our results for the session with the largest number of neurons (29 well-isolated single units) in detail and report summary results on all sessions.
For the prior models (Figure 4A), we report the test-set log-likelihood performance of all models (Exp, HN, Flow) relative to the Exp1-model as the baseline. We find that our flexible normalizing flow model (Flow) achieves the best performance, improving the score from the exponential distribution (Exp) by 0.095 bits per neuron per trial, amounting to 2.755 bits across 29 neurons per trial. For the likelihood models (Figure 4B), we find that using an MLP likelihood function, the model improved by 0.052 bits per pixel per trial, amounting to 87.19 bits of improvement across all  $41 \times 41$  pixels per trial, relative to the model with linear likelihood function. For the joint distribution (Figure 4C), we find that the flexible model (Flow-MLP) achieves the highest log-likelihood score, offering an improvement of  $1.8452\mathrm{e} - 3$  bits per pixel per neuron per trial, amounting to 89.951 bits across all  $41 \times 41$  pixels and 29 neurons per trial. We observed that in each of the cases, flexible models (Flow prior, MLP likelihood) offer much higher log-likelihood performance, with the same trend found across all sessions (Appendix D).
Figure 3:  $\log p(\mathbf{x},\mathbf{r})$  of models on simulated data (trial averaged, in bits). Column denotes the model generating the samples (data) and rows the trained NSC model. Since the exponential prior in HNH has a non-negative support and does not match that of ONF and Gauss, the scores for HNH under ONF and Gauss data are unavailable.
c
Figure 4: Log-likelihood scores (in bits) of generative models on population recordings (test set) as fit on recording session with the highest number of neurons ( $n = 29$ ). Error-bars denote the standard error of mean across trials. A. Prior models,  $p(\mathbf{r})$ : log-likelihood (lower bound) relative to the baseline Exp1 prior model. The score is averaged across neurons and trials. Note that for the prior models on discrete spike counts,  $\mathbf{r}$ , we can only obtain a lower bound on  $p(\mathbf{r})$ . Here we show the importance-sampling bound (Equation (11)) with 1000 samples. B. Likelihood models  $p(\mathbf{x}|\mathbf{r})$ : absolute log-likelihood of likelihood functions, averaged across image pixels and trials. C. Joint models  $p(\mathbf{x},\mathbf{r})$ : log-likelihoods relative to the baseline Exp1-Lin generative model. The score is averaged across pixels, neurons and trials.
Figure 5: Posterior performance of NSC models, along with system identification model ("Sysident"). Error bars represent the standard error of mean over neuronal responses. All metrics are averaged over number of neurons and trials. A. The lower bound of log-likelihood in bits, for session with 29 neurons. We compute the lower-bound since we are evaluating the Gamma-posterior on (discrete) spike counts, and full-likelihood is intractable (Appendix G). B. Same as A but across all 12 sessions (purple: average, gray: single session). C. Single-trial correlation, for session with 29 neurons. D. Same as C across all 12 sessions
Learning the posterior distribution For each of the trained generative models,  $p(\mathbf{x},\mathbf{r};\theta^{*})$  we approximated the model's posterior distribution  $p(\mathbf{r}|\mathbf{x};\theta^{*})$  using an approximate posterior  $q(\mathbf{r}|\mathbf{x};\phi)$  trained on samples drawn from the trained generative model (Equation (10), Algorithm 1). We evaluated the posterior distribution for each generative model by computing their mean log-likelihood on real neuronal responses conditioned on real images from the test set (Figure 5A, B). We also computed a single-trial correlation between the mean of the learned posterior distributions and neuronal responses (Figure 5C, D). Finally, we compared the posterior distribution to a deep system identification model.
We find that, in general, a more flexible trained generative model tends to yield a higher posterior predictive performance. Based on the log-likelihood evaluation, our flexible generative model (Flex-MLP) gained as much as 1.39 bits per neuron per trial compared to the baseline Exp1-Lin model and 0.61 bits per neuron per trial compared to Exp-Lin model (the Hoyer & Hyvarinen model [9]). In terms of single-trial correlation performance, our flexible generative model (Flex-MLP) achieved  $10\%$  higher correlation compared to the Exp1-Lin baseline and  $8\%$  higher correlation compared to Exp-Lin. When averaged across all sessions, we find that Flow-Lin performs best, almost on
par with the Flow-MLP, which achieves 0.019 bits per neuron per trial less. Furthermore, a system identification model trained on the dataset of real neuronal responses performed better than the best NSC generative model, gaining 0.17 bits per neuron per trial and  $17\%$  higher correlation per neuron per trial compared to Flex-MLP.
All model training was performed using backpropagation and gradient descent and we provide training, compute and infrastructure details in Appendix E.
The main focus of this work was to develop a way to answer the question, "How well does NSC explain neurophysiological data quantitatively?". While NSC is a prominent normative theory for probabilistic computation in the brain, and the literature has provided much qualitative insight, our work is the first to offer a quantitative paradigm for empirically testing it using brain responses to ecological, natural stimuli. Our framework additionally lets us formulate more flexible generative models — which can be better informed by the data — and employ standard metrics such as log-likelihood to quantitatively evaluate alternative generative models under NSC. Furthermore, inverting the learned generative model has allowed us to obtain the posterior distribution, which is equivalently a neuronal response predictive model. Importantly, this let us compare NSC models to models outside of NSC's theoretical framework, such as system identification models, allowing us to benchmark the predictive performance of NSC models. Our results demonstrated that the flexible generative models outperformed classical models in terms of both generative and predictive model performance, yet system identification models achieve superior response-predictive performance compared to even our best generative models. We now discuss some limitations of our current study and discuss a number of open questions and imminent future directions.
Limitations I: Assumption of strict 1:1 neuron-latent mapping: One limitation in our current study is that we only use a 1:1 identity mapping between the activity of neurons and latent variables in our formulation of NSC. Abiding by this restriction could limit the capacity of the NSC models, especially considering some existing work in NSC that have qualitatively explored more flexible mappings. For example, Orbán et al. [13] model membrane potential values (responses) as a nonlinear function of posterior samples. Furthermore, Savin and Denève [76] map responses of  $N$  neurons to  $D$  latent variables where  $N > D$ . Many more ways of how  $\mathbf{r}$  and  $\mathbf{z}$  relate are conceivable. However, our formulation with a separate  $\mathbf{r}$  and  $\mathbf{z}$  allows us to, in principle, incorporate different mappings and learn the corresponding generative models. Since the focus of this study was on the aspect of fitting NSC models to data, we chose the simplest (original) interpretation of NSC where  $\mathbf{r} \equiv \mathbf{z}$ .
Limitations II: Definition of a "sample" as total spike counts: We defined a "sample" as the total spike count of neuronal activity within a specific time window following the stimulus, which is not necessarily what literature works do. However, to our knowledge, there is no generally agreed upon or rigorous definition of a "sample" in NSC. While NSC was originally motivated with firing rate/spike counts over a 500ms window as the sample [9,21], many alternative definitions such as membrane potential over 10ms [13] have been employed. It is unclear on what generally applicable metric — other than goodness of fit to data — such a definition could be evaluated. This in fact served to us as another motivating factor for striving towards a data-driven evaluation of sampling models that would allow one to compare such choices in an informed manner. In this work, we chose the total spike count as the working definition.
Limitations III: Better generative models are needed: Advances in deep learning architectures, latent variable models, and transfer learning have greatly enhanced the capabilities of generative models in machine learning. We believe the models we chose, although more expressive than classical models, are still limiting, especially considering that our likelihood  $p(\mathbf{x}|\mathbf{r})$  uses linear or MLP decoding from neurons to images, with a simple Gaussian noise model. To capture the rich and complex nature of neuronal representations of natural images, we believe it is necessary to consider more sophisticated generative models, that even incorporate a natural image prior, that would eventually close the gap in predictive performance between system identification performance and NSC generative models. Furthermore, an important avenue of research is identifying biological mechanisms that underlie NSC (i.e. sampling from the posterior) [14, 77-82]. It is worth noting that our deep learning-based generative models are not meant to be mechanistic models of NSC neurons.
Rather, we believe that our approach lays the foundation for alternative biologically plausible models to be quantitatively evaluated and compared.
Why do system identification models perform better than NSC generative models? System identification models are directly trained discriminatively, i.e.,  $\min_{\theta} \mathbb{E}_{\mathbf{x}}[D_{\mathrm{KL}}(p_{\mathrm{true}}(\mathbf{r}|\mathbf{x}) || p_{\mathrm{model}}(\mathbf{r}|\mathbf{x};\theta))]$ , to predict neuronal responses to natural images and deep-learning based ones are currently SOTA. There is still much room to build better generative models that would better explain the data (see Limitations III). However, for a given dataset of responses to stimuli from a fixed stimulus distribution, we do not expect the posterior of even the ideal generative model to surpass the performance of the ideal system identification model because the generative model training, i.e.,  $\min_{\theta} \{\mathbb{E}_{\mathbf{x}}[D_{\mathrm{KL}}(p_{\mathrm{true}}(\mathbf{r}|\mathbf{x}) || p_{\mathrm{model}}(\mathbf{r}|\mathbf{x};\theta))] + D_{\mathrm{KL}}(p_{\mathrm{true}}(\mathbf{x}) || p_{\mathrm{model}}(\mathbf{x};\theta))\}$ , does not provide any advantage over system identification in response prediction unless some specific inductive biases are introduced in the generative model.
Why bother fitting NSC models if they fail to quantitatively compete with system identification?
If we change the stimulus distribution  $p(\mathbf{x})$  to  $p_{\mathrm{new}}(\mathbf{x})$  with markedly different stimulus statistics and let the sensory neurons adapt to  $p_{\mathrm{new}}(\mathbf{x})$ , we would expect the system identification model's performance to drop on  $p_{\mathrm{new}}(\mathbf{x})$ . The system identification model might have to be retrained on a new dataset of responses under  $p_{\mathrm{new}}(\mathbf{x})$ . This is the case where we would expect the NSC's learned generative model to be beneficial. Specifically, change in  $p(\mathbf{x})$  to  $p_{\mathrm{new}}(\mathbf{x})$  may entirely derive from the change in prior  $p(\mathbf{z})$  to  $p_{\mathrm{new}}(\mathbf{z})$ , while  $p(\mathbf{x}|\mathbf{z})$  remains fixed. Hypothetically, this is since  $p(\mathbf{x}|\mathbf{z})$  represents the invariant "physical" process by which the latents (e.g., the identity of an animal) give rise to observations (e.g., the appearance of the animal). Consequently, if NSC accurately describes a neural population, i.e.,  $\mathbf{r} \sim p(\mathbf{z}|\mathbf{x}) \propto p(\mathbf{x}|\mathbf{z})p(\mathbf{z})$ , the neuronal adaptation can be accounted for by simply learning  $p_{\mathrm{new}}(\mathbf{z})$ , i.e.,  $\mathbf{r}_{\mathrm{new}} \sim p_{\mathrm{new}}(\mathbf{z}|\mathbf{x}) \propto p(\mathbf{x}|\mathbf{z})p_{\mathrm{new}}(\mathbf{z})$ , keeping  $p(\mathbf{x}|\mathbf{z})$  fixed. We believe such out-of-context generalization is a theoretical strength of NSC, and is a consequence of its normative nature (responses being "samples" from the posterior distribution). Such normative hypotheses are neither present in the purely phenomenological system identification models and nor is it straightforward to equip them with normative assumptions.
The above insight thus helps us identify potential future experiments to test NSC models utilizing our framework since it lets us learn the generative model  $p(\mathbf{x}|\mathbf{z})$ $p(\mathbf{z})$  via  $p(\mathbf{x}|\mathbf{r})$ $p(\mathbf{r})$  (NSC assumption). Namely, one could perform experiments in which we let the neural population adapt to different sensory contexts with expected shifts in  $p(\mathbf{z})$ . Using our NSC framework, we would expect to be able to predict how neuronal responses should change (as reflected in updated  $\mathbf{r}_{\mathrm{new}} \sim p_{\mathrm{new}}(\mathbf{z}|\mathbf{x})$ ) under new contexts.
Why have previous works not fit NSC models to data? We attribute the lack of such attempts to (1) limitations in data availability, (2) complexities involved in training flexible machine learning and inference algorithms on recorded data, and (3) the philosophical approach behind normative theories. Normative theories describe how a biological system ought to function in order to tackle fundamental tasks. They propose models with parameters that are optimized for those tasks, without relying on actual experimental data [83, 84]. Typically normative theories are evaluated using qualitative agreements between the proposed models and data. NSC is itself a normative theory. In contrast, phenomenological approaches such as system identification propose models whose parameters are directly learned from experimental data. Normative and phenomenological approaches have historically been developed independently of each other. Similar to Mlynarski et al. [83], who interpolate between phenomenological and normative models via maximum entropy priors, our approach allows us to get the best of both worlds: state-of-the-art deep learning-based system identification models from phenomenological approaches and the theoretical underpinnings of the normative NSC. System identification provides us with expressive models that faithfully model and predict the activity of thousands of neurons to rich natural stimuli. NSC, on the other hand, goes beyond what experimental data alone could offer by letting us hypothesize how neurons encode uncertainty about the stimulus, reflecting the posterior distribution over latent variables in a generative model of the world, thus allowing us make novel predictions such as about generalizability across stimulus contexts and design relevant experiments.
We thank all the reviewers for their valuable and constructive feedback. We additionally thank Jakob Macke, Xaq Pitkow, Ralf Haefner, Gergő Orbán, members of Sinz-, Walker-, Tolias-lab for helpful and stimulating discussions. SS and FHS are supported by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP 06, project number: 276693517. KKL is supported by German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ: 01IS18039A). KR and AST are supported by the National Eye Institute, National Institutes of Health (NIH), USA with award numbers R01 EY026927, and Core Grant for Vision Research with grant number T32-EY-002520-37. AST, KR and FHS are supported by the National Science Foundation Collaborative Research in Computational Neuroscience, USA with grant number IIS-2113173, Germany with FKZ: 01GQ2107. GHD is supported by The National Institute of Mental Health, NIH, USA with grant number T32MH015144. EYW is supported by the National Institute of Neurological Disorders and Stroke, NIH, USA with grant number 1U19NS107609-01.