Title: Taking the neural sampling code very seriously: A data-driven approach for evaluating generative models of the visual system
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Sampling Code

Probabilistic Inference

Bayesian Brain

Macaque V1

Natural Images

Population Recordings

Normalizing Flows

Probabilistic Models

Computational Neuroscience

Theoretical Neuroscience
Abstract: Prevailing theories of perception hypothesize that the brain implements perception via Bayesian inference in a generative model of the world.
One prominent theory, the Neural Sampling Code (NSC), posits that neuronal responses to a stimulus represent samples from the posterior distribution over latent world state variables that cause the stimulus.
Although theoretically elegant, NSC does not specify the exact form of the generative model or prescribe how to link the theory to recorded neuronal activity.
Previous works assume simple generative models and test their qualitative agreement with neurophysiological data.
Currently, there is no precise alignment of the normative theory with neuronal recordings, especially in response to natural stimuli, and a quantitative, experimental evaluation of models under NSC has been lacking.
Here, we propose a novel formalization of NSC, that (a) allows us to directly fit NSC generative models to recorded neuronal activity in response to natural images, (b) formulate richer and more flexible generative models, and (c) employ standard metrics to quantitatively evaluate different generative models under NSC.
Furthermore, we derive a stimulus-conditioned predictive model of neuronal responses from the trained generative model using our formalization that we compare to neural system identification models.
We demonstrate our approach by fitting and comparing classical- and flexible deep learning-based generative models on population recordings from the macaque primary visual cortex (V1) to natural images, and show that the flexible models outperform classical models in both their generative- and predictive-model performance.
Overall, our work is an important step towards a quantitative evaluation of NSC. 
It provides a framework that lets us \textit{learn} the generative model directly from neuronal population recordings, paving the way for an experimentally-informed understanding of probabilistic computational principles underlying perception and behavior.

Taking the neural sampling code very seriously: A data-driven approach for evaluating generative models of the visual system

Abstract

Prevailing theories of perception hypothesize that the brain implements perception via Bayesian inference in a generative model of the world. One prominent theory, the Neural Sampling Code (NSC), posits that neuronal responses to a stimulus represent samples from the posterior distribution over latent world state variables that cause the stimulus. Although theoretically elegant, NSC does not specify the exact form of the generative model or prescribe how to link the theory to recorded neuronal activity. Previous works assume simple generative models and test their qualitative agreement with neurophysiological data. Currently, there is no precise alignment of the normative theory with neuronal recordings, especially in response to natural stimuli, and a quantitative, experimental evaluation of models under NSC has been lacking. Here, we propose a novel formalization of NSC, that (a) allows us to directly fit NSC generative models to recorded neuronal activity in response to natural images, (b) formulate richer and more flexible generative models, and (c) employ standard metrics to quantitatively evaluate different generative models under NSC. Furthermore, we derive a stimulus-conditioned predictive model of neuronal responses from the trained generative model using our formalization that we compare to neural system identification models. We demonstrate our approach by fitting and comparing classical- and flexible deep learning-based generative models on population recordings from the macaque primary visual cortex (V1) to natural images, and show that the flexible models outperform classical models in both their generative- and predictive-model performance. Overall, our work is an important step towards a quantitative evaluation of NSC. It provides a framework that lets us learn the generative model directly from neuronal population recordings, paving the way for an experimentally-informed understanding of probabilistic computational principles underlying perception and behavior.

1 Introduction

Our environment is riddled with sensory stimuli that are noisy, ambiguous, and often incomplete, necessitating organisms to handle uncertainty in their sensory observations. Bayesian models of

perception and behavior have thus grown in prominence, successfully accounting for an extensive array of tasks across perception [1, 2], cognition [3], sensory-motor learning [4] and decision making [5-8]. These models posit that the brain maintains a statistical generative model of the world, where sensory observations x are generated from unknown, world-state variable z . Upon encountering a stimulus x , perception in the brain is conceptualized as probabilistically inferring the world-state variable z that caused x . In other words, to perceive x , the brain would invert the generative model to compute the posterior distribution over z : p ( z | x ) . One can express the posterior via Bayes' rule as: p ( z | x ) ∝ p ( x | z ) p ( z ) , where p ( z ) is the prior distribution of z and p ( x | z ) is the conditional distribution characterizing how well a given z describes x . While this has been an influential framework, the neuronal underpinnings of probabilistic inference remain challenging to conceptualize and test experimentally. To this end, the Neural Sampling Code (NSC) [9-16] is a prominent theory that offers a unique link between neuronal responses and probabilistic inference. Specifically, NSC posits that neuronal responses, r , to a given stimulus, x , can be thought of as samples drawn from the posterior distribution: r ∼ p ( z | x ) (Figure 1).

Background and related work Prevailing literature on NSC uses simple and restrictive generative models and performs qualitative comparisons of model predictions with neurophysiological data to test the theory. Notably, existing NSC works use simple prior- and conditional distributions with pre-specified parameters. For example, a popular choice for the conditional distribution of images (stimuli) has been Gaussian with a likelihood function that linearly combines pre-specified filters. Hoyer and Hyvärinen [9] learn these filters via independent component analysis on natural images, whereas Haefner, Berkes, and Fiser [12] use oriented Gabor filters instead. Similarly, a popular choice for the prior is the exponential distribution with a pre-specified rate parameter [9]. These choices are inspired by (a) what is already known about sensory neurons, especially in the primary visual cortex (V1), and (b) the fact that it renders posterior computation mathematically simpler. In the examples above, the choice of filters reflects well-known findings that the receptive fields of V1 neurons resemble (Gabor-like) orientation filters [17-20], and the exponential prior is motivated by the principle of sparse coding [9, 20]. Importantly, these parameters and distributions - and thereby, the generative models - are not informed or learned explicitly from neurophysiological data. Rather, these works typically sample from the posterior of the assumed generative model in response to strongly parameterized stimuli (e.g., noisy oriented gratings). The models - and thereby the theory - are then evaluated based on how well the samples qualitatively capture specific neurophysiological phenomena such as the mean-variance relationship [9, 21, 22], task-induced noise correlation structures [12], and contextual modulation in V1 neurons [16].

In contrast, recent advances in deep learning-based neural system identification models have set new standards in providing expressive models that can faithfully predict neural population responses to natural stimuli [23-39], and offer experimentally verifiable insights at the single-neuron level [31, 4042]. Additionally, advances in generative modeling, especially of images, have clearly demonstrated the effectiveness of deep, highly nonlinear, generative models such as auto-regressive models [43, 44], variational autoencoders [45-47], normalizing flows [48-51], and diffusion models [52-54]. Given the complexity of high-dimensional natural stimuli and real-world tasks, it is paramount that NSC be considered under a generative model that can match such complexity.

Our objective and contributions Here we ask: what exactly is the brain's generative model? More specifically, can we identify the brain's generative model from NSC population responses to natural stimuli? Although simple generative models and qualitative evaluations in the NSC literature have offered us great insight into the potential generative models of the brain and engendered support for the theory, there remains a conspicuous gap in the quantitative evaluation of NSC, particularly in response to natural stimuli. In this work, we bridge this gap by proposing a formalization of NSC that 1 allows us to directly fit NSC generative models to recorded neuronal activity in response to natural images, 2 formulate richer and more flexible generative models, and 3 employ standard metrics such as log-likelihood and single trial correlation to quantitatively evaluate different generative models under NSC. As opposed to specifying a generative model that ought to be maintained by the brain, our framework allows us to learn the generative model directly from neurophysiological data. Learning expressive generative models in a data-driven fashion additionally lets us take advantage of population recordings of large and ever-increasing scale in the field [55-57]. Furthermore, our formalization 4 lets us derive a stimulus-conditioned predictive model of neuronal responses from the trained generative model, which can be directly compared to state-of-the-art system identification

z

x

r

∼

p

(

z

|

x

(1)

)

z

2

z

1

A

B

C

Neuron 2 responses,

r

2

Neuron 1 responses,

r

1

p

(

z

|

x

(2)

)

p

(

z

|

x

(3)

)

z

x

p

(

z

|

x

)

p

(

x

|

z

)

p

(

z

)

r

Internal model

(1)

sample

x

models. The predictive model has the ability to provide experimentally-verifiable, neuron-specific predictions from the normative theory.

We demonstrate our approach by fitting classical generative models from NSC literature and flexible deep learning-based generative models on macaque primary visual cortex (V1) population responses to natural images. We show that the flexible models outperform classical models in both their generative- and predictive-model performance. Overall, this work presents an important step towards a quantitative evaluation of NSC, paving the way for a data-driven approach in learning the generative model of the brain.

2 Fitting the Neural Sampling Code

2.1 Theory

An explicit formalization of NSC We begin by formalizing NSC as a latent variable probabilistic model z -→ x -→ r , where z represents the world state variable that underlies the observable stimulus x (Figure 1 A ). Subsequently, the stimulus x gives rise to the neuronal responses r via the posterior p ( z | x ) (Figure 1 B ). NSC posits that the neuronal responses r elicited by stimulus x can be interpreted as stochastic samples from the posterior distribution p ( z | x ) (Figure 1 B, C ). However, the exact relation between z and r is often left unspecified. For instance, it is not clear what aspect of the neuronal response (e.g., firing rate, presence or absence of spikes, or membrane potential) should be treated as a sample. In fact, most previous works do not make a distinction between r and z , and simply equate an aspect of the neuronal response such as firing rate with the latent sample. Here, we make this assumption explicit and treat the neural response r as a random variable that matches the latent random variable z in stimulus-conditioned distributions:

Equation 2 is a slight abuse of notation to state the equivalence in the stimulus-conditioned distributions of r and z , more formally stated as:

where d = denotes equality in distribution or density function. By marginalizing the stimulus, we find that the marginal distribution of r must also match that of z :

Explicitly formalizing NSC with distinct r and z has two advantages. Firstly, the resulting formulation provides the crucial link between the generative model z → x and observed responses r , serving as the basis for learning the generative model from the responses. This also provides the basis for a neuron-specific model comparison between different NSC models as well as the possibility to make predictions for specific neurons that can be experimentally tested. Secondly, the explicit link highlights the possibility to explore more flexible mappings between z and r . For instance, one could assume that the latent variable z is encoded in the membrane potential, but what we observe are spike counts, i.e. r = f ( z ) for some stochastic mapping f . This relation can, in turn, become part of the model, which can then be fitted to real data and compared to alternative versions of the model. In this work, we choose to learn the generative models from the data under the simplest mapping of r ≡ z (Section 3.2). Please see Section 4 for a discussion on alternative mappings between r and z .

In NSC, we note that the latent variables are what underlie the stimulus (such as the intensity of an oriented filter in 1 A ) and are not necessarily any task-relevant experimenter-defined variables (such as orientation in an orientation-discrimination task). This is in contrast to the alternative theory of probabilistic population codes [58, 59], where typically, the latent variable is explicitly defined to be the task-relevant experimenter-defined variables.

Learning the generative model under NSC One way to quantitatively test an NSC generative model p ( z , x ) is testing how well the response distribution p ( r | x ) approximates the posterior p ( z | x ) of the generative model, i.e., testing equation 3. However in reality p ( z , x ) , and consequently p ( z | x ) , is unknown to us. However, our formalization allows us to learn the generative model p ( z , x ) via learning the joint distribution p ( r , x ) . The equivalence between p ( z , x ) and p ( r , x ) follows from equations 2, 3 and 4:

The joint distribution can then simply be fitted to recorded stimulus-response pairs { x ( i ) , r ( i ) } N i =1 by maximizing the likelihood:

where θ are the parameters of the generative model, that we split into θ L and θ P for parameters relevant to the likelihood and prior, respectively. Provided that θ L and θ P do not overlap, the generative models can be learned by learning the likelihood and prior separately.

Evaluating NSC on data Fitting generative models under NSC on recorded data allows us to compare the generative models quantitatively by evaluating their performances as log-likelihood on a held-out test set. Furthermore, once we have learned the generative model p ( r , x ; θ ∗ ) , we can invert it to arrive at the posterior p ( r | x ; θ ∗ ) . This provides a neuronal encoding model p ( r | x ) under specific assumptions of the NSC model, allowing us to predict neural responses to arbitrary new stimuli. The performance of this predictive model can serve as yet another metric for quantitative model comparison under NSC. Additionally, the posterior allows us to compare the generative models to the normative-theory-free system identification models. It is important to note that our quantification does not make any assumption about the kind of stimuli x . Existing works on NSC use parametric stimuli from classical neuroscience experiments and perform a qualitative comparison between model- and real-neuronal responses to the same stimuli. Our formulation, on the other hand, allows us to compare different NSC models on natural images , the type of stimuli the visual system has evolved to process.

2.2 Models

Following previous work in NSC [9-16], here we focus on vision and develop generative models under NSC for natural image stimuli x and spike counts r recorded from the visual cortex (Figure 2 A ). Developing generative models entails developing models for the prior p ( r ) and the likelihood p ( x | r ) (Figure 2 B ). Additionally, we also fit an approximate posterior q ( r | x ) . We summarize our fitting methodology in an algorithm towards the end of the section (Algorithm 1).

A

B

z

/

r

x

θ

L

θ

P

ϕ

r

MLP :

θ

L

/u1D4A9

(

x

|

μ

,

σ

2

⋅

I

)

System identification

CNN :

ϕ

p

Γ

(

r

j

|

α

,

β

)

x

C

Prior

Likelihood

Posterior

Flow transform,

:

T

-

1

ω

∈

θ

P

g

∼

N

(0,1)

Fixation spot

Grayscale

natural images

Awake fixating

rhesus macaque

V1 recordings

via depth probe

Quantizer

⌊

ζ

⌋

r

Spike

sorting

Spikes

Dequantizer:

ν

∈

θ

P

ζ

Parameterized

generative model

Prior Spike counts r are discrete. Hence, we can neither directly fit standard literature models, such as an exponential [9] or a Laplace distribution [20] to the discrete variable r , nor can we straightforwardly fit more common flexible density models such as normalizing flows [49, 60], as they are all continuous density models. A common approach to remedy this is to employ uniform dequantization, where the discrete quantity is converted into a continuous signal by adding uniform random noise [43, 61, 62]. We adopt the more general approach of variational dequantization where the noise distribution is learned instead of being fixed to be uniform [63-66]. In this method, prior distribution over discrete r is captured by positing a generative model involving a continuous latent variable ζ linked to the discrete response r via a deterministic quantizer function: P ( r ) = ∫ P ( r | ζ ) p ( ζ ) d ζ , where P ( r | ζ ) is the quantizer and p ( ζ ) is the continuous prior .

Since the integral is usually intractable, the whole model is fit by optimizing the evidence lower bound (ELBO):

where q ( ζ | r ; ν ) is the approximate posterior distribution with parameters ν , p ( ζ ; ω ) is the continuous prior with parameters ω , and H ( q ( ζ | r ; ν )) = -E ζ ∼ q ( ζ | r ; ν ) [log q ( ζ | r ; ν )] is the conditional entropy of the dequantizing distribution. Note that Equation (8) only provides a lower-bound to log P ( r ) , and a tighter bound via importance-weighted sampling [66-68] (Appendix A).

In our work, we only consider factorized prior distributions, i.e., we treat neurons to be a priori independent log P ( r ) = ∑ i log P ( r i ) . This choice is informed by both the nature of the V1 neural data that showed limited correlation across all stimuli and the simplicity of the fit it provides. The same independence assumption was applied for the continuous prior distribution over the dequantized responses ζ . Given this dequantizer framework, we explore three different NSC priors by varying the distribution over the continuous latent p ( ζ ; ν ) :

Exponential ( Exp ), 1 λ exp -ζ λ H ( ζ ) , as found in the original NSC model by Hoyer & Hyvärinen [9].

Normalizing flow ( Flow ): p ( ζ ; ω ) = p base ( T -1 ( ζ ; ω )) · | ∂ T -1 ( ζ ; ω ) ∂ ζ | , where we choose p base to be a standard normal, and T -1 represents the following series of invertible mappings with learnable parameters ω : [ affine , tanh , affine , tanh , affine , tanh , affine , softplus -1 ] , where softplus -1 ( y ) = log( e y -1) , affine ( y ) = ay + b with learnable parameters a and b . softplus -1 ensures that the support of ζ is non-negative, since we are ultimately interested in modeling the distribution of (non-negative) spike counts (Figure 2 C , 'Prior' sub-panel).

Half-normal ( HN ), √ 2 σ √ π exp ( -ζ 2 2 σ 2 ) H ( ζ ) , where H ( ζ ) is the heavyside function.

For the dequantizer distribution, q ( ζ | r ; ν ) , we utilize a conditional normalizing flow-based flexible distribution, as in [66] (Appendix B).

Likelihood We model the likelihood as an isotropic Gaussian distribution:

where the parameters mean, µ ( i ) ∈ R | x | and variance, σ ( i ) ∈ R | x | > 0 are functions of response, r ( i ) , and | x | is the number of dimensions of x . We consider (1) a linear function where µ = w µ r ( i ) + b µ and σ = exp w σ r ( i ) + b σ ( Lin ) and (2) a nonlinear function µ = w µ MLP ( r ( i ) ) + b µ and σ = exp w σ MLP ( r ( i ) )+ b σ , where MLP ( · ) is a neural network ( MLP ) (Figure 2 C 'Likelihood' sub-panel).

Posterior In many cases, the posterior distribution for a desired generative model is not analytically tractable and must be approximated, commonly using variational inference or Markov Chain Monte Carlo sampling [69-71]. Here, since we learn the generative model p ( x , r ; θ ∗ ) , we can approximate the true posterior p ( r | x ; θ ∗ ) by fitting a model posterior q ( r | x ; φ ) to samples from p ( x , r ; θ ∗ ) directly via maximum log-likelihood:

where x ′ , r ′ ∼ p ( x , r ; θ ∗ ) , are samples from the trained generative model. We model the posterior distribution of responses conditioned on images as a factorized Gamma distribution, following state-of-the-art (SOTA) work in system identification [72]: p ( r | x ( i ) ) = ∏ S j =1 p Γ ( r j | α ( i ) , β ( i ) ) , where x ( i ) is the i th image, r j is the j th neuron out of | r | = S total neurons, and the parameters concentration, α ( i ) and rate, β ( i ) are functions of the image, x ( i ) . Since these functions map an image to response distribution parameters, we model them using a convolutional neural network model (Figure 2 C 'Posterior' sub-panel), following SOTA system identification work [25, 27, 73] (Appendix C).

3 Experiments

3.1 Synthetic data

We simulated 10,000 pairs of images and neuronal responses from the following three classical NSC models: 1 a Hoyer & Hyvärinen model (HNH) with an exponential prior [9], 2 an Olshausen &Field (ONF) model where the prior is a Laplace distribution [20], and 3 a full Gaussian model (Gauss) where the prior is an isotropic Gaussian with mean 0 and variance σ 2 r . All the three models share a common linear, isotropic Gaussian likelihood p ( x | r ) = N ( x | A r , σ 2 I ) , where A is the factor loading matrix learned via standard independent component analysis model (ICA) with a complete basis on natural image patches [9, 74]. Additionally, we sampled image-response pairs from 4 our flexible model with Flow prior (described in Section 2.2) and MLP-based likelihood (Section 2.2), where all parameters were randomly initialized. For any given generative model, we first sample neuronal responses from the prior via r ( i ) ∼ p ( r ) , and then sample corresponding images via

∑

(

)

x ( i ) ∼ p ( x | r ( i ) ) , where i ∈ 1 , . . . , 10 , 000 . We hold out a set of 1,000 pairs as the test set. We fitted all models on the datasets simulated from the classical as well as the flexible models via Equation (7) and computed joint log-likelihoods of the trained models on the held-out test set as log p ( x ( i ) , r ( i ) ) = log p ( x ( i ) | r ( i ) ) +log p ( r ( i ) ) . For the classical models, maximum likelihood estimates of the parameters were obtained analytically (Appendix F). We trained the flexible model using gradient descent.

We find that (1) the flexible model fits responses and images simulated under other NSC models well, i.e., learns p ( r ) and p ( x | r ) and closely approximates the loglikelihood of the true models. Importantly, it outperforms the fit of other NSC models with mismatched generative distributions, consistently being the best model after the ground-truth model (first 3 columns in Figure 3). Furthermore, the flexible model is capable of generating complex image and response distributions that could not be easily captured by the classical generative models (column 4 in Figure 3). This demonstrates that our framework allows for NSC model fitting and that the flexible model has the ability to flexibly capture the data distribution across a wide range of generative models. Critically, a flexible model could fit complex generative models that cannot be modeled well by other classical models.

3.2 Neurophysiological data

Data description Next, we demonstrate applying our approach to real neuronal data. We used 32-channel laminar

-

92.55

-

93.62

-

131.58

-

381.58

-

388.44

-

386.00

-

360.19

-

355.80

-

358.54

Gauss

HNH

ONF

Gauss

HNH

ONF

Flex

Data

Models

-

122.73

Flex

-

217.94

-

523.58

-

460.91

-

449.81

Best

Worst

N/A

NeuroNexus arrays (Figure 2 A ) to record population activity from the primary visual cortex (V1) of two awake male rhesus macaque monkeys ( Macaca mulatta ) [29] as they fixated on grayscale natural images sampled from the ImageNet dataset [75]. All the experiments concerned with the recordings adhered to the National Institutes of Health, United States guidelines, and received approval from the Institutional Animal Care and Use Committee. Each image was presented for 120 ms, and spike counts between 40 ms and 160 ms after the image onset were computed and used as the neuronal response r . The image stimulus x used for modeling is 41 × 41 px. For more details on the experiments and data collection, refer to Cadena et al. [29]. We collected data across 12 recording sessions, each having approximately 16,000 image-response pairs and at least 16 well-isolated single units. We split the dataset into approximately 10,000 pairs for training, 3,000 pairs for validation, and 3,000 for testing (for exact details on all sessions, see Appendix D). We do not aggregate data across sessions and fit models separately for each session since the images can differ from session to session, and not all neurons have seen every image.

Fitting the generative model Given a dataset of images and responses, { x , r } N i =0 , we fit the likelihood p ( x | r ; θ L ) on the image-response pairs and the prior p ( r ; θ P ) on the responses, r as in Equation (7). We fit all of the generative models on each recording session as following procedure described in Table 1. Below, we describe our results for the session with the largest number of neurons (29 well-isolated single units) in detail and report summary results on all sessions.

For the prior models (Figure 4 A ), we report the test-set log-likelihood performance of all models (Exp, HN, Flow) relative to the Exp1-model as the baseline. We find that our flexible normalizing flow model (Flow) achieves the best performance, improving the score from the exponential distribution (Exp) by 0.095 bits per neuron per trial, amounting to 2.755 bits across 29 neurons per trial. For the likelihood models (Figure 4 B ), we find that using an MLP likelihood function, the model improved by 0.052 bits per pixel per trial, amounting to 87.19 bits of improvement across all 41 × 41 pixels per trial, relative to the model with linear likelihood function. For the joint distribution (Figure 4 C ), we find that the flexible model (Flow-MLP) achieves the highest log-likelihood score, offering an improvement of 1.8452e-3 bits per pixel per neuron per trial, amounting to 89.951 bits across all 41 × 41 pixels and 29 neurons per trial. We observed that in each of the cases, flexible models (Flow prior, MLP likelihood) offer much higher log-likelihood performance, with the same trend found across all sessions (Appendix D).

A

B

C

Prior

p

(

r

)

Joint

p

(

x

,

r

)

Hoyer & Hyvärinen

model

Relative log-likelihood (lower bound)

Log-likelihood

Relative log-likelihood (lower bound)

Likelihood

p

(

x

|

r

)

Log-likelihood

(lower bound)

Correlation

NSC generative models

NSC generative models

A

C

D

B

Figure 5: Posterior performance of NSC models, along with system identification model ('Sysident'). Error bars represent the standard error of mean over neuronal responses. All metrics are averaged over number of neurons and trials. A . The lower bound of log-likelihood in bits, for session with 29 neurons. We compute the lower-bound since we are evaluating the Gamma-posterior on (discrete) spike counts, and full-likelihood is intractable (Appendix G). B . Same as A but across all 12 sessions (purple: average, gray: single session). C . Single-trial correlation, for session with 29 neurons. D . Same as C across all 12 sessions

Learning the posterior distribution For each of the trained generative models, p ( x , r ; θ ∗ ) , we approximated the model's posterior distribution p ( r | x ; θ ∗ ) using an approximate posterior q ( r | x ; φ ) trained on samples drawn from the trained generative model (Equation (10), Algorithm 1). We evaluated the posterior distribution for each generative model by computing their mean log-likelihood on real neuronal responses conditioned on real images from the test set (Figure 5 A, B ). We also computed a single-trial correlation between the mean of the learned posterior distributions and neuronal responses (Figure 5 C, D ). Finally, we compared the posterior distribution to a deep system identification model.

We find that, in general, a more flexible trained generative model tends to yield a higher posterior predictive performance. Based on the log-likelihood evaluation, our flexible generative model (FlexMLP) gained as much as 1.39 bits per neuron per trial compared to the baseline Exp1-Lin model and 0.61 bits per neuron per trial compared to Exp-Lin model (the Hoyer & Hyvärinen model [9]). In terms of single-trial correlation performance, our flexible generative model (Flex-MLP) achieved 10% higher correlation compared to the Exp1-Lin baseline and 8% higher correlation compared to Exp-Lin. When averaged across all sessions, we find that Flow-Lin performs best, almost on

par with the Flow-MLP, which achieves 0.019 bits per neuron per trial less. Furthermore, a system identification model trained on the dataset of real neuronal responses performed better than the best NSC generative model, gaining 0.17 bits per neuron per trial and 17% higher correlation per neuron per trial compared to Flex-MLP.

All model training was performed using backpropagation and gradient descent and we provide training, compute and infrastructure details in Appendix E.

4 Discussion

The main focus of this work was to develop a way to answer the question, 'How well does NSC explain neurophysiological data quantitatively ?'. While NSC is a prominent normative theory for probablistic computation in the brain, and the literature has provided much qualitative insight, our work is the first to offer a quantitative paradigm for empirically testing it using brain responses to ecological, natural stimuli. Our framework additionally lets us formulate more flexible generative models - which can be better informed by the data - and employ standard metrics such as loglikelihood to quantitatively evaluate alternative generative models under NSC. Furthermore, inverting the learned generative model has allowed us to obtain the posterior distribution, which is equivalently a neuronal response predictive model. Importantly, this let us compare NSC models to models outside of NSC's theoretical framework, such as system identification models, allowing us to benchmark the predictive performance of NSC models. Our results demonstrated that the flexible generative models outperformed classical models in terms of both generative and predictive model performance, yet system identification models achieve superior response-predictive performance compared to even our best generative models. We now discuss some limitations of our current study and discuss a number of open questions and imminent future directions.

Limitations I: Assumption of strict 1:1 neuron-latent mapping : One limitation in our current study is that we only use a 1:1 identity mapping between the activity of neurons and latent variables in our formulation of NSC. Abiding by this restriction could limit the capacity of the NSC models, especially considering some existing work in NSC that have qualitatively explored more flexible mappings. For example, Orbán et al. [13] model membrane potential values (responses) as a nonlinear function of posterior samples. Furthermore, Savin and Denève [76] map responses of N neurons to D latent variables where N > D . Many more ways of how r and z relate are conceivable. However, our formulation with a separate r and z allows us to, in principle, incorporate different mappings and learn the corresponding generative models. Since the focus of this study was on the aspect of fitting NSC models to data, we chose the simplest (original) interpretation of NSC where r ≡ z .

Limitations II: Definition of a 'sample' as total spike counts : We defined a 'sample' as the total spike count of neuronal activity within a specific time window following the stimulus, which is not necessarily what literature works do. However, to our knowledge, there is no generally agreed upon or rigorous definition of a 'sample' in NSC. While NSC was originally motivated with firing rate/spike counts over a 500ms window as the sample [9, 21], many alternative definitions such as membrane potential over 10ms [13] have been employed. It is unclear on what generally applicable metric other than goodness of fit to data - such a definition could be evaluated. This in fact served to us as another motivating factor for striving towards a data-driven evaluation of sampling models that would allow one to compare such choices in an informed manner. In this work, we chose the total spike count as the working definition.

Limitations III: Better generative models are needed: Advances in deep learning architectures, latent variable models, and transfer learning have greatly enhanced the capabilities of generative models in machine learning. We believe the models we chose, although more expressive than classical models, are still limiting, especially considering that our likelihood p ( x | r ) uses linear or MLP decoding from neurons to images, with a simple Gaussian noise model. To capture the rich and complex nature of neuronal representations of natural images, we believe it is necessary to consider more sophisticated generative models, that even incorporate a natural image prior, that would eventually close the gap in predictive performance between system identification performance and NSC generative models. Furthermore, an important avenue of research is identifying biological mechanisms that underlie NSC (i.e. sampling from the posterior) [14, 77-82]. It is worth noting that our deep learning-based generative models are not meant to be mechanistic models of NSC neurons.

Rather, we believe that our approach lays the foundation for alternative biologically plausible models to be quantitatively evaluated and compared.

Whydosystem identification models perform better than NSC generative models? System identification models are directly trained discriminatively, i.e., min θ E x [ D KL ( p true ( r | x ) || p model ( r | x ; θ ))] , to predict neuronal responses to natural images and deep-learning based ones are currently SOTA. There is still much room to build better generative models that would better explain the data (see Limitations III). However, for a given dataset of responses to stimuli from a fixed stimulus distribution , we do not expect the posterior of even the ideal generative model to surpass the performance of the ideal system identification model because the generative model training, i.e., min θ { E x [ D KL ( p true ( r | x ) || p model ( r | x ; θ ))] + D KL ( p true ( x ) || p model ( x ; θ )) } , does not provide any advantage over system identification in response prediction unless some specific inductive biases are introduced in the generative model.

Why bother fitting NSC models if they fail to quantitatively compete with system identification? If we change the stimulus distribution p ( x ) to p new ( x ) with markedly different stimulus statistics and let the sensory neurons adapt to p new ( x ) , we would expect the system identification model's performance to drop on p new ( x ) . The system identification model might have to be retrained on a new dataset of responses under p new ( x ) . This is the case where we would expect the NSC's learned generative model to be beneficial. Specifically, change in p ( x ) to p new ( x ) may entirely derive from the change in prior p ( z ) to p new ( z ) , while p ( x | z ) remains fixed. Hypothetically, this is since p ( x | z ) represents the invariant 'physical' process by which the latents (e.g., the identity of an animal) give rise to observations (e.g., the appearance of the animal). Consequently, if NSC accurately describes a neural population, i.e, r ∼ p ( z | x ) ∝ p ( x | z ) p ( z ) , the neuronal adaptation can be accounted for by simply learning p new ( z ) , i.e., r new ∼ p new ( z | x ) ∝ p ( x | z ) p new ( z ) , keeping p ( x | z ) fixed. We believe such out-of-context generalization is a theoretical strength of NSC, and is a consequence of its normative nature (responses being 'samples' from the posterior distribution ). Such normative hypotheses are neither present in the purely phenomenological system identification models and nor is it straightforward to equip them with normative assumptions.

The above insight thus helps us identify potential future experiments to test NSC models utilizing our framework since it lets us learn the generative model p ( x | z ) p ( z ) via p ( x | r ) p ( r ) (NSC assumption). Namely, one could perform experiments in which we let the neural population adapt to different sensory contexts with expected shifts in p ( z ) . Using our NSC framework, we would expect to be able to predict how neuronal responses should change (as reflected in updated r new ∼ p new ( z | x ) ) under new contexts.

Why have previous works not fit NSC models to data? We attribute the lack of such attempts to (1) limitations in data availability, (2) complexities involved in training flexible machine learning and inference algorithms on recorded data, and (3) the philosophical approach behind normative theories. Normative theories describe how a biological system ought to function in order to tackle fundamental tasks. They propose models with parameters that are optimized for those tasks, without relying on actual experimental data [83, 84]. Typically normative theories are evaluated using qualitative agreements between the proposed models and data. NSC is itself a normative theory. In contrast, phenomenological approaches such as system identification propose models whose parameters are directly learned from experimental data. Normative and phenomenological approaches have historically been developed independently of each other. Similar to M ł ynarski et al. [83], who interpolate between phenomenological and normative models via maximum entropy priors, our approach allows us to get the best of both worlds: state-of-the-art deep learning-based system identification models from phenomological approaches and the theoretical underpinnings of the normative NSC. System identification provides us with expressive models that faithfully model and predict the activity of thousands of neurons to rich natural stimuli. NSC, on the other hand, goes beyond what experimental data alone could offer by letting us hypothesize how neurons encode uncertainty about the stimulus, reflecting the posterior distribution over latent variables in a generative model of the world, thus allowing us make novel predictions such as about generalizability across stimulus contexts and design relevant experiments.

References

Yair Weiss, Eero P Simoncelli, and Edward H Adelson. 'Motion illusions as optimal percepts'. In: Nature neuroscience 5.6 (2002), pp. 598-604.

David C Knill and Whitman Richards. Perception as Bayesian inference . Cambridge University Press, 1996.

Charles Kemp et al. 'A probabilistic model of theory formation'. In: Cognition 114.2 (2010), pp. 165-196.

Konrad P Körding and Daniel M Wolpert. 'Bayesian integration in sensorimotor learning'. In: Nature 427.6971 (2004), pp. 244-247.

Marc O Ernst and Martin S Banks. 'Humans integrate visual and haptic information in a statistically optimal fashion'. In: Nature 415.6870 (2002), pp. 429-433.

Aaron C Courville, Nathaniel D Daw, and David S Touretzky. 'Bayesian theories of conditioning in a changing world'. In: Trends in cognitive sciences 10.7 (2006), pp. 294-300.

Konrad Kording. 'Decision theory: what" should" the nervous system do?' In: Science 318.5850 (2007), pp. 606-610.

Nathaniel D Daw et al. 'Cortical substrates for exploratory decisions in humans'. In: Nature 441.7095 (2006), pp. 876-879.

Patrik Hoyer and Aapo Hyvärinen. 'Interpreting neural response variability as Monte Carlo sampling of the posterior'. In: Advances in neural information processing systems 15 (2002).

József Fiser et al. 'Statistically optimal perception and learning: from behavior to neural representations'. In: Trends in cognitive sciences 14.3 (2010), pp. 119-130.

Pietro Berkes et al. 'Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment'. In: Science 331.6013 (2011), pp. 83-87.

Ralf M Haefner, Pietro Berkes, and József Fiser. 'Perceptual decision-making as probabilistic inference by neural sampling'. In: Neuron 90.3 (2016), pp. 649-660.

Gerg˝ o Orbán et al. 'Neural variability and sampling-based probabilistic representations in the visual cortex'. In: Neuron 92.2 (2016), pp. 530-543.

Rodrigo Echeveste et al. 'Cortical-like dynamics in recurrent circuits optimized for samplingbased probabilistic inference'. In: Nature neuroscience 23.9 (2020), pp. 1138-1149.

Camille Rullán Buxó and Cristina Savin. 'A sampling-based circuit for optimal decision making'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 1416314175.

Dylan Festa et al. 'Neuronal variability reflects probabilistic inference tuned to natural image statistics'. In: Nature communications 12.1 (2021), p. 3635.

David H Hubel and Torsten N Wiesel. 'Receptive fields of single neurones in the cat's striate cortex'. In: The Journal of physiology 148.3 (1959), p. 574.

S Marˆ celja. 'Mathematical description of the responses of simple cortical cells'. In: JOSA 70.11 (1980), pp. 1297-1300.

John G Daugman. 'Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters'. In: JOSA A 2.7 (1985), pp. 11601169.

Bruno A Olshausen and David J Field. 'Emergence of simple-cell receptive field properties by learning a sparse code for natural images'. In: Nature 381.6583 (1996), pp. 607-609.

AF Dean. 'The variability of discharge of simple cells in the cat striate cortex'. In: Experimental Brain Research 44.4 (1981), pp. 437-440.

David J Tolhurst, J Anthony Movshon, and Andrew F Dean. 'The statistical reliability of signals in single neurons in cat and monkey visual cortex'. In: Vision research 23.8 (1983), pp. 775-785.

Daniel LK Yamins et al. 'Performance-optimized hierarchical models predict neural responses in higher visual cortex'. In: Proceedings of the national academy of sciences 111.23 (2014), pp. 8619-8624.

Alexander S Ecker et al. 'A rotation-equivariant convolutional neural network model of primary visual cortex'. In: arXiv preprint arXiv:1809.10504 (2018).

Santiago A Cadena et al. 'Deep convolutional models improve predictions of macaque V1 responses to natural images'. In: PLoS computational biology 15.4 (2019), e1006897.

Benjamin Cowley and Jonathan W Pillow. 'High-contrast 'gaudy' images improve the training of deep neural network models of visual cortex'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 21591-21603.

Konstantin-Klemens Lurz et al. 'Generalization in data-driven models of primary visual cortex'. In: BioRxiv (2020), pp. 2020-10.

Mohammad Bashiri et al. 'A flow-based latent state generative model of neural population responses to natural images'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 15801-15815.

Santiago A Cadena et al. 'Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks'. In: bioRxiv (2022), pp. 2022-05.

Konstantin F Willeke et al. 'Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization'. In: bioRxiv (2023), pp. 2023-05.

Pawel A Pierzchlewicz et al. 'Energy Guided Diffusion for Generating Neurally Exciting Images'. In: bioRxiv (2023), pp. 2023-05.

Charles F Cadieu et al. 'Deep neural networks rival the representation of primate IT cortex for core visual object recognition'. In: PLoS computational biology 10.12 (2014), e1003963.

Eleanor Batty et al. 'Multilayer recurrent network models of primate retinal ganglion cell responses'. In: International Conference on Learning Representations . 2016.

Ján Antolık et al. 'Model constrained by visual hierarchy improves prediction of neural responses to natural scenes'. In: PLoS computational biology 12.6 (2016), e1004927.

Lane McIntosh et al. 'Deep learning models of the retinal response to natural scenes'. In: Advances in neural information processing systems 29 (2016).

William F Kindel, Elijah D Christensen, and Joel Zylberberg. 'Using deep learning to reveal the neural code for images in primary visual cortex'. In: arXiv preprint arXiv:1706.06208 (2017).

David Klindt et al. 'Neural system identification for large populations separating 'what' and 'where''. In: Advances in Neural Information Processing Systems 30 (2017).

Martin Schrimpf et al. 'Brain-score: Which artificial neural network for object recognition is most brain-like?' In: BioRxiv (2018), p. 407007.

Fabian Sinz et al. 'Stimulus domain transfer in recurrent models for large scale cortical population prediction on video'. In: Advances in neural information processing systems 31 (2018).

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. 'Neural population control via deep image synthesis'. In: Science 364.6439 (2019), eaav9436.

Carlos R Ponce et al. 'Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences'. In: Cell 177.4 (2019), pp. 999-1009.

Edgar Y Walker et al. 'Inception loops discover what excites neurons most using deep predictive models'. In: Nature neuroscience 22.12 (2019), pp. 2060-2065.

Benigno Uria, Iain Murray, and Hugo Larochelle. 'RNADE: The real-valued neural autoregressive density-estimator'. In: Advances in Neural Information Processing Systems 26 (2013).

Aaron Van den Oord et al. 'Conditional image generation with pixelcnn decoders'. In: Advances in neural information processing systems 29 (2016).

Diederik P Kingma and Max Welling. 'Auto-encoding variational bayes'. In: arXiv preprint arXiv:1312.6114 (2013).

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 'Stochastic backpropagation and approximate inference in deep generative models'. In: International conference on machine learning . PMLR. 2014, pp. 1278-1286.

Rewon Child. 'Very deep vaes generalize autoregressive models and can outperform them on images'. In: arXiv preprint arXiv:2011.10650 (2020).

Laurent Dinh, David Krueger, and Yoshua Bengio. 'Nice: Non-linear independent components estimation'. In: arXiv preprint arXiv:1410.8516 (2014).

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 'Density estimation using real nvp'. In: arXiv preprint arXiv:1605.08803 (2016).

Mathieu Germain et al. 'Made: Masked autoencoder for distribution estimation'. In: International conference on machine learning . PMLR. 2015, pp. 881-889.

Durk P Kingma and Prafulla Dhariwal. 'Glow: Generative flow with invertible 1x1 convolutions'. In: Advances in neural information processing systems 31 (2018).

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 'Denoising diffusion probabilistic models'. In: Advances in neural information processing systems 33 (2020), pp. 6840-6851.

Jiaming Song, Chenlin Meng, and Stefano Ermon. 'Denoising diffusion implicit models'. In: arXiv preprint arXiv:2010.02502 (2020).

Robin Rombach et al. 'High-resolution image synthesis with latent diffusion models'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2022, pp. 10684-10695.

Ian H Stevenson and Konrad P Kording. 'How advances in neural recording affect data analysis'. In: Nature neuroscience 14.2 (2011), pp. 139-142.

Peiran Gao and Surya Ganguli. 'On simplicity and complexity in the brave new world of large-scale neuroscience'. In: Current opinion in neurobiology 32 (2015), pp. 148-155.

Cole Hurwitz et al. 'Building population models for large-scale neural recordings: Opportunities and pitfalls'. In: Current opinion in neurobiology 70 (2021), pp. 64-73.

Wei Ji Ma et al. 'Bayesian inference with probabilistic population codes'. In: Nature neuroscience 9.11 (2006), pp. 1432-1438.

Jeffrey M Beck et al. 'Probabilistic population codes for Bayesian decision making'. In: Neuron 60.6 (2008), pp. 1142-1152.

Danilo Rezende and Shakir Mohamed. 'Variational inference with normalizing flows'. In: International conference on machine learning . PMLR. 2015, pp. 1530-1538.

L Theis, A van den Oord, and M Bethge. 'A note on the evaluation of generative models'. In: International Conference on Learning Representations (ICLR 2016) . 2016, pp. 1-10.

Aaron Van den Oord and Benjamin Schrauwen. 'Factoring variations in natural images with deep gaussian mixture models'. In: Advances in neural information processing systems 27 (2014).

Jonathan Ho et al. 'Flow++: Improving flow-based generative models with variational dequantization and architecture design'. In: International Conference on Machine Learning . PMLR. 2019, pp. 2722-2730.

Christina Winkler et al. 'Learning likelihoods with conditional normalizing flows'. In: arXiv preprint arXiv:1912.00042 (2019).

Emiel Hoogeboom et al. 'Integer discrete flows and lossless compression'. In: Advances in Neural Information Processing Systems 32 (2019).

Emiel Hoogeboom, Taco S Cohen, and Jakub M Tomczak. 'Learning discrete distributions by dequantization'. In: arXiv preprint arXiv:2001.11235 (2020).

Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 'Importance weighted autoencoders'. In: arXiv preprint arXiv:1509.00519 (2015).

Justin Domke and Daniel R Sheldon. 'Importance weighting and variational inference'. In: Advances in neural information processing systems 31 (2018).

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 'Variational inference: A review for statisticians'. In: Journal of the American statistical Association 112.518 (2017), pp. 859-877.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning . Vol. 4. 4. Springer, 2006.

David JC MacKay. Information theory, inference and learning algorithms . Cambridge university press, 2003.

Konstantin-Klemens Lurz et al. 'Bayesian Oracle for bounding information gain in neural encoding models'. In: The Eleventh International Conference on Learning Representations . 2023. URL: https://openreview.net/forum?id=iYC5hOMqUg .

Luca Baroni et al. 'Learning invariance manifolds of visual sensory neurons'. In: NeurIPS Workshop on Symmetry and Geometry in Neural Representations . PMLR. 2023, pp. 301-326.

Aapo Hyvärinen. 'Independent component analysis: recent advances'. In: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371.1984 (2013), p. 20110534.

Jia Deng et al. 'Imagenet: A large-scale hierarchical image database'. In: 2009 IEEE conference on computer vision and pattern recognition . Ieee. 2009, pp. 248-255.

Cristina Savin and Sophie Denève. 'Spatio-temporal Representations of Uncertainty in Spiking Neural Networks'. In: Advances in Neural Information Processing Systems . Ed. by Z Ghahramani et al. Vol. 27. Curran Associates, Inc., 2014, pp. 2024-2032.

Lars Buesing et al. 'Neural dynamics as sampling: a model for stochastic computation in recurrent networks of spiking neurons'. In: PLoS computational biology 7.11 (2011), e1002211.

Yanping Huang and Rajesh P Rao. 'Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks'. In: Advances in neural information processing systems 27 (2014).

Guillaume Hennequin, Laurence Aitchison, and Máté Lengyel. 'Fast sampling-based inference in balanced neuronal networks'. In: Advances in neural information processing systems 27 (2014).

Xingsi Dong et al. 'Adaptation Accelerating Sampling-based Bayesian Inference in Attractor Neural Networks'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 21534-21547.

Paul Masset et al. 'Natural gradient enables fast sampling in spiking neural networks'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 22018-22034.

Shirui Chen et al. 'Expressive probabilistic sampling in recurrent neural networks'. In: arXiv preprint arXiv:2308.11809 (2023).

Wiktor M ł ynarski et al. 'Statistical analysis and optimality of neural systems'. en. In: Neuron 109.7 (Apr. 2021), 1227-1241.e5.

Daniel Levenstein et al. 'On the role of theory and modeling in neuroscience'. In: Journal of Neuroscience 43.7 (2023), pp. 1074-1088.

Adam Paszke et al. 'Pytorch: An imperative style, high-performance deep learning library'. In: Advances in neural information processing systems 32 (2019).

Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 (2014).

Peer reviews:
comments: SUMMARY:
The authors used several generative models to fit the image data and the corresponding neural responses. And for each generative model, they used the variational inference method to get their posterior.

STRENGTHS:
The article is written in a clear manner.

WEAKNESSES:
The authors used several generative models to fit the image data and the corresponding neural responses. Neither the generative model nor the fitting method is biologically plausible. This cannot explain the working mechanisms of the neural system; it can only assess how well the model fits the data. And it is evident that more complex models tend to fit better. I fail to see the significance of this work.

QUESTIONS:
1. What do the two conditional probabilities in Eq.(1) mean? Could you provide a detailed explanation?
 When talking on NSC, I believe the latent variables z should be abandoned. Please see [1].

2. What does the term normative theory/assumption mean?

3. What insights can we gain from the the generative model fitted by neuronal population recordings?

Reference:
[1] Shivkumar, Sabyasachi, et al. "A probabilistic population code based on neural samples." Advances in neural information processing systems 31 (2018).

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel formalization of the Neural Sampling Code (NSC) in this paper. The authors aim to bridge the gap in existing NSC research that needs more quantitative alignment between normative theory and neuronal recordings. \n\nThe authors propose formalization enables fitting NSC generative models directly to neuronal activity, creating richer, more flexible models and allowing standard metrics to evaluate different generative models.\n\nThe authors propose to compare classical and deep learning-based generative models on population recordings in V1 in response to natural images. Their framework outperforms classical models in both generative- and predictive-model performance. They also propose a method for obtaining stimulus-conditioned predictive models of neuronal responses that can be compared to existing neural system identification models.\n\nSTRENGTHS:\nThe authors propose a novel formalization of the Neural Sampling Code (NSC), allowing for the direct fitting of NSC generative models to recorded neuronal activity. Also, the paper presents a new way to evaluate different generative models under NSC using standard metrics quantitatively. \n\nThe study compares classical and deep learning-based generative models, providing valuable insights into the capabilities of different generative models in capturing the structure of real neuronal responses.\n\nThe research is grounded in the practical application of the theory, using neuronal activity data from a macaque primary visual cortex (V1) in response to natural images. \n\nFinally, the authors spend a significant amount of space identifying the limitations and suggesting areas for future research, promoting ongoing dialogue and exploration in the field.\n\nWEAKNESSES:\nI believe that the main weaknesses of the current paper have been presented by the authors in the discussion section but do not undermine the validity of the current study. \n\nHowever, one important limitation lies in the experimental section. The paper provides limiting experimental results when it comes to different neural recordings. In order to clearly evaluate the validity of the claim, one would need to explore other brain areas, probably still in the early sensory processing regions.\n\nQUESTIONS:\nHow does the model perform in other brain regions?\n\nLIMITATIONS:\nClearly laid out by the authors", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors employed a generative model approach to predict V1 neural responses, aiming to evaluate the Neural Sampling Code (NSC) using a quantitative metric. The primary potential contribution lies in the theoretical formulation and the idea that neural response prediction can be utilized to assess the NSC hypothesis, especially on a per-trial basis.\n\nSTRENGTHS:\nThe authors employed a generative model approach to predict V1 neural responses. This is a novel contribution in the area of single neuron response prediction, even though generative models like VAE, beta-VAE, and hierarchical VAE have been commonly used in analyzing fMRI data, particularly for reconstructing the input image. \n\nThe primary potential contribution lies in the theoretical formulation and the idea that neural response prediction can be utilized to assess the NSC hypothesis, especially on a per-trial basis.  The proposal that the performance of a  generative model for predicting neural response can provide a quantitative metric for evaluating the Neural Sampling Code (NSC) is new and interesting, even though I have not been completely convinced by this work presented (see below).  Nevertheless,  I put this paper as a \"marginal accept\" for the idea and the theoretical contribution.  \n\nNovel and substantial macaque V1 data.\n\nWEAKNESSES:\nThe authors employed a generative model approach to predict V1 neural responses, aiming to evaluate the Neural Sampling Code (NSC) using a quantitative metric.   However, the response prediction performance, as measured by correlation, was  poor compared to the standard state-of-the-art feedforward neural network system identification model. Hence, the virtue and the effectiveness of utilizing a generative model in this context remain unclear.  \n\nMore importantly,  it is unclear how this work confirms or refutes the NSC hypothesis. Moreover,  the earlier conceptual framework and theoretical formulation of the NSC propose that samples of population activities within a 10-20 msec window are used to approximate the posterior via Markov chain Monte Carlo (MCMC). In contrast, this work uses responses within a 120 msec duration throughout the entire stimulus presentation as a sample. Consequently, I find it hard to relate the current results to the original NSC idea. Additional arguments and explanations are necessary to establish the connections more convincingly.\n\nQUESTIONS:\nHow does the current work speak for NSC hypothesis? Does the result in this paper confirm or refute it?\n\nLIMITATIONS:\nNo discussion of societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a statistical framework to infer the generative models hold in the brain from population activity. This study first analytically proves that it is feasible to infer the generative model from population activity. Then, the study compares various combinations of prior and likelihood to fit synthetic and empirical data. They found that the flow-MLP model outperformed other models. In particular, the flow-MLP model allows to flexibly capture the true generative models.\n\nSTRENGTHS:\nOverall I am positive about this study. This study is well motivated, interesting, and technically sound. The discussion part is also inspiring. \n\n1. Neural sampling codes (NSC) is an important theory of population activity in the brain. This study definitely provides a practical framework that neural sampling codes can be useful in analyzing high-dimensional neural data.\n2. Using population activity to infer generative models is an excellent idea. This study is well motivated.\n3. The experiment is complete and neat. The study includes three priors x two likelihood functions, as well as a system identification model for comparison.\n\nWEAKNESSES:\n1. The writing can be improved. The overall texts are clear but some mathematical notations lack explicit explanations.\n2. The normalizing flow model needs more detail.\n3. More explanations are needed about in what situation this approach is useful in future empricial experiments. Or how experimentalistscan  design and perform better experiments to promote this line of research.\n\nQUESTIONS:\n1. The logic of neural sampling codes is very clear. I am not doing this line of research but I have know  the difference between neural sampling codes (NSC) and probablistic population codes (PPC) for a long time. More insights are needed to compare NSC and PPC. Is it also feasible to infer generative models under the framework of PPC. More discussions are needed.\n2. I am really confused by several mathematical notations here.\n   * In Eq1, I understand the meaning of r, x, z. but the footnote $r|x$ and $z|x$ are unclear to me. Given this is the key equation in this paper. \n   * Eq2, \"This subtle and seemingly trivial match implies that there exists a one-to-one mapping between the  space of neuronal responses and the latent variable underlying the stimulus\". This sentence is very confusing to me.\n   * Lines85-87, \"However, intriguingly, our formalization in equation 1 allows us to directly learn the generative model pz,x (z, x) by instead learning the probabilistic mapping between the neural responses and the stimuli\". I think this is they key logic. However, given the mathematical notations in Eq1&2 are confusing. This logic is really hard to follow.\n   * In Eqs.6&7, what are the $w$ and $v$? I assume they are parameters of the continous latent distribution? Better to clarify.\n3. In what situations is this approach useful. Can the authors give some cases where experimentalists can apply this approach to their data, or can design a better experiment to test relevant theories.\n\nLIMITATIONS:\nThis study overall is well done. Some notations should be better explained and more discussions are needed in terms of relevance to empirical experiments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}