Author response: Modelling the neural code in large populations of correlated neurons
2021-07-21

Article Figures and data Abstract Introduction Results Discussion Materials and methods Appendix 1 Appendix 2 Appendix 3 Appendix 4 Data availability References Decision letter Author response Article and author information Metrics Abstract Neurons respond selectively to stimuli, and thereby define a code that associates stimuli with population response patterns. Certain correlations within population responses (noise correlations) significantly impact the information content of the code, especially in large populations. Understanding the neural code thus necessitates response models that quantify the coding properties of modelled populations, while fitting large-scale neural recordings and capturing noise correlations. In this paper, we propose a class of response model based on mixture models and exponential families. We show how to fit our models with expectation-maximization, and that they capture diverse variability and covariability in recordings of macaque primary visual cortex. We also show how they facilitate accurate Bayesian decoding, provide a closed-form expression for the Fisher information, and are compatible with theories of probabilistic population coding. Our framework could allow researchers to quantitatively validate the predictions of neural coding theories against both large-scale neural recordings and cognitive performance. Introduction A foundational idea in sensory neuroscience is that the activity of neural populations constitutes a ‘neural code’ for representing stimuli (Dayan and Abbott, 2005; Doya, 2007): the activity pattern of a population in response to a sensory stimulus encodes information about that stimulus, and downstream neurons decode, process, and re-encode this information in their own responses. Sequences of such neural populations implement the elementary functions that drive perception, cognition, and behaviour (Pitkow and Angelaki, 2017). Therefore, by studying the encoding and decoding of population responses, researchers may investigate how information is processed along neural circuits, and how this processing influences perception and behaviour (Wei and Stocker, 2015; Panzeri et al., 2017; Kriegeskorte and Douglas, 2018). Given a true statistical model of how a neural population responds to (encodes information about) stimuli, Bayes’ rule can transform the encoding model into an optimal decoder of stimulus information (Zemel et al., 1998; Pillow et al., 2011). However, when validated as Bayesian decoders, statistical models of neural encoding are often outperformed by models trained to decode stimulus-information directly, indicating that the encoding models miss key statistics of the neural code (Graf et al., 2011; Walker et al., 2020). In particular, the correlations between neurons’ responses to repeated presentations of a given stimulus (noise correlations), and how these noise correlations are modulated by stimuli, can strongly impact coding in neural circuits (Zohary et al., 1994; Abbott and Dayan, 1999; Sompolinsky et al., 2001; Ecker et al., 2016; Kohn et al., 2016; Schneidman, 2016), especially in large populations of neurons (Moreno-Bote et al., 2014; Montijn et al., 2019; Bartolo et al., 2020; Kafashan et al., 2021; Rumyantsev et al., 2020). Statistically validating theories of population coding in large neural circuits thus depends on encoding models that support accurate Bayesian decoding, effectively capture noise-correlations, and efficiently fit large-scale neural recordings. There are at least two classes of neural recordings for which established models have facilitated such analyses. Firstly, for recordings of binary spike-counts, pairwise-maximum entropy models (Schneidman et al., 2006; Lyamzin et al., 2010; Granot-Atedgi et al., 2013; Tkačik et al., 2013; Meshulam et al., 2017; Maoz et al., 2020) have been used to investigate the structure of the retinal code (Ganmor et al., 2015; Tkačik et al., 2015). Secondly, when modelling dynamic spike-train recordings, generalized linear models (GLMs) have proven effective at modelling spatio-temporal features of information processing in the retina and cortex (Pillow et al., 2008; Park et al., 2014; Runyan et al., 2017; Ruda et al., 2020). Nevertheless, many theories of neural coding are formulated in terms of unbounded spike-counts (Ma et al., 2006; Beck et al., 2011a; Ganguli and Simoncelli, 2014; Makin et al., 2015; Yerxa et al., 2020), rather than the binary spike-counts of pairwise maximum entropy models. Furthermore, neural correlations are often low-dimensional (Arieli et al., 1996; Ecker et al., 2014; Goris et al., 2014; Rabinowitz et al., 2015; Okun et al., 2015; Semedo et al., 2019), in contrast with the correlations that result from the fully connected, recurrent structure of standard GLMs. Although there are extensions of the GLM approach that capture shared-variability (Vidne et al., 2012; Archer et al., 2014; Zhao and Park, 2017), they seem unable to support exact Bayesian decoding. Similarly, methods such as factor analysis that model unbounded spike-counts as continuous variables have proven highly effective at modelling neural correlations in large-scale recordings (Yu et al., 2009; Cunningham and Yu, 2014; Ecker et al., 2014; Semedo et al., 2019), yet it is also unknown if they can support accurate Bayesian decoding. Towards modelling spike-count responses and accurate Bayesian decoding in large populations of correlated neurons, we develop a class of encoding model based on finite mixtures of Poisson distributions. Within neuroscience, Poisson mixtures are widely applied to modelling the spike-count distributions of individual neurons (Wiener and Richmond, 2003; Shidara et al., 2005; Goris et al., 2014; Taouali et al., 2016). Outside of neuroscience, mixtures of multivariate Poisson distributions are an established model of multivariate count distributions that effectively capture correlations in count data (Karlis and Meligkotsidou, 2007; Inouye et al., 2017). Building on the theory of exponential family distributions (Wainwright and Jordan, 2008; Macke et al., 2011b), our model extends previous mixture models of multivariate count data in two ways. Firstly, we develop a tractable extension of Poisson mixtures that captures both over- and under-dispersed response variability (i.e. where the response variance is larger or smaller than the mean, respectively) based on Conway-Maxwell Poisson distributions (Shmueli et al., 2005; Stevenson, 2016). Secondly, we introduce an explicit dependence of the model on a stimulus variable, which allows the model to accurately capture changes in response statistics (including noise correlations) across stimuli. Importantly, the resulting encoding model affords closed-form expressions for both its Fisher information and probability density function, and thereby a rigorous quantification of the coding properties of a modelled neural population (Dayan and Abbott, 2005). Moreover, the model learns low-dimensional representations of stimulus-driven neural activity, and we show how it captures a fundamental property of population codes known as information-limiting correlations (Moreno-Bote et al., 2014; Montijn et al., 2019; Bartolo et al., 2020; Kafashan et al., 2021; Rumyantsev et al., 2020). We apply our mixture model framework to both synthetic data and recordings from macaque primary visual cortex (V1), and demonstrate that it effectively models responses of populations of hundreds of neurons, captures noise correlations, and supports accurate Bayesian decoding. Moreover, we show how our model is compatible with the theory of probabilistic population coding (Zemel et al., 1998; Pouget et al., 2013), and could thus be used to study the theoretical coding properties of neural circuits, such as their efficiency (Ganguli and Simoncelli, 2014), linearity (Ma et al., 2006), or information content (Moreno-Bote et al., 2014). Results A critical part of our theoretical approach is based on expressing models of interest in exponential family form. An exponential family distribution p⁢(n) over some data n (in our case, neural responses) is defined by the proportionality relation p(n)∝eθ⋅s(n)b(n), where θ are the so-called natural parameters, s⁢(n) is a vector-valued function of the data called the sufficient statistic, and b⁢(n) is a scalar-valued, non-negative function called the base measure (Wainwright and Jordan, 2008). The exponential family form allows us to modify and extend existing models in a simple and flexible manner, and derive analytical results about the coding properties of our models. We demonstrate our approach with applications to both synthetic data, and experimental data recorded in V1 of anaesthetized and awake macaques viewing drifting grating stimuli at different orientations (for details see Materials and methods). Extended Poisson mixture models capture spike-count variability and covariability Our first goal is to define a class of models of stimulus-independent, neural population activity, that model neural activity directly as spike-counts, and that accurately capture single-neuron variability and pairwise covariability. We base our models on Poisson distributions, as they are widely applied to modelling the trial-to-trial distribution of the number of spikes generated by a neuron (Dayan and Abbott, 2005; Macke et al., 2011a). We will also generalize our Poisson-based models with the theory of Conway-Maxwell (CoM) Poisson distributions (Sur et al., 2015; Stevenson, 2016; Chanialidis et al., 2018). The two-parameter CoM-Poisson model contains the one-parameter Poisson model as a special case, however, whereas the Poisson model always has a Fano factor (FF; the variance divided by the mean) of 1, the CoM-Poisson model can exhibit both over- (FF>1) and under-dispersion (FF<1), and thus capture the broader range of Fano factors observed in cortex (Stevenson, 2016). The other key ingredient in our modelling approach are mixtures of Poisson distributions, which have been used to model complex spike-count distributions in cortex, and also allow for over-dispersion (Shidara et al., 2005; Goris et al., 2014; Taouali et al., 2016; Figure 1A). In our case, we mix multiple, independent Poisson distributions in parallel, as such models can capture covariability in count data as well (see Karlis and Meligkotsidou, 2007 for a more general formulation of multivariate Poisson mixtures than what we consider here). To construct such a model, we begin with a product of independent Poisson distributions (IP distribution), one per neuron. We then mix a finite number of component IP models, to arrive at a multivariate spike-count, finite mixture model (see Materials and methods). Importantly, although each component of this mixture is an IP distribution, randomly switching between components induces correlations between the neurons (Figure 1B,C). Figure 1 Download asset Open asset Poisson mixtures and Conway-Maxwell extensions exhibit spike-count correlations, and over- and under-disperson. (A) A Poisson mixture distribution (red), defined as the weighted sum of three component Poisson distributions (black; scaled by their weights). FF denotes the Fano Factor (variance over mean) of the mixture. (B, C) The average spike-count (rate) of the first and second neurons for each of 13 components (black dots) of a bivariate IP mixture, and 68% confidence ellipses for the spike-count covariance of the mixture (red lines; see Equations 6 and 7). The spike-count correlation of each mixture is denoted by r. (D) Same model as A, except we shift the distribution by increasing the baseline rate of the components. (E, F) Same model as A, except we use an additional baseline parameter based on Conway-Maxwell Poisson distributions to concentrate (E) or disperse (F) the mixture distribution and its components. IP mixtures can in fact model arbitrary covariability between neurons (see Materials and methods, Equation 7); however, they are still limited because the model neurons in an IP mixture are always over-dispersed. To overcome this, it is helpful to consider factor analysis (FA), which is widely applied to modelling neural population responses (Cunningham and Yu, 2014). IP mixtures are similar to FA, in that FA represents the covariance matrix of neural responses as the sum of a diagonal matrix that helps capture individual variance, and a low-rank matrix that captures covariance (see Bishop, 2006), and FA and IP mixtures can be fine-tuned to capture covariance arbitrarily well. However, whereas FA has distinct parameters for representing means and diagonal variances, the means and variances in an IP mixture are coupled through shared parameters (see Materials and methods, Equation 6). Our strategy will thus be to break this coupling between means and variances by granting IP mixtures an additional set of parameters based on the theory of CoM-Poisson distributions. To do so, we first show how to express an IP mixture as the marginal distribution of an exponential family distribution. Note that an IP mixture with dK components may be expressed as a latent variable model over spike-count vectors n and latent component-indices k, where 1≤k≤dK. In this formulation we denote the kth component distribution by p(n∣k), and the probability of realizing (switching to) the k th component by p⁢(k). The mixture model over spike-counts n is then expressed as the marginal distribution p(n)=∑k=1dKp(n∣k)p(k)=∑k=1dKp(n,k), of the joint distribution p(n,k). Under mild regularity assumptions (see Materials and methods), we may reparameterize this joint distribution in exponential family form as (1) p(n,k)∝eθN⋅n+θK⋅δ(k)+n⋅ΘNK⋅δ(k)∏i=1dNni!, where the vectors θN and θK, and matrix ΘNK are the natural parameters of p(n,k), and δ(k)=(δ2(k),...,δdK(k)) is the Kronecker delta vector defined by δj⁢(k)=1 if j=k, and 0 otherwise. This representation affords an intuitive interpretation. In general, the natural parameters of an IP distribution are the logarithms of the average spike-counts (firing rates), and the natural parameters of the first component distribution p(n∣k=1) of an IP mixture are simply θN. The natural parameters of the k th component for k>1 are then the sum of the ‘baseline’ parameters θN and column k-1 from the matrix of parameters ΘN⁢K (Equation 13, Materials and methods). Because the dimension of the baseline parameters θN is much smaller than the total number of parameters in a given mixture, the baseline parameters provide a relatively low-dimensional means of affecting all the component distributions of the given mixture, as well as the probability distribution over indices p⁢(k) (Figure 1D; see Materials and methods, Equation 12 for how the index-probabilities p⁢(k) depend on θN). We next extend Equation 1 with the theory of CoM-Poisson distributions, and define the latent variable exponential family (2) p(n,k)∝eθN⋅n+θN∗⋅lf(n)+θK⋅δ(k)+n⋅ΘNK⋅δ(k), where lf⁢(n)=(log⁡(n1!),...,log⁡(ndN!)) is the vector of log-factorials of the individual spike-counts, and θN∗ are a set of natural parameters derived from CoM-Poisson distributions (see Materials and methods). Based on this construction, each component p⁢(n∣k) is a product of independent CoM-Poisson distributions, and when θN∗=−1, we recover an IP mixture defined by Equation 1 with parameters θN, θK, and ΘN⁢K. The first component of this model p(n∣k=1) has parameters θN and θN∗, and as with the IP mixture, the parameters θN are translated by column k-1 of ΘNK when k>1. However, the parameters θN∗ are never translated, and remain the same for each component distribution (Equation 16, Materials and methods, and see Equation 15 for formulae for the index-probabilities p⁢(k)). We refer to models defined by Equation 2 as CoM-based (CB) mixtures, and θN∗ as CB parameters. Due to the addition of the CB parameters θN∗, a CB mixture breaks the coupling between the spike-count means and variances that is present in the simpler IP mixture (Equation 17, Materials and methods). In Figure 1D–F, we demonstrate how changing the parameters of a CB mixture can concentrate or disperse both the mixture distribution and its components, and that a CB mixture can indeed exhibit under-dispersion. To validate our mixture models, we tested if they capture variability and covariability of V1 population responses to repeated presentations of a grating stimulus with fixed orientation (dN=43 neurons and dT=355 repetitions of 150 ms duration in one awake macaque; dN=70 and dT=1,200 of duration 70 ms in one anaesthetized macaque). We fit our mixtures to the complete datasets with expectation-maximization (EM, a standard choice for training finite mixture models [McLachlan et al., 2019] see Materials and methods). The CB mixture accurately captured single-neuron variability (Figure 2A–B, red symbols), including both cases of over-dispersion and under-dispersion. On the other hand, the simpler IP mixture (Figure 2A–B, blue symbols) cannot accommodate under-dispersion due to its mathematical limits, and demonstrated limited ability to model over-dispersion due to the coupling between the mean and variance (Equation 6). Figure 2 Download asset Open asset CoM-based parameters help Poisson mixtures capture individual variability in V1 responses to a single stimulus. We compare Independent Poisson (IP) mixtures (Equation 1) and CoM-Based (CB) mixtures (Equation 2) on neural population responses to stimulus orientation x=20∘ in V1 of awake (dN=43 neurons and dT=355 trials) and anaesthetized (dN=70 and dT=1,200) macaques; both mixtures are defined with dK=5 components for both data sets (see Materials and methods for training algorithms). A,B: Empirical Fano factors of the awake (A) and anaesthetized data (B), comparing IP (blue) and CB mixtures (red). C,D: Histogram of the CB parameters θN∗ for the CB mixture fits to the awake (C) and anaesthetized (D) data. Values of θN∗<−1 denote under-dispersed mixture components, values >-1 denote over-dispersed components. To understand how the CB parameters allow the CB mixture to overcome the limits of the IP mixture, we plot a histogram of the CB parameters θN∗ for both fits (Figure 2C–D). If the CB parameter of a given CoM-Poisson distribution is <-1, >-1, or =-1, then the CoM-Poisson distribution is under-dispersed, over-dispersed, or Poisson-distributed, respectively. When a CB mixture is fit to the awake data (Figure 2C), we see that it learns a range of values for the CB parameters around −1, to accommodate the variety of Fano factors observed in the awake data (Figure 2A). On the anaesthetized data, even though IP mixtures can capture over-dispersion, the IP mixture underestimates the dispersion of neurons due to the coupling between the mean and variance (Figure 2B). The CB mixture thus uses the CB parameters to further disperse its model neurons (Figure 2D). In contrast with individual variability, we found that both mixture models were flexible enough to qualitatively capture pairwise noise correlation structure in both awake and anaesthetized animals (Figure 3A–B), and that the distributions of modelled neural correlations were broadly similar when compared to the data (Figure 3C–D). In Appendix 1, we rigorously compare IP mixtures, CB mixtures, and FA on our datasets, and show that although FA is better than our mixture models at capturing second-order statistics in training data, IP mixtures and CB mixtures achieve comparable predictive performance as FA when evaluated on held-out data. Figure 3 Download asset Open asset IP and CB mixtures effectively capture pairwise covariability in V1 responses to a single stimulus. Here we analyze the pairwise statistics of the same models from Figure 2. (A, B) Empirical correlation matrix (upper right triangles) of awake (A) and anaesthetized data (B), compared to the correlation matrix of the corresponding IP mixtures (lower left triangles). (C, D) Noise correlations highlighted in A and B, respectively. (E, F) Highlighted noise correlations for CB mixture fit. (G,H) Histogram of empirical noise correlations, and model correlations from IP and CB mixtures. Extended Poisson mixture models capture stimulus-dependent response statistics So far, we have introduced the exponential family theory of IP and CB mixtures, and shown how they capture response variability and covariability for a fixed stimulus. To allow us to study stimulus encoding and decoding, we further extend our mixtures by inducing a dependency of the model parameters on a stimulus. When there are a finite number of stimulus conditions and sufficient data, we may define a stimulus-dependent model with a lookup table, and fit it by fitting a distinct model at each stimulus condition. However, this is inefficient when the amount of data at each stimulus-condition is limited and the stimulus-dependent statistics have structure that is shared across conditions. A notable feature of the exponential family parameterizations in Equations 1 and 2 is that the baseline parameters influence both the index probabilities and all the component distributions of the model. This suggests that by restricting stimulus-dependence to the baseline parameters, we might model rich stimulus-dependent response structure, while bounding the complexity of the model. In general, we refer to any finite mixture with stimulus-dependent parameters as a conditional mixture (CM), and depending on whether the CM is based on Equations 1 and 2 and, we refer to it as an IP- or CB-CM, respectively. Although there are many ways we might induce stimulus-dependence, in this paper we consider two forms of CM: (i) a maximal CM, which we implement as a lookup table, such that all the parameters in Equations 1 and 2 and depend on the stimulus, and (ii) a minimal CM, for which we restrict stimulus-dependence to the baseline parameters θN. This results in the CB-CM (3) p(n,k∣x)∝eθN(x)⋅n+θN∗⋅lf(n)+θK⋅δ(k)+n⋅ΘNK⋅δ(k), where x is the stimulus, and θN(x) are the stimulus-dependent baseline parameters, and we recover a minimal, IP-CM by setting θN∗=−1. The IP-CM again affords an intuitive The first component of an IP-CM has stimulus-dependent natural parameters and thus the stimulus-dependent or of the th neuron given is where is the th of The natural parameters of the k th component for k>1 are then the sum of θN(x) and column k-1 of ΘN⁢K. given the of the th neuron is a of where the is the exponential function of of column k-1 of ΘN⁢K (see Equation 13, Materials and methods). a CB-CM this still as an of the behaviour of its components (see Equation and the Towards the of we study a minimal, CB-CM with neurons, dK=5 mixture components, and randomly parameters (see Materials and methods). Moreover, we that the stimulus is the orientation of a and that the of the component distributions have a which is a widely applied model of neural to stimuli et al., 2017). We may achieve such a by the stimulus-dependent baseline parameters as where and are parameters, and Figure that the of the CB-CM neurons are yet many also exhibit Figure 4 Download asset Open asset a conditional mixture We compare a CB-CM with neurons, mixture components, components, and parameters to a CB-CM fit to from the A,B: of the CB-CM (A) and CB-CM are highlighted for C,D: The index probabilities of the CB-CM (C) and CB-CM where component used in and (E, F) The correlation matrix of the CB-CM (upper compared to the correlation matrix of the CB-CM (lower at stimulus orientations (E) and The of the CB-CM compared to the CB-CM at orientations and (red triangles). We also study if can be effectively fit to datasets comparable to in We generated responses from the CB-CM the CB-CM to each of orientations over the for a total of We then used this data to fit a CB-CM with the same number of components. this we derive an to model parameters (see Materials and methods). Figure that the of the CB-CM are from of the CB-CM (Figure of To the latent structure of the model, in Figure we plot the index probability for k as a function of the orientation In Figure we show that the index probabilities of the CB-CM qualitatively the true index probabilities in Figure We also that although the CB-CM the indices this has on the performance of the The index-probabilities provide a of how the complexity and structure of model correlations with the The in Figure denote two orientations that different index probabilities When a large number of index-probabilities are the of the CB-CM can exhibit complex correlations with both and values (Figure However, when one index the correlation structure (Figure In Figure we show that the also depend on stimulus we that both the FF and the of the CB-CM are from the CB-CM (Figure In our show that minimal can express stimulus-dependent response and that we can recover the structure of a CB-CM from of synthetic data with In the we rigorously the performance of on our awake and anaesthetized mixtures effectively model neural responses in macaque V1 A variety of models may be defined within the CM framework by Equations 1, 2 and Towards how effectively can model data, we compare different by their on both our awake and anaesthetized this is the same data used in Figures 2 and 3 including all We consider both IP and CB of each of the conditional (i) maximal where we a distinct mixture for each of stimulus (ii) minimal with components, and minimal with components given by where is the Kronecker delta vector with and x is the index of the stimulus. In contrast with the CM, the CM assumptions about the form of component In 1 we the number of parameters for all forms of 1 of CM models. is number of parameters in IP models, second is number of additional parameters in CB extensions of IP models, as a function of number of stimuli neurons and mixture components parameter CB To provide an measure of the performance of each CM we define the as the between the of the given CM and the of a independent Poisson model, which is a standard model of neural responses to stimuli et al., 2017). We then the predictive performance of our models with of the information 2 that the CM achieve comparable and better than the independent Poisson on both the awake and anaesthetized data. Figure that a performance as the model complexity of is In all the CB models their IP and with parameters. The achieve performance on both In well on the anaesthetized data more on the awake data, and maximal exhibit the Nevertheless, a more statistical as they also between stimulus and may still even where performance is On the other hand, even though maximal achieve they simply do by the performance of mixtures