Parametric Copula-GP model for analyzing multidimensional neuronal and behavioral relationships

1 INTRODUCTION . Recent advances in imaging and recording techniques have enabled monitoring the activity of hundreds to several thousands of neurons simultaneously ( Jun et al. , 2017 ; Helmchen , 2009 ; Dombeck et al. , 2007 ) . These recordings can be made in awake animals engaged in specifically designed tasks or natural behavior ( Stringer et al. , 2019 ; Pakan et al. , 2018a ; b ) , which further augments these already large datasets with a variety of behavioral variables . These complex high dimensional datasets necessitate the development of novel analytical approaches ( Saxena & Cunningham , 2019 ; Stevenson & Kording , 2011 ; Staude et al. , 2010 ) to address two central questions of systems and behavioral neuroscience : how do populations of neurons encode information ? And how does this neuronal activity correspond to the observed behavior ? In machine learning terms , both of these questions translate into understanding the high-dimensional multivariate dependencies between the recorded variables ( Kohn et al. , 2016 ; Shimazaki et al. , 2012 ; Ince et al. , 2010 ; Shamir & Sompolinsky , 2004 ) . There are two major methods suitable for recording the activity of large populations of neurons from behaving animals : the multi-electrode probes ( Jun et al. , 2017 ) , and calcium imaging methods ( Grienberger et al. , 2015 ; Helmchen , 2009 ; Dombeck et al. , 2007 ) that use changes in intracellular calcium concentration as a proxy for neuronal spiking activity at a lower temporal precision . While neuronal spiking occurs on a temporal scale of milliseconds , the behavior spans the timescales from milliseconds to hours and even days ( Mathis et al. , 2018 ) . As a result , the recorded neuronal and behavioral variables may operate at different timescales and exhibit different statistics , which further complicates the statistical analysis of these datasets . The natural approach to modeling statistical dependencies between the variables with drastically different statistics is based on copulas , which separate marginal ( i.e . single variable ) statistics from the dependence structure ( Joe , 2014 ) . For this reason , copula models are particularly effective for mutual information estimation ( Jenison & Reale , 2004 ; Calsaverini & Vicente , 2009b ) , which quantifies how much knowing one variable reduces the uncertainty about another variable ( Quiroga & Panzeri , 2009 ) . Copula models can also escape the ‘ curse of dimensionality ’ by factorising the multi-dimensional dependence into pair-copula constructions called vines ( Aas et al. , 2009 ; Czado , 2010 ) . Copula models have been successfully applied to spiking activity ( Onken et al. , 2009 ; Hu et al. , 2015 ; Shahbaba et al. , 2014 ; Berkes et al. , 2009 ) , 2-photon calcium recordings ( Safaai , 2019 ) and multi-modal neuronal datasets ( Onken & Panzeri , 2016 ) . However , these models assumed that the dependence between variables was static , whereas in neuronal recordings it may be dynamic or modulated by behavioral context ( Doiron et al. , 2016 ; Shimazaki et al. , 2012 ) . Therefore , it might be helpful to explicitly model the continuous time- or context-dependent changes in the relationships between variables , which reflect changes in an underlying computation . Here , we extend a copula-based approach by adding explicit conditional dependence to the parameters of the copula model , approximating these latent dependencies with Gaussian Processes ( GP ) . It was previously shown that such a combination of parametric copula models with GP priors outperforms static copula models ( Lopez-Paz et al. , 2013 ) and even dynamic copula models on many real-world datasets , including weather forecasts , geological data or stock market data ( HernándezLobato et al. , 2013 ) . Yet , this method has never been applied to neuronal recordings before . In this work , we increase the complexity of both marginal and copula models in order to adequately describe the complex dependencies commonly observed in neuronal data . In particular , we use conditional marginal models to account for changes of the single neuron statistics and mixtures of parametric copula models to account for changes in tail dependencies . We also improve the scalability of the method by using stochastic variational inference . We develop model selection algorithms , based on the fully-Bayesian Watanabe–Akaike information criterion ( WAIC ) . Finally and most importantly , we demonstrate that our model is suitable for estimating mutual information . It performs especially well when the parametric model can closely approximate the target distribution . When it is not the case , our copula mixture model demonstrates sufficient flexibility and provides close information estimates , comparable to the best state-of-the-art non-parametric information estimators . The goal of this paper is to propose and validate the statistical Copula-GP method , and illustrate that it combines multiple desirable properties for neuroscience applications : interpretability of parametric copula models , accuracy in density and information estimation and scalability to large datasets . We first introduce the copula mixture models and propose model selection algorithms ( Sec . 2 ) . We then validate our model on synthetic data and compare its performance against other commonly used information estimators ( Sec . 3 ) . Next , we demonstrate the utility of the method on real neuronal and behavioral data ( Sec . 4 ) . We show that our Copula-GP method can produce bivariate models that emphasize the qualitative changes in tail dependencies and estimate mutual information that exposes the structure of the task without providing any explicit cues to the model . Finally , we measure information content in the whole dataset with 5 behavioral variables and more than 100 neurons . 2 PARAMETRIC COPULA MIXTURES WITH GAUSSIAN PROCESS PRIORS . Our model is based on copulas : multivariate distributions with uniform marginals . Sklar ’ s theorem ( Sklar , 1959 ) states that any multivariate joint distribution can be written in terms of univariate marginal distribution functions p ( yi ) and a unique copula which characterizes the dependence structure : p ( y1 , . . . , yN ) = c ( F1 ( y1 ) . . . FN ( yN ) ) × ∏N i=1 p ( yi ) . Here , Fi ( · ) are the marginal cumulative distribution functions ( CDF ) . Thus , for each i , Fi ( yi ) is uniformly distributed on [ 0,1 ] . For high dimensional datasets ( dimy ) , maximum likelihood estimation for copula parameters may become computationally challenging . The two-stage inference for margins ( IFM ) training scheme is typically used in this case ( Joe , 2005 ) . First , univariate marginals are estimated and used to map the data onto a multidimensional unit cube . Second , the parameters of the copula model are inferred . Conditional copulas Following the approach by Hernández-Lobato et al . ( 2013 ) , we are using Gaussian Processes ( GP ) to model the conditional dependencies of copula parameters : p ( y|x ) = c ( F1 ( y1|x ) , . . . , FN ( yN |x ) ∣∣∣x ) × [ N∏ i=1 p ( yi|x ) ] . ( 1 ) In the most general case , the marginal PDFs p ( yi|x ) and CDFs Fi ( yi|x ) and the copula c ( . . . |x ) itself can all be conditioned on x . In our framework , x is assumed to be one-dimensional . A Gaussian Process is ideally suited for copula parametrization , as it provides an estimate of the uncertainty in model parameters , which we utilize in our model selection process ( Sec . 2.1 ) . Conditional marginals Previous works on conditional copulas ( Hernández-Lobato et al. , 2013 ) relied on the assumption that marginal distributions remain constant . This assumption might not hold for some real-life datasets , including neuronal recordings . Thus , we propose to use conditional marginal distributions instead . Note , that the interpretation of the copula model itself would depend on the assumptions made for the marginal distributions ( see Appx . C.1 for further discussion ) . In order to estimate marginal CDFs F ( yi|x ) , we use the non-parametric fastKDE ( O ’ Brien et al. , 2016 ) algorithm , which allows for direct estimation of the conditional distributions . These CDFs are then used to map the data onto a unit hypercube using the probability integral transform : F ( yi|x ) → ui ∼ u [ 0,1 ] , such that ui is uniformly distributed for any x. Bivariate copula families We use 4 copula families as the building blocks for our copula models : Gaussian , Frank , Clayton and Gumbel copulas ( Fig . 1 ) . All of these families have a single parameter , corresponding to the rank correlation ( Table 1 ) . We also use rotated variants ( 90◦ , 180◦ , 270◦ ) of Clayton and Gumbel copulas in order to express upper tail dependencies and negative correlation . Since we are primarily focused on the analysis of neuronal data , we have first visualized the dependencies in calcium signal recordings after a probability integral transform , yielding empirical conditional copulas . As a distinct feature in neuronal datasets , we observed changes in tail dependencies with regard to the conditioning variable . Since none of the aforementioned families alone could describe such conditional dependency , we combined multiple copulas into a linear mixture model : c ( u|x ) = ∑M j=1 φj ( x ) cj ( u ; θj ( x ) ) , where M is the number of elements , φj ( x ) is the concentration of the jth copula in a mixture , cj is the pdf of the jth copula , and θj is its parameter . Each of the copula families includes the Independence copula as a special case . To resolve this overcompleteness , we add the Independence copula as a separate model with zero parameters ( Table 1 ) . For independent variables yind , the Independence model will be preferred over the other models in our model selection algorithm ( Sec . 2.1 ) , since it has the smallest number of parameters . Gaussian Process priors We parametrize each copula in the mixture model with an independent latent GP : f ∼ N ( µ × 1 , Kλ ( x , x ) ) . For each copula family , we constructed GPLink functions ( Table 1 ) that map the GP variable onto the copula parameter domain : θj = GPlinkcj ( fj ) , R → dom ( cj ) .Next , we also use GP to parametrize concentrations φj ( x ) , which are defined on a simplex ( ∑ φ = 1 ) : φj = ( 1− tj ) j−1∏ m=1 tm , tm = Φ ( f̃m + Φ −1 ( M −m− 1 M −m ) ) , tM = 0 , where Φ is a CDF of a standard normal distribution and f̃m ∼ N ( µ̃m × 1 , K̃λ̃m ( x , x ) ) . We use the RBF kernel Kλ ( x , x ) with bandwidth parameter λ . Therefore , the whole mixture model with M copula elements is parameterized by [ 2M −1 ] independent GPs and requires [ 2M −1 ] hyperparameters : { λ } M for θ and { λ̃ } M−1 for φ . Approximate Inference Since our model has latent variables with GP priors and intractable posterior distribution , direct maximum likelihood Type-II estimation is not possible and an approximate inference is needed . Such an inference problem with copula models has previously been solved with the expectation propagation algorithm ( Hernández-Lobato et al . ( 2013 ) , see comparison in Appx . B.4 ) , which was not suitable for large scale data . Recently , a number of scalable approximate inference methods were developed , including stochastic variational inference ( SVI ) ( Titsias , 2009 ; Cheng & Boots , 2017 ) , scalable expectation propagation ( SEP ) ( Hernández-Lobato & HernándezLobato , 2016 ) , and MCMC based algorithms ( Hensman et al. , 2015 ) , as well as a scalable exact GP ( Wang et al. , 2019 ) . We chose to use SVI due to availability of the well-established GPUaccelerated libraries : PyTorch ( Paszke et al. , 2017 ) and GPyTorch ( Gardner et al. , 2018 ) .

Summary: This manuscript models the conditional joint distribution over variables by using Copula models and copula vines.  The experimental data shows that when the observed variables are highly correlated that the proposed approach improves estimation of entropy over competing benchmark approaches (MINE and KSG) when the variables are highly correlated. Synthetic results demonstrate good improvements, and application to real scientific data seems promising.