Efficient approximation of neural population structure and correlations with probabilistic circuits
Authors: Koosha Khalvati
Samantha Johnson
Stefan Mihalas
Michael A Buice
Venue: ICLR 2023 poster (2023)
Keywords: Computational neuroscience
Probabilistic circuits
Sum Product networks
AI for science

Abstract:  We present a computationally efficient framework to model a wide range of population structures with high order correlations and a large number of neurons. Our method is based on a special type of Bayesian network that has linear inference time and is founded upon the concept of contextual independence. Moreover, we use an efficient architecture learning method for network selection to model large neural populations even with a small amount of data. Our framework is both fast and accurate in approximating neural population structures. Furthermore, our approach enables us to reliably quantify higher order neural correlations. We test our method on simulated neural populations commonly used to generate higher order correlations, as well as on publicly available large-scale neural recordings from the Allen Brain Observatory. Our approach significantly outperforms other models both in terms of statistical measures and alignment with experimental evidence.

Published as a conference paper at ICLR 2023
EFFICIENT APPROXIMATION OF NEURAL POPULATION STRUCTURE AND CORRELATIONS WITH PROBABILISTIC CIRCUITS
Koosha Khalvati
Samantha Johnson
Allen Institute, Seattle, WA 98109 koosha ahaalvati@@lleennssttttttt
University of Chicago, Chicago, IL 60637
enjjonao@uchicago..
Stefan Mihalas
Michael A. Buice
Allen Institute, Seattle, WA 98109
Allen Institute, Seattle, WA 98109 mi www.hhlllllllleennsntttteeeeeeee
stefanm@alleniistittuee org
ABSTRACT
We present a computationally efficient framework model a wide range of popu- lation structures with high order correlations and a large number of neurons. Our method is based on special type Bayesian network that has linear inference time and is founded upon the concept of contextual independence. Our framework is both fast and accurate in approximating neural population structures. Furthermore, our approach enables us to reliably quantify higher order neural correlations. We test our method on simulated neural populations commonly used to generate higher order correlations, from the Allen Brain Observatory. Our approach significantly outperforms other models both in terms of statistical measures and alignment with experimental evidence.
well
publicly available large-scale neural recordings
INTRODUCTION
With the rise and fast growth of simultaneous neural population recording, modeling population
structures and measuring correlations has become focus computational neuroscience (Abbott)
& Dayan 1999; Averbeck et al., 2006 Azeredo da Silveira & Ricke 2021 Urai et al. 2022).
Theoretical and Experimental works have demonstrated the necessity of measuring population correlations to investigate information coding (Moreno-Bore et al.) (014 Averbeck et al. 2006).
functional connectivity (Dunn et al., 2015),, learning (Ganmor et al. 2011). and arousal (Vinck et al. 2015| al. 2016). Despite significant progress in recent years, research measurement and analysis of population correlations still faces significant challenges (Kohn et al. 2016).
Exact measurement of population correlations an NP-hard problem in the general case since requires computing every form of dependency among spiking neurons. As a result, researchers have tried come up with computationally efficient ways of approximation neural correlations. Existing approaches are energy based models rooted in statistical mechanics where the energy function incorporates couplings between subsets of variables (here neurons) (Roudi) et al. 2009c; Tkačik et al., 2006 SSoh--Dickstein et al. 2011 Aurell & Ekeberg 2012). However, these methods often carry auxiliary (and even unrealistic) assumptions about the neural dynamics and do not scale up for large populations (Roudi al. 2009b).
indirect measurement of
Notably, generative models commonly used other domains such as latent variable methods are often not applicable to neural populations as spiking neural data is discrete and sparse (Zhao et al.) 2020). Furthermore, various parameters such as behavioural and emotional state of the animal affect firing patterns of neurons even in sensory cortex (Urai et al., (022). As a result, recording long enough to train these models contains many external variable changes and confounding factors that make drawing seientific conclusions difficult.

---

Published as a conference paper at ICLR 2023
It is important to distinguish whether modeling population correlations is important for predicting joint activity (called encoding) vs whether they are a channel for down-stream information flow (call decoding) (Pillow et al. 2008). In the encoding paradigm, one needs to know whether it is important to include certain correlation in a generative model of the spiking activity. In the decoding paradigm, one can, e.g., compare classifier's stimulus prediction accuracy on original and shuffled neural data to assess population correlations (Averbeck et al. [006] Pillow et al. 20088 Christensen & Pillow 2022, Runyan et al. 2017). Importantly, the type of used classifier significantly affects the result with no confirmation that the brain utilizes similar strategy /Averbeck et al. (2006).
Here we work in the encoding paradigm and take a probabilistic approach by modeling the joint probability distribution of neural activity with Bayesian networks. Inference is NP-hard in general Bayesian networks (Cooper 1990), making them impractical to model the population structure. Therefore, we utilize a special family of Bayesian networks with linear inference time, first introduced as "Arithmetic circuits" (Darwiche 2003 [Shen et al. 2016). This family of networks has been designed to take advantage of "context-specific independence" of variables mainly for the purpose of computational efficacy, which also makes it suitable to extract local structures in the data (Boutilier et al. 1996; [Shen et al. 2020). We use modification (and equivalent (Rooshenas & Lowd 2014)) of arithmetic circuits. Sum-Product Networks (SPNs). SPNs are more known and used by the community (Poon & Domingos 2011 Sanchez-Cauce al. (02T).
In particular, we adapt sum- con-prrduct networks to fit spiking neural data in order to capture a wide range of population correlation/strucuure from local global in polynomial time. Due the efficiency of architecture learning and inference in SPNs, population structure estimation is polynomial in the size of the population. In addition, we suggest a measure of high order population correlations based on our framework. Our results include fitting on neural population simulations constructed with higher order correlations, as well as large-scale neural recording in different brain regions on more than 20 mice. Our framework outperforms both energy- based and latent variable models for neural population structure estimation.
PROBLEM DEFINITION AND RELATED WORK
One of the critical problems in computational neuroscience is providing an accurate statistical description of spike trains in population of neurons. As the full representation of the data, i.c. raw spike times, is high dimensional, spike trains are binned into small time windows. The time bin should be short enough so each neuron spikes at most once in each bin (with some amount of tolerance in potentially losing some spikes). In addition, this time bin should be large enough that the assumption of temporal independence of spikes holds. With this time binning strategy, each neuron's 's activity is a binary variable (S, for neuron i is equal to 1 if there a spike in the corresponding bin, otherwise 0) and each time bin represents an i.i.d sample/instance. Therefore, spike trains of N neurons for the duration of T would be represented as a binary matrix D where K = T/At in which At is the bin length (Figure[ left plot). Consequently, the population activity has probabilistic representation P(SI.... Sw), ). and the problem turns into modelling this joint distribution, given the data. More specifically, the problem is to find a model m² from a family of models M, and optimize its free
parameters Om so as to satisfy the following:
m 8 arg max
Σ log (P(d 8))
(1)
m.M.
In the existing approaches. M is set to maximum entropy (Ising) models, which are energy- methods rooted in statistical mechanies (Roudi et al.) 2009c] Schneidman et al., (006). Since learning maximum entropy models is computationally very expensive, these models are restricted to estimate
the statistical properties of the population up to a constant order. However, going beyond second order is not computationally feasible. In fact, building the exact generative model is an intractable problem in the general case even for the second order (pairwise) correlations.
Therefore, even Pairwise Maximum Entropy (PME) model requires further estimation where more accurate approximation algorithms requires thousands of samples for each pair, making them impossible to be used for large populations of neurons (Roudi et al. 2009c] TTačik et al., (006 Sohl-Dickstein et al. 201T] Aurell& & Ekeberg 2012). Moreover, there exist plausible scenarios, such dichotomized common input to loosely coupled neurons, in which pairwise correlations
2

---

Published as a conference paper at ICLR 2023
negligible compared to higher order correlations (Macke al. 2011 Amari al. 2003). In fact, low order correlations do not necessarily capture the population structure even when the structure can be explained by "simple" models (Beretta et al. 2018 de Mulatier et al. 202I). Overall, PME is neither computationally efficient nor accurate in population structure estimation.
To improve the structure estimation, some methods modelled higher order correlations through the energy based approaches, e.g. "k-pairwise" correlations (aaiaaaiiinnnnnkkkkk Tkačik et al., 2014). Restricted Boltzmann Machines (RBMs) on top of pair-wised correlations (semi RBMS (sRBMs)) (Köster al., 2014). and sparse low-order correlations (Ganmor et al. 201T). Founded upon energy based models, these methods also needs computationally expensive sampling strategies to estimate the partition function. Moreover, to approximate the partition function, these methods often rely on aseudo-likelihon which needs extensive amount of data points to accurate.
Here we use special type of Bayesian network with linear inference, and consequently fast learning time, to estimate distributions of joint neural activity. Our network approximates these distributions by ignoring correlations based on their effect size, rather than statistical order. Moreover, the normalization process to obtain real probability values also linear in the size of the network. Thus, it is significantly more successful than energy- besgy-baased models both in terms of efficiency and accuracy.
3 MODEL
Our framework is based on Sum-Product Networks (SPNs). An SPN is rooted directed acyclic graph representing a joint probability distribution of given variables. This distribution is the result of a hierarchical combination of alternating mixtures (sum nodes) and factorizations (product nodes), with given variables as the leaf nodes of the network (Poon & Domingos 2011 (Sanchez- et al.) 2021). Specifically, each leaf node (a node with no children) represents univariate probability distribution. When the variable is categorical, the Icaf node is a variable indicator (I(z)). The scope of each leaf node is singleton where its element is the variable that the leaf represents. Other nodes are either sum or product. A product node represents the product of its children (connected by a link). A sum node represents the (normalized) weighted sum of its children. The weight of each child is shown as the label of the link from the sum node to that child. The scope of sum and product nodes is one-node SPN, sum product networks
the union of scopes of their children. Starting from a leaf can be built bottom- up by combining smaller SPNs through sum or product nodes. The root of the SPN represents a valid joint probability distribution the SPN is complete and decomposable. An SPN is complete iff all children of each sum node have the same scope. A sum-product network is decomposable iff all children of each product node have disjoint scopes.
To model the joint probability distribution of the given variables, the weights on the links of sum nodes should be learned with learning algorithm such as gradient descent or Dnpectatio-Maximizaaie (EM), given the data (Sanchez-Cauce et al., 2021). Similar to other Bayesian networks, the structure of the graph needs to be determined beforehand. One approach is to use a random dense graph and rely on the parameter learning algorithm to select important components. This strategy requires data sets orders of magnitudes larger than that gathered in neural experiments. Network structure can also be learned from the data directly based the general computational properties of sum and product nodes (Gens & Pedro 2013). Sum nodes represent the sum of probability distributions over the same set of variables (see completeness above). Therefore, they explain the data best if their child distributions represent disjoint states, or in other words different "contexts" in the data. A Product node computes the joint distribution of its children with a product, explicitly carrying an assumption of independence among the factors contributing to the product. Children of product node can be determined by using a dependencce/correlttioo between each pair of features, subjected to a threshold. In other words, to make tree/sub-tree with a product node as the root, a graph (totally different from the main SPN tree) built where two feature nodes are connected if and only if they are considered dependent according criterion. Each connected component of this graph is a sub-tree in the sum-product architecture where its root is linked to the root product node.
measure such as mutual information as a link/edge
Picturing the data in matrix form where each row is an instance (or sample) and each column represents feature (or variable), one can think of the action of sum node as splitting the rows (samples) and that of a product node as splitting the columms (features). This means that recursive splittings of data rows (e.g. by common clustering method) producing sum node, and based on
3

---

Published as a conference paper at ICLR 2023
.288
9
.4
.1
2
.4
71
1 111 1
1s.
Lsp
S.
50
100
150
200
250
300
Time (ms)
30
3
0
Modelling Spiking activity with Sum Product Network(SPN) Left: Neural population activity is often represented by a binary matrix, obtained from binning each neuron's spike train (here 10 ms). Here, the firing rates of neurons to 3 are 10, 20, and 40 Hz. Due the high correlation of these neurons, bins with all 1, or all O are more frequent compared to the activity of 3 independent neurons with the same firing rates. Right: An spn tree (here nS P P fit to the population in the left plot. Since it assumes independence, its weights reflect the mean probability of a spike in each time bin after training (shown blue). Given data instance (here 001), each node represents the probability of its children, meaning that the root represents the probability (likelihood) of the instance.
Figure 1:
variables (e.g. via performing independence tests) resulting in product node would give us a suitable SPN structure for the given data (Gens & Pedro 2013 Vergari et al. 2015).
3.1 NEURAL POPULATION ANALYSIS WITH nSPNS
We construct family of SPNs that we call Neural SPNs (nSPN) model the joint probability distribution of spiking in populations of neurons. These models are represented by directed graphs where leaves of an nSPN representing spiking activity of N neurons correspond to Is, and Is, (1 S < N) indicating whether neuron i fired or not in given time bin (see section 2)
Shallow architecture of nSPNs: The simplest structure of an nSPN consists of product node as a root, linked to N sum nodes. Each sum node represents the spike probability of a neuron i, linked to II, and H,; leaf nodes. This model, shown as P basically a Naive Bayes model, assuming complete independence among neurons. The right plot in figure [ shows an example nSPN' trained on the joint activity of three neurons in the left plot. As mentioned before, each node of an SPN represents the probability of the leaves of the (sub-)tree of which it is the root, As result, (normalized) link weights of each sum node are aligned with the mean spike probability of the related neuron (note the assumption of independence). Furthermore, given assignment of probabilities to the Ieaf nodes and a configuration of spiking activity, the root represents the probability of that configuration' 's occurrence according to the model (shown by the red number near the root node).
Within the SPN framework, a simple extension of the Naive Bayes model involves the introduction of multiple contexts, with population spiking being described by a Naive Bayes assumption conditioned on the context. If there are two such contexts with associated probability, the model consists of a sum node the root linked two product nodes, each of which a root of an nSPP11 (sub)-tree. Similarly, model can include an arbitrary number of contexts by increasing the number of product nodes under the root, c.g. nSPN No for b product nodes/contexts.
In principle, one can construct a shallow graph sufficient to model any data set by adding enough product nodes. One problem with this approach is that there exist simple networks with deeper structure for which a 3-layer model like we have described will require an exponential number of nodes (Delalleau & Bengio 201T). which in addition to tractability also raises the issue of data limitations. Moreover, all existing parameter learning algorithms for SPNs find local optima and shallow networks get stuck in local optima more frequently. Most importantly, when the data is limited deeper architectures of our framework avoid over-fitting as their structure implements regularization.
Deep architecture of nSPNs: As computing all correlations is intractable, we seek an approach for constructing graphs that trades some fidelity for efficiency. SPN structure learning algorithms naturally achieve that by separating low correlation variables (according to an independence test and threshold on its value) at a product node. Importantly, in deep layers of an SPN, variables considered independent in given context. We utilized learnSPN (Vergari et al., 20155 Molina et al., 2019), shown by SPN1, due to its more conservative approach for splitting. This method has two
4

---

Published as a conference paper at ICLR 2023
hyper-parameter an independence threshold and a minimum number of data instances to continue splittin. These hyper parameters can be tuned either by a cross validation procedure or based on the data size. Notably, due our small data size, applied Laplace-like regularization to the method to prevent pair-weights of [O. IJ (absolute zero makes the log-likelihood -inf, so we regularized it to [10-9, -10-9.)
Measure of Neural Correlation: The simplest architecture, nSSNN assumes independence be- tween all neurons. On the other hand, the more complicated structures in our framework only assume independence within each context where the number of contexts are always more than 1. This means that in stationary periods where the firing patterns of neurons do not change significantly, nSPN' fits to the data as well as more complex structures if and only if all neurons are independent of each other. Any improvement in goodness of fit for more complicated structures is due to the existence of neural correlations. As result, we can use the difference between the average log-likelihood (equation D of the best fitting PSS and nSPNN as a measure for neural correlations. For example, our best model is nSPN our measure for population neural correlation is AllsPN UnSPN- UnSPN..
4 RESULTS
We start with simulation of population activity in which we can control the presence of fluctuations. We use a network of homogeneous exponential integrate-anc and- (EIF) neurons with common fluctuating input where each neuron's membrane voltage, V:(1 S i S N)) evolves follows:
I' V. - Are
Ve AT
+ L(t). ((t) = 0 Tm [I1 (t,(t) + (t).
(2)
Notably, it has been shown that pairwise maximum entropy (PME) models are incapable of estimating this network very well especially when the firing rate of neurons is low, neural correlations are high, or the population size is large Amari et al. (2003); Macke et al. (201 Leen & Bhea-Browwn (2013). Moreover, experimental studies show the existence of higher order correlations in the brain Binder & Powers 2001): Khue-Trong & Rieke 2008). Therefore, the network above has been used as tool to assess the performance of methods for capturing beyond pairwise correlations Macke et al. 1b); I Leen & Biea-Broonjj (2015). Similarly, we generate multiple data sets using varying parameters to get different values for the mean activity μ and (pairwise) correlation p and compare the goodness of fit between NSPNNN/2 and PME models for population size of N (all parameter values in Supplementary Material). Note that n.S1 NNSP is chosen to have the same number number of parameters as the PME, N (N + 1)/2 (nSPNN/22 has N²/2 1 free parameters to be more precise).
Pairwise Maximum Entropy models have minimal assumptions about the spiking pattern of the population beyond the firing rate of each neuron and the relationship between each two neurons. In full generality, PME model is complex with many parameters, and is computationally expensive in general. In fact, as mentioned before, an exact fit of PME is NP-hard. Therefore, we instead fit to the probability that the a population of N neurons will generate spikes overall in a single time bin, i.e.:
PPMe (k)
exp (ack + Bk3)
k
where a and 3 are free parameters, obtained from fitting the model to population activity. Importantly, instead of fitting the PME the whole joint distribution obtained a and ,3 by directly optimizing the above equation. This measure reflects the maximum possible power of any PME model without any need to actually obtain all variables (it essentially provides bound the performance of the full model). Note that, the distribution of average population activity fully reflects the goodness of fit in fully homogeneous populations Roudi al. (2009a). However, there no guarantee that existing approximation methods for PMEs achieve such fit.
As opposed the PME, we fitted an nSPN to the whole joint distribution (200 iterations of Expectation Maximization (EM) implemented in SPFlow library Mooina et al. (2019)) giving the PME a huge advantage. Specifically, the difference between the model's prediction and the EIF (total) population spike distribution is quantified with the Conson-Shannon (JS) (JS)-divergence of (PPIF((K)) (k)) where M is the averaged distribution, i.e. (k) = 1/ Pmmodet(b (k) + 2 (Frr(k) and (.Ill.) is the Heilback-Leiblee divereivee Since JS-div scales with log( (N), we report the normalized value, i.e. JS-div (diiv//o////// N). We set À =30 and .59 to make
[P (k) (k))
5

---

Published as a conference paper at ICLR 2023
p equal to .1 and .25 respectively, keeping other parameters the same as above. In both cases, the mean activity rate I is .1. The number of neurons in the network is N [8, 16, 32, 64, 128), and each data point is obtained from training on 100000 and testing on another 100000 samples. As shown in the left panel of figure 2 nSPN models the EIF network significantly better than PME (the JS-divergence is orders of magnitude lower) especially when the number of neurons and correlation (A) are high (dotted lines À = .59, solid lines À = .3).
the
We also compared our method to other approaches for capturing joint probability distributions of spiking activity. We trained a Bernoulli Restricted Boltzmann Machine (RBM) with N visible and N/2 hidden units (N2 22 free parameters). The JS-div of the RBM was an order of magnitude higher than the SPN (figure 2 left plot). Moreover, the SPN scales better with the increase of N (Note that the y-axis is in log-scale). Importantly, the RBM is one of the best known latent-variable model in capturing population correlations (Köster et al. 2014).
One of the goals of these benchmarks is assess model performance as a function of population size (N). Hence in some simulations we include single, global input to all neurons (governed by parameter À in the equation below). In addition, we introduce correlated subpopulations by adding a common input to specific subpopulation both pairwise (via X²) and in quadruples (via X²).
(tt) = 7 + v Tm [V1 XC,(t) + VX (A2 + Aa) + (t2,,,(t) + VAaGaa(t).. (4)
For simplicity, we assumed these common inputs are among adjacent nodes: A1 < k S N/2 (ox-1,2(t) = 2 (t), and L1 S. k S. N/4: (,2) = ... (t). We repeated the previous set of fits (A .59, .3), this time with X2 A = .05. As demonstrated in tigure 2 middle left plot, SPN outperformed the other models by orders of magnitude again.
We also performed simulations with heterogeneous inputs, resulting in heterogeneous population activity. Neurons are divided into four groups with different voltage reset values and different values of \ (values can be found the supplementary material). This network was simluated with both d2 = X4= 0 (Figure 2 middle-right: dotted lines) and )a a .05 (Figure middle-right: solid lines). Our SPN significantly outperformed the other models. Note that due to the heterogencous network activity, the PME direct fitting results are not valid lower-bound for JS-div of a pairwise model anymore.
We also tested recurrent networks with heterogeneous connectivity. We analyzed a known connected "balanced" network of excitatory and inhibitory spiking neurons Brunel (2000). The dynamics of the membrane voltage of each neuron in this network is:
Fm V,' = -V, + (t), RI(()) = Fm Σ Jig
Σ k
8(t 1) - D),
(5)
where (t) are the synaptic currents caused by spikes arriving at synapses of neuron i (more details in the supplementary material). We specifically fit the models to two stable setups (figure 8, e and d) of the original paper with "stationary" and "slow oscillation" global activity [Brunel (2000). As shown in the right plot of figure 2 JS-div of the SPN was orders of magnitude smaller than other methods. Importantly, here the size of the networks were always 1000, and each fitting was on the first N excitatory neurons. Moreover, similar to our previous network, JS-div div from fitting PME directly to the population activity is no longer valid lower-bound for pairwise models.
EXPERIMENTAL NEURAL RECORDINGS
To assess our method on experimental neural recordings we modelled the neural data from the Neuropixels Visual Coding data set of the Allen Brain Observatory (utps://observatory.brrn map.org) (Siegle et al.) 202I). In this data set, different stimuli (Gabors, flashes, drifting gratings, etc) were shown to 26 mice while the neural activity of their visual cortical and sub-cortical regions were recorded simultaneously with multiple Neuropixels probes. We modelled the neural population structure during the viewing of drifting gratings in 4 directions (0', 45", 90" 135%)) with the contrast level of .9. Each of these 4 stimuli was shown to 26 mice 75 times. We analyzed the neural activity (spikes) of six regions in the visual cortex: VISp. VISI. VISal, VISpm, VISrI. VISam. Following other studies amalyzing neural correlations Köster et al. 2014);; Tkačik et al. (2014), the time bin was set to At = 20ms, balancing for minimizing spike loss due to binning, and accounting for activity delays between neurons.
6

---

Published as a conference paper at ICLR 2023
10-11
me
10
10
[Sadiofog
[S.dwwoog
IS S.dwtog N
ciwilog N
10
10-1.
20"1
10
10
10.4
10
10
32 Population
64 128
128
Population size
Population
Sample population
Figure 2: Simulation of neural population activity confirms the practicality of our SPN-based framework to measure correlations among neurons. Left: Normalized JS-divergence versus population size for two conditions with similar mean activity rate (μ = .10) but different correlations (dotted lines: X = .59, solid lines: À = .3)). nSPN outperformed both RBM, as well as the best possible PME. Middle left: Similar to the left plot, but with additional common inputs to pairs and quadruples (A2 \ = 0.05; dotted lines: À -59, solid lines: À = .30). Middle right: Same comparison on heterogeneous network with or without local inputs (dotted lines: X = X = O, solid lines: X2 A = 0.05). Right: Same analysis on N excitatory neurons of a well-known sparsely connected network with total population size of 1000 in two regimes, stationary global activity with irregularly firing neurons (dotted lines), and slow oscillation of global activity with very rare irregular firing (solid lines).
Each stimulus duration was 2 seconds. We analyze the response from 400ms after stimulus onset to 800ms after stimulus onset. The delay is to make sure the corresponding signal has reached the neurons in all areas. Moreover, individual neurons' change upon exposure to the same sensory stimuli (via adaptation), hence the short period. For each grating direction, we concatenated data from 5 presentations of 400ms to get 100 data points. The order of presentation of grating directions is random. In order minimize the effects of state changes in the animal, we concatenate trials from subsequent presentations of the same grating direction. This data processing results in 15 blocks of population activity per each tuple (mouse, direction, area), each of which we modelled with a n.SPN. Notably, we used the same set of hyper parameters for all fits. Hyper parameter tuning of n. for each fit via cross validation improved the overall fit only slightly, demonstrating the robustness of our method.
We also fit our simplest structure nSPN' (Naive Bayes) in order to have a correlation -freattiooo-eeeeeee baseline and compute the difference, AllsPN. We fit both models on modified version of data in which we shuffle the spikes of each neuron in the 400ms of single stimulus showing for the primary visual cortical population (as the most sensitive area events). We observed large difference between AllsPN i in the original and shuffled data. This difference confirmed our assumption about ASPN in the original data is mostly the product of neural correlations (Figure left).
Since the population is not homogeneous, this time we used Minimum Flow Probability (MFP) (Sohl-Dickstein et al. 2011) that approximates pairwise maximum entropy variables through direct fit to the data (implementation of (Lee & Daniels 2019)). To the best of our knowledge this is the most accurate methods of pairwise approximation. We also fitted an RBM to the data. The number of hidden units in the RBM was determined through 10-fold cross- cross-validation within the 100 samples.
The (total) population activity is not an ideal measure for performing model comparison. However, given the number of samples, i.e. 100, more desirable measures are prohibitive. Compared to other models, nSPN produced significantly lower normalized JS-div for all of the 6 regions (Figure 3 middle plot). In fact, J-div of n. was lower than other methods for nearly every single (animal, area, direction). The right plot figure shows this comparison v.s. RBM.
We also tested our model on four short (340 ms) movic clips, cut from the "natural movie one" stimuli of the same data set, which is a one-shot 30 second video repeated 60 times. The frame rate is 30 Hz, making each of our clip consists of 10 frames, starting from frame number 240, 390, 490 and 890. Similar to the previous stimulus, we grouped 5 consequent clips together, which resulted in 12 blocks of population activity. The time bin was set to 17 ms, meaning that each block included 100 data points. We repeated the analysis described above for this stimulus type and got consistent results (Figure 4). Notably, AllsPN was higher in general in this stimulus, probably because of longer time difference (at least 30 seconds) among the trials of each group. Because of this, very high computational cost, and poor results on simpler stimulus type, did not include M PF in
7

---

Published as a conference paper at ICLR 2023
#(animal, direction)
40-
Shuffi Original
0.14
(PMC
0.035
30-
0.12
RBM IS (-diiilogiN)
B-divtog/W)
0.10
0.025
0.08
0.015
Visi
0.06
VISel
10
0.04
Visem
0.005
Vi5pm
0
0.02
Visri
0.25
1.25 Allss
2.25
0.00
0.00
0.01 0.02 0.03 NSPN Js div/log(N)
V1pp
VISI
Figure Modeling high order correlations in neural recordings during drifting grating stimulus. Tested on large-scale Neuropixel recordings from mouse visual cortex upon exposure to during drifting grating stimulus, nSPNN explains neural population structure and correlations in different regions of mouse brain significantly better than pairwise entropy model estimation (MPF) and RBM. Left: Distributions of AllsPN by experiment for the unshuffled (orange) vs shuffled (blue) spike distributions. Middle: Box plots of the distribution of normalized JS-divergence over experimental sessions by visual cortical area. Right: Comparison of JS JS-diveree of SPN with RBM for every single (animal, area, direction) (each dot).
# (animal, clip)
20
Original
Shuffled
0.025
15PN
0.025
RBM JS (-ddivlog)N)
15
0.320
dvologgt
0.015
10-
3.015-
visi
5
0..310
Visemn
0.005
v15pm
0
0.0G5
Visrl
0.25
1.25 Allsw
2.25
0.005
0.015 NSPN div/log(N)
0.025
ViSp ViSI
ViSai Vi5ammiSm viS
Figure 4: Modeling high order correlations in neural recordings during natural movies. Same analysis discussed in figure B on four different natural movie clips. Left: Distributions of AllsNN by experiment for the unshuffled (orange) vs shuffled (blue) spike distributions. Middle: Box plots of the distribution of normalized JS-div over experimental sessions by visual cortical area. Right: Comparison of JS-div of SPN with RBM for every single (animal, area, movie clip) (each dot).
our analysis for this stimulus. Moreover, as observed in the right plot of figure 4 superiority of our model is bolder compared to the simpler stimulus of drifting gratings (right plot of figure
FURTHER ANALYSIS ON THE ROLE OF NEURAL CORRELATIONS
Without knowledge of the ground truth, data interpretation is much harder especially when the data limited and high dimensional. To further investigate the role of (spatial) neural correlations in population activity we extended our analysis to dynamic latent models that look at each trial as a whole (Macke et al. 2011a, Pfau et al. 2013 Gao et al., 2015). Therefore, these models take long temporal correlations into account. While these models are usually applied motor tasks with longer trial duration, temporal correlations could affect the population structure in our tasks well, especially during the natural movie stimulus. We fit Poisson Linear Dynamical System (PLDS) with nuclear norm penalized rate estimation (Pfau et al. 2013 to the population activity of primary visual cortex (VISp) as example extensions our analysis during both natural movie clips and drifting gratings similar to our previous fits. The number of latent factors were determined by cerre-oo-ouutcross validation over trials (which were 5). As shown in the left plot of figure [5] JS- divergence of nSPN were lower for every single stimulus and mouse except one (animal, stimulus) pair, especially the more complex stimuli, i.e. movie clips. Notably, latent dynamical systems have demonstrated the ability to capture pairwise correlations, in addition to temporal dynamics of the
code from: https: /bbukkkeeet.
tbucket. wwwpccettorg//mccceec wwt.//oggg/.eee..oooeee.o wttpp//kwe..pppppppppppp..eeeeee
8

---

Published as a conference paper at ICLR 2023
0.04
Dor cost of MM wail papi aieemer
PLDS div/log(NN)
0.03
0.2
papil
0.0-
0.02
0.0-
-3.2
0.01
Driting gratings
-0.4
-0.4
Natural movies
-0.0
-0.5
0.01 0.02 0.03 0.04 nSPN (S-div/log(N)
-0.9
Višp ViS
ViSal ViSam M5pm ViSit
Vi5p
ViSi
ViSei
ViSam Vi5pm viSri
Figure 5: Further analysis on neural population correlation. (left) Poisson Linear Dynamical systems which model temporal evolution and correlations of neural activity explain VISp population structure worse than n.P in almost all mice and stimulus conditions. (middle) The Pearson correlation between Allepn and pupil diameter over experiments significantly negative in visual cortical areas. (right) Same approach applied on MPF, Allmps can not fully capture this phenomenon.
system (c.g. temporal cross correlations in (Macke et al. 201 Therefore, this result suggests that higher order short-term/spatial correlations (20 ms) plays more crucial role here. Using mixtures, nSPN can capture (discretized) temporal dynamics but (probably) not as good as methods with explicit components of modelling dynamics.
Our second attempt on checking the validity of our conclusions involved previous neuroscience literature. Many experiments have suggested that task engagement the form of attention and arousal has the effect of reducing neural correlations, decreasing the level of synchrony (Gandal et al. 2012
Uhlhaas et al., 2009), and even measurable behavioral changes (Cohen & Maunsell| 2009). These experiments are mostly based on very noisy data such as the Local Field Potential (LFP) and rough
estimates signal frequency (Gandal et al. 2012 Pfeffer et al. 2022, Vinck et al. 2013). Having a quantitatively reliable measure based spiking data, i.e. AllsPN. during drifting grating viewing. Specifically, in each animal we looked at the (ratio to minimum) pupil diameter (a common measure of arousal in mice) changes each of the chunks described above for all directions. To alleviate the effect of trail-to-trial variability (e.g. caused by behavioral state change) on neural correlations we removed (mouse, direction) pairs with AllsPN more than 25 in the primary visual area for the shuffled data. The mean distribution of Pearson correlation coefficient between pupil diameter and AllsIN across all animals was significantly below zero for all regions as shown in Fig. (left). To the best of our knowledge, even with LFP data, there is no work demonstrating the effect of arousal on higher visual areas during an experiment (See (Vinck) al. 2015) for VISp).
examined the effect of arousal
We repeated the same process for MPF by using the difference between log-likelihood of the full (first and second order) model with the first-order only version (second order variables set to zero). The result was significantly different (Fig. 5 right). Importantly and as mentioned, previous literature results (especially VISp) strongly in favour of the SPN results. Importantly, the stimulus in the previous VISp study was also drifting gratings (Vinck et al.) 2015). Moreover, this result further demonstrates the inability of the PME in modelling the full joint population activity (as opposed to the total population activity).
DISCUSSION
Correlations are an important channel of information in neural activity, with strong consequences for coding properties. Using SPNs, we have constructed a computationally efficient approach to modeling structure and correlations in populations of spiking neurons. Tested on simulated and experimental data, our approach outperformed both energy-based and latent variable methods. Due to the nature of our scientific question in this paper, i.e. population correlations, we focused on short time spans in which the external variables and animal state remain constant. Our framework, however, has the capacity to be applied more broadly in computational neuroscience, for example to capture population dynamics in different time scales or as a de-mixing tool for coded features in individual or population of neurons.

---

Published as a conference paper at ICLR 2023
Reproducibility:
The data set used in this paper. Allen Brain Observatory, is publicly available online. All parameters of the simulated data has been specified the result section and/or the supplementary material. Similarly, all parameters related to the methods results section and/or the supplementary material. In addition, all the code is based on open- source repositories, cited in the results section. Finally, the code for data is available in https com //gtthubb.cmmKooohaa5 koosha //NeuraallP.
mentioned in the
Acknowledgment: We wish thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement and support.
REFERENCES
I. F. Abbott and Peter Dayan. The Effect of Correlated Variability on the Accuracy of a Population Code. Neural Computation, 1 11(1):91-111, January 1999. ISSN 0899-7667. doi: 10.1162/ 08997669930011822) URL http:////.....ooooo.oooo//////10
Shun-ichi Amari, Hiroyuki Nakahara. Si Wu, and Yutaka Sakai. Synchronous Firing and Higher- Order Interactions in Neuron Pool. Neural Computation, 15(1):127-142, January 2003. ISSN 0899-7667. doi: 10.116/08996600321443739 URL https://doi.org/D000
Erik Aurell and Magnus Ekeberg. Inverse Ising Inference Using All the Data. Physical Review Letters, 10809),00020, March 2012. doi: 10. 10.1100s/yyseeel..et. 10t..ett....0000000000 099001 URL (RLhttpss
/ 2ttp//0..1033/..yeeeeetttt 108. 1090000000 090201] Publisher: American
//link. aps. /ti/10.111111//
Physical Society.
Bruno B. Averbeck, Peter E. Latham, and Alexandre Pouget. Neural correlations, population coding and computation. Nature Reviews Neuroscience, 365) May 2006. ISSN 1471-0048. doi:
10.1038/nmm188 1888. URLhttps:: :/tp::///www.. wwt.w.natur.. www.nnntteee..m Publisher: Nature Publishing Group.
wtt.ticccss//nn888e11 Number:
Rava Azeredo da Silveira and Fred Rieke. The Geometry of Information Coding in Correlated Neural Populations. Annual Review of Neuroscience, 44(1):403 424, 2021.
doi: 10.1146/amuurev-neurg neuro- 120320-082744.
URL
10w.aev---eurr--22000------- 082744.
_eprint: http::/loo.og//l...///aaameeeeeo
Alberto Beretta, Claudia Battistin, Clélia De Mulatier, lacopo Mastromatteo, and Matteo Marsili. The Stochastic Complexity of Spin Models: Are Pairwise Models Really Simple? Entropy, 20( 10); 739, October 2018 ISSN 1099-4300, doi: 10.3390/c20100739, URL https:: /wwww mdpi com/ 1099- 00009944440002/10/7533 Number: 10 Publisher: Multidisciplinary Digital Publishing Institute.
M. D. Binder and R. Powers. Relationship between simulated common synaptic input and discharge synchrony in cat spinal motoneurons. Journal of Newrophysiology; 86(5):2266-22755 November 2001. ISSN 0022-3077. doi: 10. 152/jn.222000 2001 88..5.2266.
Craig Boutilier, Nir Friedman, Moises Goldszmidt, and Daphne Koller. Context-specifie independence in Bayesian networks. In Proceedings of the Tweifth international conference on Uncertainty in artificial intelligence, UAI'96, pp. 123, San Francisco, CA, USA, August 1996. Morgan Kaufmann Publishers Inc. ISBN 978- 978-1-55860-41244
Nicolas Brunel. Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons. Journal of Computational Neuroscience, 8(3): May 2000. ISSN 1573-6873, doi: 10. 00.10233/A:10089953300222 URL 113
Amelia J. Christensen and Jonathan W. Pillow. Reduced neural activity but improved coding in rodent higher-order visual cortex during locomotion. Nature Communications, 13(1):1676. March 2022. ISSN 2041-1722 1723. doi: 10. 147--222-20200
btRpsttps: : www. nature]
Number: Publisher: Nature Publishing Group
10

---

Published as a conference paper at ICLR 2023
Marlene R. Cohen and John H. R. Maunsell. Attention improves performance primarily by reducing interneuronal correlations. Nature Neuroscience, 12(12):1594-160 December 2009. ISSN 1546- 1726. doi: 10.1038/nn.2433
http://://wwwwwttteeeeeeemmm Number: 12 Publisher: Nature Publishing Group.
2439
Gregory F. Cooper. The computational complexity of probabilistic inference using bayesian be- lief networks. Artificial Intelligence, 42(2):393-405 March 1990. ISSN 0004-3702. doi:
10.1016/000443772000000300000000 ott.eete////..ieeeeeeeee........ ott.://cce...e.e...oooooooooeeee scinnce
artticleesL1/00447722200000
Adnan Darwiche. A differential approach to inference in Bayesian networks. Journal of the ACM,
5063):20-335, May 2003. ISSN 0004-5411. doi: 70.11457765568.765577. URL https:
11411......1.111 1ee.11115775555 765570
Clélia de Mulatier, Paolo P. Mazza, and Matteo Marsili. Statistical Inference of Minimally Complex
Models, September 2021. URL htl.httpp//aaxii www.:/orgabb///0//// 00520 arXiv:2008.00520 [physics, q-bio, stat].
Olivier Delalleau and Yoshua Bengio. Shallow vs. Deep Sum-Product Networks.
In Advances in Neural Information Processing Systems, volume 24. Curran Asso- ciates, Inc., 2011. URL
wtt.aurpsss.coppaaee/11111/1///. 8 aatbbo/2644eeebb122222aaaaa33333 Absst html
Brent Doiron, Ashok Litwin-Kuma, Robert Rosenbaum, Gabriel
Ocker, and Krešimir Josić. The mechanics of state-dependent neural correlations. Nature Neuroscience, 19(3):383-393, March 2016. ISSN 1546 1726. doi: 10.1038/mn.4243
htt.hhtps:///wi http://wwww.att...eeeeeae com/ Number: 3 Publisher: Nature Publishing Group.
articles/ nn. 4242
Benjamin Dunn, Maria Mørreaunet, and Yasser Roudi. Correlations and functional connections in population of grid cells. PLoS computational biology, 11())ee004052, February 2015. ISSN 1553-7358. doi: 10. 10.1371/jurral.pccb 0ctpp/mm0a..ccc....002....... 1004052.
Michael J. Gandal, J. Christopher Edgar, Kerstin Klook, and Steven J. Siegel. Gamma synchrony: towards translational biomarker for the treatment resistant symptoms of schizophrenia. Neurophar- macology, 62(3):1504-1515 March 2012. ISSN 0028-3908, doi: 10. n0.10166)..uurophamm..mm 201 2..1.02.
007. URL https: www.//wwwww..// ncbi
m1m. nih gww.gov//oocoo http:///mm/aarrrraaee//////222//
Elad Ganmor, Ronen Segev, and Elad Schneidman. Sparse low-order interaction network underlies highly correlated and learnable neural population code. Proceedings of the National Academy of Sciences, 108((3))9679-98))
1019641108.. URL https // //www. pnas org/ doi/10. 10w3//0a0//ss 00106411110 National Academy of Sciences.
June 2011. doi: 10.
Publisher: Proceedings of the
Yuanjun Gao, Lars Buesing, Krishna V. Shenoy, and John P. Cunningham. High-dimensional neural spike train analysis with generalized count linear dynamical systems. Proceedings of the 28th International Conference Neural Information Processing Systems Volume 2, NIPS' 15, pp. 2044-2052, Cambridge, MA, USA, December 2015. MIT Press.
Robert Gens and Domingos Pedro. Learning the Structure of Sum-Product Networks. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pp. 873-880, Atlanta, Georgia, USA, June 2013. PMLR. URL htttpss://proceeddrnrg ml wttp.ggg......../ press/ v28/ 0288/ges13..m html
Issue: 3.
Philipp Khuc-Trong and Fred Rieke. Origin of correlated activity between parasol retinal ganglion cells. Nature neuroscience, (011)):14431333 November 2008. ISSN 1097-6256. doi: 10. 1038/ nn.2199. URL URR/:ttss://wwe ncbi nlm. nih. gwt. httpa////o////e/atataeeeeeee//ee http://rriccclee//M27755733333
Adam Kohn, Ruben Coen-Cagli, Ingmar Kanitscheider, and Alexandre Pouget. Correlations and Neuronal Population Information. Anmual review of neuroscience, 39:237-256 July 2016. ISSN 0147-006X. doi: 10. 1 14/4///mreee 00-1137188111 URL hhtpps: www. mebi..nnnn. nlm. nih. mmt..po//gmm//rrttccll/////sss//

---

Published as a conference paper at ICLR 2023
Urs Köster, Jascha Sohl-Dickstein, Charles M. Gray, and Bruno A. Olshausen. Mod- eling Higher-Order Correlations within Cortical Microcolumns. PLOS Computational
Biology, 10(7):e July 2014. ISSN 1553-7358. 1003684 URL ht fourna/ s.poos wtrplos.oggooooooo.o..oooooo .rgg www.ombbol/aattccleee..o.. 1371/journa pabi 1003684 Publisher: Public Library of Science.
doi: 10. 10.137//jourma.phhb
Edward D. Lee and Bryan C. Daniels. Convenient Interface to Inverse Ising (ConIll)): A Python
Package for Solving Ising-Type Maximum Entropy Models. Journal of Open Research Software, 7(1):3 March 2019. ISSN 2049-9647.
doi: 10.5334/jors.117. URL https
//openresearchsoftwweae wtt.:eeeeeeeeoceoeooooeeoeeeeeee
meta metajnl. wtt.///tttttteeeeeeeeeeo
jors
David A. Leen and Eric Shea-Brown. A Simple Mechanism for Beyond-Pairwiss, Correlations in Integrate-and-Fire Neurons. Journal of Mathematical Neuroscience, 5(1):30, December 2015. ISSN 2190-8567. doi:
-400-005--0----..
Jakob H Macke, Lars Buesing, John P Cunningham, Byron M Yu, Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking in neural popula- tions. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011a. URL
httppppp.ccppppe////////aaaaaa htmml
Jakob H. Macke, Manfred Opper, and Matthias Bethge. Common input explains higher-order correlations and entropy in simple model of neural population activity. Physical Review Letters, 106(20):208102, May 2011b. ISSN 1079-7114. doi: 10.1 hwto:PPhysReLeee.16..2000000
Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Pranav Subramani, Nicola Di Mauro, Pascal Poupart, and Kristian Kersting, SPFlow: An Easy and Extensible Library for Deep Probabilistic Learning using uun-Product Networks, 2019. _eprint: arXiv: 1901.01900.0704.
Rubén Moreno-Bote, Jeffrey Beck, Ingmar Kanitscheider, Xaq Pitkow, Peter Latham, and Alexandre Pouget. information-limiting correlations. Nature Neuroscience, 17(10);1410-444 1417 October 2014. ISSN 1546-1726.
doi: 10. 0/38//n.3807. URL [RRLhhtts:://ae
na wwwuurr...eee.t.eee wwt.//ommmaa...... aatmm/rttceees////
nn. nn. 3807
Number: 10 Publisher: Nature Publishing Group.
David Pfau, Eftychios Pnevmatikakis, and Liam Paninski. Robust learning of low dimenssnsi
dynamics from large neural ensembles. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL ht
http:://///epeeeeeeeeeee. neur htt.thtts::///rreeeeeiiii cc/
paper /20rr//1133// 2013//01hh.. 47a65822 9ew.9eb22..... 9 w0w.:/0022888442----882
.html
Thomas Pfeffer, Christian Keitel, Daniel S Kluger, Anne Keitel, Alena Russmann, Gregor Thut, Tobias Donner, and Joachim Gross. Coupling of pupil- and neuronal population dynamics reveals diverse influences of arousal on cortical processing. eLife, 11 11:790900, February 2022. ISSN 2050-084X. doi: 10.7554/el.ife.71888
URL.https:://de http///oo.or///..///// wtt.oo/g............... 7554/eLi 71890
Publisher: eLife Sciences Publications, Ltd.
Jonathan W. Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, and Eero P. Simoncelli. Spatio-temporal correlations and visual signalling in complete neuronal population. Nature, 454(7207)9995-99) August 2008. ISSN 1476-4687. doi: 10.1038/nature07144 140. URL htttps:/:/w/w nature com/ /tta///rrccce///aarrr111000
Number: 7207 Publisher: Nature Publishing Group.
Hoifung Poon and Pedro Domingos. Sum-product networks: new deep architecture. In 2011 IEEE International Conference Computer Vision Workshops (ICCV Workshops), pp. 689-690, November 2011. doi: 10. 11110///////21111 66130100
Amirmohammad Rooshenas and Daniel Lowd. Learning Sum-Produch Networks with Direct and Indirect Variable Interactions. In Proceedings of the 31st International Conference on Machine Learning. pp. 710-718. PMLR, January 2014. URL
mwr.smmmm.rr..mmmm .mr. p
v32 rooshenas141 html
ISSN: 1938-7228.
Yasser Roudi, Erik Aurell, and John Hertz. Statistical physics of pairwise probability models. Frontiers in Computational Neuroscience, 3, 2009a. ISSN 1662-5188. doi: 10.3389nncuro.00 022.2009. URL httpttttp/////tttt frontiersinni www.aarerrri...ooo/aarraaaaieeaa wttp://uo......uuoo.oooo 022 2009
12

---

Published as a conference paper at ICLR 2023
Yasser Roudi, Sheila Nirenberg, and Peter E. Latham. Pairwise Maximum Entropy Models for Study- ing Large Biological Systems: When They Can Work and When They Can't tt PLoS Computational Biology, 1000380. May 2009b. ISSN 1553-734X. doi: 10. 10.137 fii 1000300 URL
https: / www. ncbi n m. n miniiihhh gov http:///w/aatiiesseesspccca////t PPC227469767
Yasser Roudi, Joanna Tyrcha, and John Hertz. Ising model for neural data: Model quality and approximate methods for extracting functional connectivity. Physical Review E, 79(5):051915 May 2009c. doi: 10. (003//hysRee..7.051111.
htt.ttts:: ink. www.kkappp.......m...m....mmm wwwps.orr/oo///// Publisher: American Physical Society.
1103/PhysRevE. 79. 0110051111 915
Caroline A. Runyan, Eugenio Piasini, Stefano Panzeri, and Christopher D. Harvey, Distinct timescales of population coding across contex. Nature, 548(7665):92 96.9999966696 August 2017. ISSN 1476-4687.
doi: 10.1038/nature23022
htLthtts:://wwt nature. com Number: 7665 Publisher: Nature Publishing Group.
Raquel Sanchez-Cauce, Iago Paris, and Francisco Javier Diez. Sumpproduct networks: A survey, IEEE transactions on pattern analysis and machine intelligence, PP, February 2021. ISSN 1939-3539.
doi: 00.1109/TPAM1.2022 2021..0611198
Elad Schneidman, Michael J. Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in neural population. Nature, 1407:)4044400075510000-0011 April 2006. ISSN 1476-4687. doi: 10. 10tt//at URL/tttps: www nature. com/
articles http:/nuuurre0000 Number: 7087 Publisher: Nature Publishing Group.
Yujia Shen, Arthur Choi, and Adnan Darwiche. Tractable Operations for Arithmetic Circuits of Probabilistic Models. In Advances in Neural Information Processing Systems, volume 29. Cur- ran Associates, Inc., 2016. URL https: /ttttps://ppocceeiiiin neurips. wwwpeeurrsssccppppppeeeeee cctpappe/2216/
html
Yujia Shen, Arthur Choi, and Adnan Darwiche. A New Perspective on Learning Context-Specific Independence. In Proceedings of the 10th International Conference on Probabilistic Graphical Models, pp. 425 436. PMLR, February 2020. URL https::pp/.........t....t http::::s//proceednge. mlr htw.pnngns.i.....s.s/..pp press/ ISSN: 2640-3498.
v1 38 shen 0a h.tmm
Joshua H. Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K. Ramirez, Hannah Choi, Jennifer A. Luviano. Peter A. Groblewski, Ruweida Ahmed, Anton Arkhipov, Amy Bernard, Yazan N. Billeh, Dillan Brown, Michael A. Buice, Nicolas Cain, Shiella Caldejon, Linzy Casal, Andrew Cho, Maggie Chvilicek, Timothy C. Cox, Kael Dai, Daniel J. Denman, Saskia E. J. de Vries, Roald Dietzman, Luke Esposito, Colin Farrell, David Feng, John Galbraith, Marina Garrett, Emily C. Gelfand, Nicole Hancock, Julie A. Harris, Robert Howard, Brian Hu, Ross Hytnen, Ramakrishnan Iyer, Erika Jessett, Katelyn Johnson, India Kato, Justin Kiggins, Sophie Lambert, Jerome Lecoq, Peter Ledochowitsch. Jung Hoon Lee, Arielle Leon, Yang Li, Elizabeth Liang, Fuhui Long. Kyla Mace, Jose Melchior, Daniel Millman, Tyler Mollenkopf, Chelsea Nayan, Lydia Ng. Kiet Ngo, Thuyahn Nguyen, Philip Nicovich, Kat North, Gabriel Koch Ocker, Doug Ollerenshaw, Michael Oliver, Marius Pachitariu, led Perkins, Melissa Reding, David Reid, Miranda Robertson, Kara Ronellenfitch, Sam Seid, Cliff Slaughterbeck, Michelle Stoccklin, David Sullivan, Ben Sutton, Jackie Swapp, Carol Thompson, Kristen Turner, Wayne Wakeman, Jennifer D. Whitesell, Derrie Williams, Ali Williford, Rob Young, Hongkui Zeng, Sarah Naylor, John W. Phillips, R. Clay Reid, Stefan Mihalas, Shawn R. Olsen, and Christof Koch. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592(7852):86-92, April 2021. ISSN 1476-4687. doi: 10. (0.10388441586-000-000 020 -2000000001100
URL URLpttps::/ http::://www.eatt
com www./a/ticle///eeoooooooooeoeeee
Number: 7852
Publisher: Nature Publishing Group
Jascha Sohl-Dickstein, Peter B. Battaglino, and Michael R. DeWeese. New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow. Physical Review Letters, 107(22): 220601, November 2011, doi: 10. 103/Phys 107.220601. httlthtts::// / aps Publisher: American Physical Society.
107. 220601
G. Tkačik, Elad Schneidman, Michael J. Berry, and W. Bialek. Ising models for networks of real neurons. 2006.
13

---

Published as a conference paper at ICLR 2023
Gašper Tkačik, Olivier Marre, Dario Amodei, Elad Schneidman, William Bialek, and Michael Berry li. Searching for Collective Behavior Large Network of Sensory Neurons. PLOS Computational Biology, 10(1)=e1003400 January 2014. ISSN 1553-7358. doi: 10.1371/joumal pebi. 100..80 URL http:/ttt/////rrmeeateee
os org/ / www.ommbol//..occc.ccooooooo
10. 10.1371/ pebi wttpp....1000000 1003408
Publisher: Public Library of Science.
Peter Uhlhaas, Gordon Pipa, Bruss Lima, Lucia Melloni, Sergio Neuenschwander, Danko Nikolić, and Wolf Singer. Neural synchrony in cortical networks: history, concept and current status. Frontiers in Integrative Neuroscience, 3, 2009. ISSN 1662-5145, doi: 10.338%//euro.07.011..
URL.
https: // www. Frontiers, org/ art www/10eeellloo. 3389 neuro 07. 017 2009
Anne E. Urai, Brent Doiron, Andrew M. Leifer, and Anne K. Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature Neuroscience, 25(1):11-
19, January 2022. ISSN 1546-1726. doi: 10. (50000///1111000-0000000000
URL https:/ //
www. www.nattue.co.. com/ Publishing Group.
Number: Publisher: Nature
Antonio Vergari, Nicola Di Mauro, and Floriana Esposito. Simplifying, Regularizing and Strength- ening Sum-Product Network Structure Learning. In Annalisa Appice, Pedro Pereira Ro- drigues, Vítor Santos Costa, João Gama, Alípio Jorge, and Carlos Soares (eds.), Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pp. 343-358, Cham, 2015. Springer International Publishing. ISBN 978-3-319-23525-7. doi: 10.10077977--3319-2355----
Martin Vinck, Renata Batista-Brito, Ulf Knoblich, and Jessica A. Cardin. Arousal and Locomotion Make Distinct Contributions to Cortical Activity Patterns and Visual Encoding. Neuron, 86(3): doi:
740-754, May 2015, ISSN 0896-6273.
URLhttps:://
www. cew. www./////wwnn/nnnnee / S0896 627 (15) 00252-
Publisher: Elsevier.
He Zhao, Piyush Rai, Lan Du, Wray Buntine, Dinh Phung, and Mingyuan Zhou. Variational Autoencoders for Sparse and Overdispersed Discrete Data. Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, pp. 1684-1694, PMLR, June 2020. ISSN: 2640-3498.
URL httpps://proceeeciigg... mlr http:///g......pp..ssss//oaaaaa.
14


Reviews:
[{"review_id": "l678qAzCkU", "reviewer": "Reviewer_Lrts", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Learning the correlation structure of neuron population is tuff. Usually, only pair-wise linear correlations can be measured. It is hard to compute higher-order correlations in the population. The authors design a Sum Product Network (SPN) to learn higher-order correlations of a neuron population. Overall, the paper is well written, however, I feel not convinced that the current model could have a larger application. ", "summary_of_the_review": "I feel excited reading the first half of the paper, as the authors set up the problem and direct the audience very well. However, the result section is a bit disappointing. The keyword ‘higher order correlation’ is repeatedly mentioned in the paper, however, only indirect measurement (delta_ll) is provided to suggest that the model learns the correlation. Without more information about the model and better validation of the model, it is really hard to generalize what has been presented in the paper. ", "strength_and_weaknesses": "Strength: \nCompare to existing methods such as pairwise maximum entropy models, the SPN model is more computationally efficient and fits neuron spiking better. \nWeakness: \n1.\tThe interpretability of the current model is low.  \nThe authors show that the SPN models could use correlation information in predicting population neuron activities, as the current model performs better than the naïve Bayesian model which does not model neuronal interactions. However, it is not clear to me how to interpret higher-order correlations using the current model. Which part of the model learns the higher-order correlation? Can we read out pairwise correlation from the model parameters? \n2.\tNot well-supported interpretation\nThe authors suggest that the delta_ll value is a measure of ‘correlation’ (Fig. 5), which I found not well supported. First of all, the relationship between delta_ll and higher-order/linear correlations is not well established. In particular at the physiological neuron population. Second, it is a very low-resolution measure as one value (delta_ll) is computed for one dataset while the biological physiology (eg. pupil dynamics) changes much faster. Third, the fact that a control model (MPF) does achieve similar results on the majority dataset makes me confused even more about this measurement.  \n3.\tModel is only tested with simple data\nThe simulation data seems to only have a simple correlation structure (eq. 3). The correlation of the simulated neuron population is generated through common input, with no synaptic interactions or coupling. Also, it seems the whole population is receiving the same common time-varying input, which would probably make the correlation structure super simple. Plus the correlation between the simulation set is pretty high (>0.1). In addition, the real data set is also a simple dataset: visual neuron response evoked by four drifting gratings. Such a dataset is expected to have a low-dimensional correlation structure. It is not clear to me whether the model performance on these simple datasets would translate easily to more broad applications. Plus, other existing approaches like GLM might work as well on these simple data. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YNYulPMQawa", "reviewer": "Reviewer_S4f9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to model correlations between neurons in large populations which is based on sum-product networks (SPNs). They show that it outperforms two baselines (pairwise maximum entropy models and restricted Boltzmann machines) on a synthetic dataset and the Allen Brain Observatory neuropixels dataset.\n", "summary_of_the_review": "Potentially interesting and novel (within application domain) approach, whose performance is somewhat difficult to judge due to limited comparisons with existing work.\n\n### [Update after rebuttal]\n\nThe authors have included an additional baseline and clarified their contributions w.r.t. SPNs. I now support the paper's publication.", "strength_and_weaknesses": "### Strengths\n\n + Estimating correlations between neurons from limited data is a long-standing, but not completely solved problem\n + Approach seems promising and novel (to my knowledge), at least within the application domain\n + Explicit reproducibility statement and promise of code being published\n\n\n### Weaknesses\n\n 1. Important baselines for modeling neuronal correlations missing\n 1. Relationship to prior work on SPNs somewhat unclear\n\n\n### 1. Baselines\n\nAlthough the two baselines chosen by the authors (PME & RBM) are sensible choices, several alternative methods are missing. For instance, the literature on noise correlations has shown that common mode or low-rank fluctuations are abundant. To model such effects, latent variable models have been developed and yield state of the art performance. It is not clear to me why the authors consider an RBM \"the best latent variable model\" (p.2) - at least Koster et al. (2014) don't show that. Examples for alternative latent variable models include Poisson Linear Dynamical Systems (Macke et al. NeurIPS 2014) or deep gamma dynamical systems (Guo et al. NeurIPS 2018). Also relevant: Pfau et al., NeurIPS 2013, Ramesh et al., NeurIPS 2019 Neuro-AI workshop or very simple baselines like exponential family PCA or Factor Analysis. As it stands, it is not possible to tell from the evidence presented whether or not the authors' model is competitive.\n\n\n### 2. Relationship to prior work on SPNs\n\nOn p.3 the authors comment that \"there exist few algorithms for structure learning Gens & Pedro (2013); Vergari et al. (2015),\" but they developed their own method. Why not use existing methods? How does the method developed here differ from existing methods and how does it compare in terms of performance? While I am not familiar with the literature on SPNs, the present paper seems to mostly do knowledge transfer from other fields and apply an existing method in a new context. It doesn't become clear, however, where the authors make novel contributions because existing approaches from other fields are not applicable and need to be modified, and what these modifications are.\n\n\n### Other\n\n- On p.7 the authors comment that their \"model's main target is to capture the largest correlations with its available resources (free parameters).\" Why is this a reasonable thing to do in a model where all pairs are expected to have the same correlations due to the common input?\n\n- Related to the previous point, it is not clear to me why in a model where all correlations are driven by a single common input, the authors model should outperform an RBM by two orders of magnitude. What's the mechanism of this huge improvement? In such a simple data-generating process, I would expect any model with a single common mode to perform quite well. What additional inductive bias does the authors' model have that other baselines don't have?\n\n- I could not follow the description of how the authors partition the data on the bottom of p.7 / top of p.8. What does it mean to \"combine[d] the data of 5 consequent presentations with the same direction\"? How does this \"keep the state of the animal [...] as unchanged as possible\"? Why does this processing result in 15 blocks? Blocks of what? Relatedly, I couldn't follow the bottom of p.8 and what exactly Fig. 4, left, shows (e.g. what's the difference between \"SPN\" and \"SPN CV\"?)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9esNHNM-k4", "reviewer": "Reviewer_gQaa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method for modeling neural correlations\nbased on sum-product networks. The idea is that inference of the\nstatistical structure of neural population activity is made easier by\nthe assumptions encoded in the choice of this type of network, which\nenforce a hierarchical structure of alternating mixtures and\nfactorizations. Within this class of models, the model structure is\nchosen with a procedure based on hierarchical clustering computed from\nan information metric. Once the model structure is fixed, model\nparameters are learned by EM. The paper introduces sum-product\nnetworks, describes the proposed method, and compares it with existing\nmethods on synthetic data as well as on neural recordings.", "summary_of_the_review": "This is a good paper. The proposed technique is novel and original,\nand is shown to be effective at capturing the structure of neural\npopulation activity. The theory is well explained and the concrete\nbenchmarks are convincing.", "strength_and_weaknesses": "## Strengths\n1. The proposed method is novel and highly original.\n2. The comparisons with synthetic and recorded data convincingly show\n   that at least in the examples analyzed here the method outperforms\n   some notable baselines.\n3. The paper also does a good job of introducing sum-product networks\n   to a potentially broad audience.\n\n## Weaknesses\nIn my view this is a generally solid paper; however, here are some\nsuggestions for strengthening the paper further by increasing coverage\nof related literature.\n1. When discussing comparisons to existing approaches to model neural\n   correlations, it could be appropriate to mention recent works on the\n   statistical complexity of maximum entropy models for binary data\n   (Beretta et al 2018, De Mulatier et al 2020). These have shown that\n   the complexity of a model depends not so much on the order of the\n   interaction but more on their mutual arrangement, and also that it\n   is possible to efficiently search through the space of \"simple\"\n   maximum entropy models. I wonder if anything is known about\n   potential efficient approximations of the Bayesian evidence of\n   sum-product networks that could be used in a formal model-selection\n   context.\n2. When discussing neural correlations from a neuroscience perspective\n   and its relationship with brain state, consider citing Cohen and\n   Maunsell 2009, which has shown that attention modulates pairwise\n   correlations in monkey visual cortex, and that this modulation of\n   population structure has substantial behavioral implications.\n3. Another class of approaches aimed at uncovering the statistical\n   structure of population data that seem worth mentioning is that\n   based on GLMs (Pillow et al 2008). Despite their limitations, these\n   methods scale well to large populations and are applicable to\n   complex experiments involving rapidly-varying external variables\n   (see e.g. Runyan et al 2017).\n4. Moreover, it would be great if the authors could add the K-pairwise\nmodel (Tkacik et al 2014) to their comparisons in Figure 3 and 4. That\nmodel, which is already cited in the present paper, is considered a\nparticularly powerful (while practical) higher-order extension of\npairwise maxent models and seems like a good candidate to benchmark\nagainst. However this is just a suggestion that I think could make the\npaper's message even stronger, not something necessary for acceptance.\n\n### Typos:\n- page 5, bottom: \"if and only OF all neurons are independent...\"\n  (of→if)\n- page 7: \"The RBM performs better with SMILE size populations.\"\n  (smile→small)\n- page 7: \"their average log-likelihood difference (...) increases\n  with γ\". I think here γ should be replaced by λ.\n- please check for consistency of the symbol used for \"milliseconds\"\n  (or seconds). Sometimes it's ms, sometimes msc, sometimes msec. The\n  correct SI abbreviation is ms (s for seconds).\n\n### References:\n\nBeretta, Alberto, Claudia Battistin, Clélia De Mulatier, Iacopo Mastromatteo, and Matteo Marsili. 2018. “The Stochastic Complexity of Spin Models: Are Pairwise Models Really Simple?” Entropy 20 (10): 739. https://doi.org/10.3390/e20100739.\n\nMulatier, Clélia de, Paolo P. Mazza, and Matteo Marsili. 2021. “Statistical Inference of Minimally Complex Models.” arXiv. https://doi.org/10.48550/arXiv.2008.00520.\n\nCohen, Marlene R., and John H. R. Maunsell. 2009. “Attention Improves\nPerformance Primarily by Reducing Interneuronal Correlations.” Nature\nNeuroscience 12 (12): 1594–1600. https://doi.org/10.1038/nn.2439.\n\nPillow, Jonathan W., Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, and Eero P. Simoncelli. 2008. “Spatio-Temporal Correlations and Visual Signalling in a Complete Neuronal Population.” Nature 454 (7207): 995–99. https://doi.org/10.1038/nature07140.\n\nRunyan, Caroline A., Eugenio Piasini, Stefano Panzeri, and Christopher D. Harvey. 2017. “Distinct Timescales of Population Coding across Cortex.” Nature 548 (7665): 92–96. https://doi.org/10.1038/nature23020.\n\nTkačik, Gašper, Olivier Marre, Dario Amodei, Elad Schneidman, William Bialek, and Michael J. Berry. 2014. “Searching for Collective Behavior in a Large Network of Sensory Neurons.” PLOS Computational Biology 10 (1): e1003408+. https://doi.org/10.1371/journal.pcbi.1003408.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3lBv_c6Ni9q", "reviewer": "Reviewer_Ae2s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the use of Sum-Product Networks (SPNs) to study spike train recordings of populations of neurons. It compares SPNs against latent variable and energy-based methods on a synthetic problem: Correlations are estimated for a network of homogenous exponential integrate-and-fire neurons. It also applies SPNs to experimentally obtained neural recordings from the Allen Brain Observatory.", "summary_of_the_review": "The application of SPNs to neural population analysis is interesting and I am surprised this has not been explored previously. The results are promising, but empirical evaluation on synthetic examples with known ground truth is currently limited to a just a single problem. I think that a more thorough empirical analysis and a discussion of limitations would be critical for practioners considering application of SPNs to their data analysis problems.\n\n### Update after rebuttal\n\nI appreciate the addition of new experiments and updated my overall recommendation.", "strength_and_weaknesses": "**Strengths:** The use of SPNs for neural population analysis of spike train data is, to the best of my knowledge, a novel application. On problems and metrics considered in the paper, results overall show promising improvements over the baselines considered.\n\n**Weaknesses:** To me, the biggest weakness of the paper is its narrow empirical evaluation -- the authors only consider one example in which ground-truth is known. I would have hoped for more extensive evaluation on problems with synthetic data, in particular, considering heterogenous networks with known ground truth. In addition, I found a discussion of SPNs limitations for neural data analysis to be missing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster