Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method to model correlations between neurons in large populations which is based on sum-product networks (SPNs). They show that it outperforms two baselines (pairwise maximum entropy models and restricted Boltzmann machines) on a synthetic dataset and the Allen Brain Observatory neuropixels dataset.

strength_and_weaknesses: ### Strengths

 + Estimating correlations between neurons from limited data is a long-standing, but not completely solved problem
 + Approach seems promising and novel (to my knowledge), at least within the application domain
 + Explicit reproducibility statement and promise of code being published


### Weaknesses

 1. Important baselines for modeling neuronal correlations missing
 1. Relationship to prior work on SPNs somewhat unclear


### 1. Baselines

Although the two baselines chosen by the authors (PME & RBM) are sensible choices, several alternative methods are missing. For instance, the literature on noise correlations has shown that common mode or low-rank fluctuations are abundant. To model such effects, latent variable models have been developed and yield state of the art performance. It is not clear to me why the authors consider an RBM "the best latent variable model" (p.2) - at least Koster et al. (2014) don't show that. Examples for alternative latent variable models include Poisson Linear Dynamical Systems (Macke et al. NeurIPS 2014) or deep gamma dynamical systems (Guo et al. NeurIPS 2018). Also relevant: Pfau et al., NeurIPS 2013, Ramesh et al., NeurIPS 2019 Neuro-AI workshop or very simple baselines like exponential family PCA or Factor Analysis. As it stands, it is not possible to tell from the evidence presented whether or not the authors' model is competitive.


### 2. Relationship to prior work on SPNs

On p.3 the authors comment that "there exist few algorithms for structure learning Gens & Pedro (2013); Vergari et al. (2015)," but they developed their own method. Why not use existing methods? How does the method developed here differ from existing methods and how does it compare in terms of performance? While I am not familiar with the literature on SPNs, the present paper seems to mostly do knowledge transfer from other fields and apply an existing method in a new context. It doesn't become clear, however, where the authors make novel contributions because existing approaches from other fields are not applicable and need to be modified, and what these modifications are.


### Other

- On p.7 the authors comment that their "model's main target is to capture the largest correlations with its available resources (free parameters)." Why is this a reasonable thing to do in a model where all pairs are expected to have the same correlations due to the common input?

- Related to the previous point, it is not clear to me why in a model where all correlations are driven by a single common input, the authors model should outperform an RBM by two orders of magnitude. What's the mechanism of this huge improvement? In such a simple data-generating process, I would expect any model with a single common mode to perform quite well. What additional inductive bias does the authors' model have that other baselines don't have?

- I could not follow the description of how the authors partition the data on the bottom of p.7 / top of p.8. What does it mean to "combine[d] the data of 5 consequent presentations with the same direction"? How does this "keep the state of the animal [...] as unchanged as possible"? Why does this processing result in 15 blocks? Blocks of what? Relatedly, I couldn't follow the bottom of p.8 and what exactly Fig. 4, left, shows (e.g. what's the difference between "SPN" and "SPN CV"?)

clarity,_quality,_novelty_and_reproducibility: ### Clarity
Despite being very familiar with the topic of modeling neuronal correlations, I had a fairly hard time following the paper, mostly because I'm not very familiar with SPNs and the paper didn't guide me through the material in an intuitive way. 

### Quality
I have no doubts that the modeling is technically solid, but I find it hard to conclude whether the approach is indeed as strong as claimed by the authors because of the missing comparisons with alternative strong approaches.

### Novelty
Judging novelty is somewhat difficult, mostly because I'm not familiar with the SPN framework, and neither the limitations of existing work in that direction nor how the authors overcome them are clearly discussed in the paper.

### Reproducibility
On the positive side, the paper contains an explicit reproducibility statement and promises to published the code.

summary_of_the_review: Potentially interesting and novel (within application domain) approach, whose performance is somewhat difficult to judge due to limited comparisons with existing work.

### [Update after rebuttal]

The authors have included an additional baseline and clarified their contributions w.r.t. SPNs. I now support the paper's publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Learning the correlation structure of neuron population is tuff. Usually, only pair-wise linear correlations can be measured. It is hard to compute higher-order correlations in the population. The authors design a Sum Product Network (SPN) to learn higher-order correlations of a neuron population. Overall, the paper is well written, however, I feel not convinced that the current model could have a larger application. 
strength_and_weaknesses: Strength: 
Compare to existing methods such as pairwise maximum entropy models, the SPN model is more computationally efficient and fits neuron spiking better. 
Weakness: 
1.	The interpretability of the current model is low.  
The authors show that the SPN models could use correlation information in predicting population neuron activities, as the current model performs better than the naïve Bayesian model which does not model neuronal interactions. However, it is not clear to me how to interpret higher-order correlations using the current model. Which part of the model learns the higher-order correlation? Can we read out pairwise correlation from the model parameters? 
2.	Not well-supported interpretation
The authors suggest that the delta_ll value is a measure of ‘correlation’ (Fig. 5), which I found not well supported. First of all, the relationship between delta_ll and higher-order/linear correlations is not well established. In particular at the physiological neuron population. Second, it is a very low-resolution measure as one value (delta_ll) is computed for one dataset while the biological physiology (eg. pupil dynamics) changes much faster. Third, the fact that a control model (MPF) does achieve similar results on the majority dataset makes me confused even more about this measurement.  
3.	Model is only tested with simple data
The simulation data seems to only have a simple correlation structure (eq. 3). The correlation of the simulated neuron population is generated through common input, with no synaptic interactions or coupling. Also, it seems the whole population is receiving the same common time-varying input, which would probably make the correlation structure super simple. Plus the correlation between the simulation set is pretty high (>0.1). In addition, the real data set is also a simple dataset: visual neuron response evoked by four drifting gratings. Such a dataset is expected to have a low-dimensional correlation structure. It is not clear to me whether the model performance on these simple datasets would translate easily to more broad applications. Plus, other existing approaches like GLM might work as well on these simple data. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written. The quality of the work is marginal. The application of SPN models to fitting correlated neuronal data is novel. The data set used in the current study is open source. It is not clear to me whether the model of the current work would be released. 
summary_of_the_review: I feel excited reading the first half of the paper, as the authors set up the problem and direct the audience very well. However, the result section is a bit disappointing. The keyword ‘higher order correlation’ is repeatedly mentioned in the paper, however, only indirect measurement (delta_ll) is provided to suggest that the model learns the correlation. Without more information about the model and better validation of the model, it is really hard to generalize what has been presented in the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the use of Sum-Product Networks (SPNs) to study spike train recordings of populations of neurons. It compares SPNs against latent variable and energy-based methods on a synthetic problem: Correlations are estimated for a network of homogenous exponential integrate-and-fire neurons. It also applies SPNs to experimentally obtained neural recordings from the Allen Brain Observatory.
strength_and_weaknesses: **Strengths:** The use of SPNs for neural population analysis of spike train data is, to the best of my knowledge, a novel application. On problems and metrics considered in the paper, results overall show promising improvements over the baselines considered.

**Weaknesses:** To me, the biggest weakness of the paper is its narrow empirical evaluation -- the authors only consider one example in which ground-truth is known. I would have hoped for more extensive evaluation on problems with synthetic data, in particular, considering heterogenous networks with known ground truth. In addition, I found a discussion of SPNs limitations for neural data analysis to be missing. 
clarity,_quality,_novelty_and_reproducibility: **Novelty**: As stated above, to my knowledge, the paper explores a novel application of SPNs. SPNs have been developed in prior works, a technical novelty of this paper is the method for learning correlations in neural populations.

**Reproducibility:** Training details for baseline methods are currently missing, e.g., it is unclear how RBMs were trained exactly. The authors write that the code will be made available upon publication; unfortunately it was not possible to look these details up.

**Clarity**: For the most part, the paper is clearly written. 

**Minor comments:** Adding citations in brackets would improve readability. "smile size" -> "small size".
summary_of_the_review: The application of SPNs to neural population analysis is interesting and I am surprised this has not been explored previously. The results are promising, but empirical evaluation on synthetic examples with known ground truth is currently limited to a just a single problem. I think that a more thorough empirical analysis and a discussion of limitations would be critical for practioners considering application of SPNs to their data analysis problems.

### Update after rebuttal

I appreciate the addition of new experiments and updated my overall recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel method for modeling neural correlations
based on sum-product networks. The idea is that inference of the
statistical structure of neural population activity is made easier by
the assumptions encoded in the choice of this type of network, which
enforce a hierarchical structure of alternating mixtures and
factorizations. Within this class of models, the model structure is
chosen with a procedure based on hierarchical clustering computed from
an information metric. Once the model structure is fixed, model
parameters are learned by EM. The paper introduces sum-product
networks, describes the proposed method, and compares it with existing
methods on synthetic data as well as on neural recordings.
strength_and_weaknesses: ## Strengths
1. The proposed method is novel and highly original.
2. The comparisons with synthetic and recorded data convincingly show
   that at least in the examples analyzed here the method outperforms
   some notable baselines.
3. The paper also does a good job of introducing sum-product networks
   to a potentially broad audience.

## Weaknesses
In my view this is a generally solid paper; however, here are some
suggestions for strengthening the paper further by increasing coverage
of related literature.
1. When discussing comparisons to existing approaches to model neural
   correlations, it could be appropriate to mention recent works on the
   statistical complexity of maximum entropy models for binary data
   (Beretta et al 2018, De Mulatier et al 2020). These have shown that
   the complexity of a model depends not so much on the order of the
   interaction but more on their mutual arrangement, and also that it
   is possible to efficiently search through the space of "simple"
   maximum entropy models. I wonder if anything is known about
   potential efficient approximations of the Bayesian evidence of
   sum-product networks that could be used in a formal model-selection
   context.
2. When discussing neural correlations from a neuroscience perspective
   and its relationship with brain state, consider citing Cohen and
   Maunsell 2009, which has shown that attention modulates pairwise
   correlations in monkey visual cortex, and that this modulation of
   population structure has substantial behavioral implications.
3. Another class of approaches aimed at uncovering the statistical
   structure of population data that seem worth mentioning is that
   based on GLMs (Pillow et al 2008). Despite their limitations, these
   methods scale well to large populations and are applicable to
   complex experiments involving rapidly-varying external variables
   (see e.g. Runyan et al 2017).
4. Moreover, it would be great if the authors could add the K-pairwise
model (Tkacik et al 2014) to their comparisons in Figure 3 and 4. That
model, which is already cited in the present paper, is considered a
particularly powerful (while practical) higher-order extension of
pairwise maxent models and seems like a good candidate to benchmark
against. However this is just a suggestion that I think could make the
paper's message even stronger, not something necessary for acceptance.

### Typos:
- page 5, bottom: "if and only OF all neurons are independent..."
  (of→if)
- page 7: "The RBM performs better with SMILE size populations."
  (smile→small)
- page 7: "their average log-likelihood difference (...) increases
  with γ". I think here γ should be replaced by λ.
- please check for consistency of the symbol used for "milliseconds"
  (or seconds). Sometimes it's ms, sometimes msc, sometimes msec. The
  correct SI abbreviation is ms (s for seconds).

### References:

Beretta, Alberto, Claudia Battistin, Clélia De Mulatier, Iacopo Mastromatteo, and Matteo Marsili. 2018. “The Stochastic Complexity of Spin Models: Are Pairwise Models Really Simple?” Entropy 20 (10): 739. https://doi.org/10.3390/e20100739.

Mulatier, Clélia de, Paolo P. Mazza, and Matteo Marsili. 2021. “Statistical Inference of Minimally Complex Models.” arXiv. https://doi.org/10.48550/arXiv.2008.00520.

Cohen, Marlene R., and John H. R. Maunsell. 2009. “Attention Improves
Performance Primarily by Reducing Interneuronal Correlations.” Nature
Neuroscience 12 (12): 1594–1600. https://doi.org/10.1038/nn.2439.

Pillow, Jonathan W., Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, and Eero P. Simoncelli. 2008. “Spatio-Temporal Correlations and Visual Signalling in a Complete Neuronal Population.” Nature 454 (7207): 995–99. https://doi.org/10.1038/nature07140.

Runyan, Caroline A., Eugenio Piasini, Stefano Panzeri, and Christopher D. Harvey. 2017. “Distinct Timescales of Population Coding across Cortex.” Nature 548 (7665): 92–96. https://doi.org/10.1038/nature23020.

Tkačik, Gašper, Olivier Marre, Dario Amodei, Elad Schneidman, William Bialek, and Michael J. Berry. 2014. “Searching for Collective Behavior in a Large Network of Sensory Neurons.” PLOS Computational Biology 10 (1): e1003408+. https://doi.org/10.1371/journal.pcbi.1003408.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the work is of high quality and
novel. The method seems explained at a sufficient level of detail.
summary_of_the_review: This is a good paper. The proposed technique is novel and original,
and is shown to be effective at capturing the structure of neural
population activity. The theory is well explained and the concrete
benchmarks are convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper suggests using Sum-Product Networks (SPNs) for more accurate probabilistic modeling of neural populations. The results show improved accuracy compared to a few previous baselines. Reviewer had a few concerns, such as some missing baselines, but these concerns were sufficiently addressed, so eventually all reviewers recommend accepting the paper. I recommend the authors to finish the comparison with all the baselines mentioned in the review (or explain why some baselines were omitted). 
justification_for_why_not_higher_score: This is like a straightforward application of an existing method to a known (though long-standing) problem in neuroscience. Therefore, I don't see any specific reason it should be highlighted at ICLR.
justification_for_why_not_lower_score: It is a straightforward accept - all reviewers recommended accept ([8,6,8,8]), and there are no-major concerns remaining.
Decision: Accept: poster