Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study introduces a new metric (Directed Spectrum or DS) that is compatible with linear latent factor methods (LLF) and, therefore, can be used for latent network discovery with such methods. First, the authors show that there exists a linear relationship between the DS between channels and the DS in latent networks. Using simulated data, the authors estimate latent networks by applying NMF directly to the DS estimates. They show stronger correlations with latent activations recovered by their method, as compared to other traditional metrics like spectral GC, PDC, DTF, and the like. A specific comparison reveals that GC estimates far more spurious connections as compared to the DS metric. Finally, the authors also show that, in a neuroscience dataset, the latents estimated by DS features better predict the behavioral states as compared to latents estimated by traditional metrics. 
main_review: Preamble: I am quite familiar with measures of directed communication applied in neuroscience, and routinely employ these with our research data. But I am not an expert on the theoretical underpinnings of these measures. I will, therefore, append each critique with a confidence score (10/highest to 1/lowest), to enable the authors to optimally address my criticisms in the rebuttal, considering space constraints.

Originality: 7/10 

Quality: 7/10

Clarity: 6/10

Significance: 6/10

a) Regarding the comparison with spectral GC: 

i) There is a particular challenge with GC when inferring causality because the GC often estimates weights for both forward and (spurious) backward connections. In this case, differencing the estimates has been found to provide some control over the spurious backward connections (Roebroeck, Formisano, Goebel, 2005). I would suggest the authors use this metric (Fy->x(w) - Fx->y(w)) for their comparison rather than GC per se. 

ii) Another aspect that was unclear from the main paper was whether the authors used pairwise GC or conditional GC, although both have been defined in Supplemental section C.3. If they had used pairwise GC, could they also provide results with conditional GC (or both)?

iii) Is it possible that GC performs better with data from longer time windows? Fig 3 could be plotted for different lengths of windows (e.g. with mean and errorbors, rather than showing the estimates for every window/sample).

*Confidence: 10 


b) On a related note. Please show the directed influence spectra (analog of Fig.4) with the other approaches also, at least in summary form. Did these other approaches perform comparably with DS? The proportion of spurious connections estimated with all approaches should be quantified for comparison.

*Confidence: 10


c) (line 160) The authors indicate that "Assumption 1 can be thought of as enforcing the independence of inputs to each network". How valid is this assumption in a neurobiological setting? Of course, some components of the innovations may reflect "independent noise", but visual or other sensory inputs are likely to be highly correlated among networks (e.g. the LGN projects to multiple high level visual areas, in parallel). Can the authors clarify, and perhaps explore to what level the model is tolerant to a failure of this assumption?

*Confidence: 10 


d) The behavioral context decoding section needs to be updated significantly. One would need to see what accuracies can be obtained simply from the "raw" timeseries data or from simple spectral features extracted from these data, for each context. Having established this baseline, it would then make sense to show that DS features outperform these simpler features in terms of decoding behavioral state. Otherwise, the results could be simply explained by DS+NMF producing the least "distortion" in the raw data (given that the other methods are non-linear).

*Confidence: 10 


e) The authors seek to provide a thorough and detailed description, but some equations and derivations could be made clearer, at least in the Supporting Information. For example, the "transmitted signal" (based on the conditional density) introduced in equation 9 -- which is at the heart of the model -- could be derived in the supporting information with a clear explanation of how each term maps on to the stated assumptions in the immediately preceding paragraph.

*Confidence: 8


f) Given that the main contribution of the paper seems to be theoretical, the precise relationship between DS and traditional metrics should be clearly described, at least in the Supplemental information. Such a derivation would ideally follow the Supplemental Section E on the relationship with the cross-spectral matrix and its factorization. While Supplemental section C presents the equations associated with other metrics, such a description would allow the reader to also understand the relative advantages and limitations of the DS metrics vis-a-vis traditional metrics. 

*Confidence: 7


g) On a related note, a few more empirical simulations could be presented that explore the limits of the proposed metric, and its tolerance to violations of key assumptions. For example, what happens when latents mix non-linearly. Or network interactions characterized by operations other than scaling and delay? In the brain, it is known that the phase of slow oscillations modulates the amplitude of faster oscillations. Could these types of interactions also be captured by DS, and better than DTF or GC? 

*Confidence: 7


h) As an empirical control it would be nice to see a baseline comparison with latent linear factor models directly applied to the time series data, followed by directed influence estimation with DS or other traditional methods. In the case of DS I guess the order will not matter significantly, given linearity assumptions. But how do the other methods fare when NMF is applied to the data followed by, say, GC estimation on the latents compared to NMF on the channel-wise GC estimates? Are there certain interactions that can be recovered only by the latter method vs the former?

*Confidence: 6


i) I am not sure the sample subscript 'n' adds anything that cannot be understood without it. It shows up in every term and clutters up the equations, and gets dropped by mistake occasionally (Supplemental Section S1). I'd suggest removing it.

*Confidence: 6

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a method for estimating the directional influences between channels in order estimate the existence and structure of distinct, latent networks. They call their metric the "directed spectrum" and demonstrate its superiority over Granger causality to estimate directed influences between recording channels. The motivation for their approach is the paucity of directed influence metrics that operate in conjunction with latent variable models of neural population activity.
main_review: The paper is well written and flows nicely. The metric itself is fairly simple and it would be surprising if it didn't already exist under some other name. Although the authors appear to know and cite the relevant literature for this topic. One point of concern may be that the metric could have unfavorable sampling properties. For example, it is well known that common estimators for the cross-spectral matrix may be either inconsistent and/or high-variance. Does their estimator inherit these properties? Also, could the authors comment on how the metric might be influenced by model mismatch (eg.  generative models with non-linear dynamics, Poisson measurements).

While I don't mind some of their basic assumptions I find the arguments about innovations and independence (in particular in lines 148-153) somewhat weak. It is not at all obvious that brain networks wouldn't want to bounce existing information back-and-forth. The authors would have to substantiate this claim more throughly. That said, I would prefer to accept that this is a basic assumption of the model and being clear-eyed about how that may impact results rather than swallow the claim that somehow the brain prefers to operate this way.

I must admit that I am dubious of directed measures of neural population activity in general as I find it difficult to imagine a brain with recurrent connections showing anything other than directed activity in both directions. For example, the LFP data the authors present in the Supplement seems to indicate that none of the other brain areas direct their activity to the nucleus accumbens. Would we not expect VTA to act upon NAc? That does not appear to be the case based on the DS measure. How would the authors have us understand this inconsistency?

Lastly, I'm glad that the authors have explored this method and it's great that it performs so well on the classification task but I'm curious about the authors' thoughts on what new science can be done with this approach. Is this merely a tool for functional network discovery or did they have something else in mind?

As an aside, the authors may be interested in some recent work by Yousefi et al. (2019) describing the "global coherence" in which the authors develop an explicit latent variable model for network activity that is also designed to learn the identity of latent networks.

A. Yousefi, R. S. Fard, U. T. Eden and E. N. Brown, "State-Space Global Coherence to Estimate the Spatio-Temporal Dynamics of the Coordinated Brain Activity," 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019, pp. 5794-5798, doi: 10.1109/EMBC.2019.8856634.
limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors define the Directed Spectrum (DS) of multivariate timeseries data. It is shown that under certain conditions the DS is a linear function of latent brain networks. In synthetic data, the DS (and NMF) was better able to recover ground truth networks compared to a set of other directed measures. In neural data from mice, the network activations derived from DS are better able to predict behavioral state.
main_review: The motivation for the manuscript, careful consideration of the interplay between LLFMs and measures of neural data, and results were all clear. The DS is well motivated and is put into the context of other directed measures. I have one broad suggestion in terms of the presentation of Section 3 and some minor questions and suggestions.

In my opinion, the main weakness  is that Section 3 is very dense and relies on a number of definitions and results that are only in the appendix, which made it difficult to read through as a self-contained section. I think it would benefit the clarity of Section 3 to walk the reader through some definitions (e.g., how to interpret $H^{(j)}(\omega)$) in Eq 5 and the interpretations of the terms in certain definitions (e.g., Eq 9). Some of these are in the appendix, but I would suggest using some of the additional page to make this section more self-contained.

- I found that using $x$ for both the data vector and latent factors in Eq 1 to be confusing. I would suggest changing the latent factor variable.
- In lines 131-134, I think it would be helpful to have more exposition about when it could be useful to define groups versus single channels.
- For Section 6, is it possible to better differentiate whether the DS is better for decoding as an engineered features versus whether some small subset of the inferred networks are better for decoding. Since there are only 3 behavioral contexts, it would be interesting if a small set of network activations were responsible for decoding versus if many networks are being used.
- Should equations 12 and 13 be part of one multiline equation?
- Line 278: L1 factorization -> L1 regularization
limitations_and_societal_impact: The authors address limitations of the DS based on its assumptions of stationarity and its use with LLFMs. It would be good to know how sensitive the method is to violations of Assumptions 1 or 2 as they likely don't hold exactly in neural data. If there are weak correlations in the innovations or if the transmitted signals combine sub-additively, how quickly do the estimates break down?
review_title: null

Meta-review: Through a productive discussion with the authors, the reviewers came to a borderline/accept consensus. The proposed "directed spectrum" measure is simple (in fact, the authors say it was an introduced as an intermediate quantity in a 1982 paper by Geweke), but appears to have some nice properties. While some reviewers praised the clarity of this paper, Reviewer 3jsZ and I found the paper rather dense (and the unnecessary subscripts didn't help).  I hope the authors will work to improve this aspect of the paper before publication. 
Decision: Accept (Poster)