Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a model of neural responses by capturing both stimulus dependent mean as well as variance. They transform the firing rate of each neuron using a reversible flow based model and model the transformed responses as a multi-variate gaussian. The mean of the multivariate gaussian is given by a neural network transformation of the input stimulus, and the low rank + diagonal covariance is independent of stimulus. They show the utility of this approach on synthetic as well as calcium imaging recordings from mouse visual cortex.
main_review: The paper is well written, performs relevant baseline comparisons and shows the utility on real neuroscience data. 

* I think the overall idea is quite novel. It might be helpful to describe the approach as learning one transformation of stimulus and another transformation of responses, and mapping them both to a common space where geometry is simpler. This general idea is quite different than most models in computational neuroscience that either learn encoding (stimulus-> response), or decoding (response-> stimulus) models. 

* My high-level comment is that the ideas can be simplified by removing the connection to flow models at many places. The paper uses (1) only per-neuron invertible transformations and (2) uses same transformation for each neuron, thereby not using the full power of current flow-based models.

* In terms of an ideal model, I think the main limitation is that the correlated noise in responses does not have a clear dependence on stimulus. That is, the degree of covariance between responses could be high for some stimuli and low for others in neuroscience data, but the model does not capture that. This limitation should be highlighted better in the text. 

* It seems that the transformation learned by the flow in Figure 3b is highly nonlinear and step-like (pink line). Would it be totally fine to lose the intertibility property of flow models then? All components of the model are not necessary for this real data case.

* Similarly, the ZIG model outperforms FAFLOW model for zero latents. Losing the invertibility of FLOW model for a transformation that looks more like ZIG (Figure 5) might be better overall (even for >0 factors).

* The conditional correlation plots in Figure 3a are not explained in the main text, and they seem to be the strongest results. 
* Moreover, why does flow model perform better than ZIG for 0 latents (where cells are modeled independently) for conditional correlation metric, but not for likelihood metric? 

* The interpretation of latents (Figure 4) could be cofounded by experimental factors. The spatial smoothness of latents could be explained by spatially smooth expression levels of calcium indicator, or spatially smooth recording noise. Either control for these confounds, or explain them as possible interpretation.

* The det(Jacobian) explanation in Figure 4 and text is not clear. Please explain them in non-mathematical terms to highlight the relevance/importance. 

* Color bars missing in Figure 4
limitations_and_societal_impact: no societal impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors present a Factor Analysis + flow model for modeling stimulus dependent neural responses. The FA + flow architecture allows the model to capture stimulus dependence, trial to trial correlated variability, and some features of the conditional and marginal response distributions. Validation is performed on synthetic data and real data from mouse visual cortex using 2 photon calcium imaging. The networks and inferred latent states are then compared to anatomical information and behavioral states.
main_review: The paper is clearly written and the results and figures are clear. The FlowFA model is a concise way of including both stimulus dependent and stimulus conditioned (correlated) variability into a model. The exploration of potential uses of this type of methods in Fig 4 is also compelling (although see the technical concern). I also think this model bridges the gap between shallow (Factor Analysis) and deep (unconstrained flow models) in a way that provides both interpretability and more accurate modeling which could serve as useful techniques in future models.

I have one technical concern about how some the latent state results are interpreted, some other clarifying questions/comments about the method and results, and a suggestion for the discussion.

- I think it would benefit a broad neuroscience audience to spend a little more time describing flow-based models in Section 2.1. Contrasting them with standard feed forward models which are probably more familiar might help, i.e., what mappings can you do with flow based models (like inverse) that is not available with regular models. You mention exact likelihood computation and sampling, but expanding on this would help.
- Providing a bit more information about the neural data in Section 2.4 would help. How is the response variable being preprocessed? I would assume is it florescence $\Delta F / F$, but this isn't specified. Do you truncate it to $>=0$?
- In Fig 3b, what latent dimensionality was used? Do you get qualitatively similar transformations as a function of latent dimensionality?
- Fig 4a, in the Cortex Position panels, it would be helpful to explicitly state what color indicates in the Cortex positions and RF positions panels in the figure caption. I'm assuming experimentally measured area assignment.
- For 4a, does the inclusion of correlated variability in the model improve the inferred estimates of $\delta$ compared to a model fit with $C=0$, i.e., don't take correlations into account but share parameters in the same way? This analysis isn't strictly necessary to perform, but it would strengthen the result.

Technical concern (and my main reservation for a higher rating)
- For Fig 4 b and c, $C$ is only determined up to a rotation in the latent space. Both the likelihood and equation for $\mathbf v$ are unchanged by taking $C\rightarrow CR$ for some rotation $R$. $CRR^TC^T=CC^T$ for the likelihood and the distribution of $\mathbf v$ is unchanged since $\mathbf z$ is isotropic. I believe this complicates the interpretation of Fig 4 b and c since it is not clear how sign relationships will be preserved under arbitrary rotations. There are schemes for defining a particular rotation [1] or a different measure of $C$ which is invariant to rotations could be shown, such as the leverage scores (see [2] for a definition).
- Due to the above point, it might make more sense to report the $R^2$ from regressing pupil dilation onto all latent states since the particular latent state basis is not meaningful.

1. Kaiser, H. F. (1959). Computer program for varimax rotation in factor analysis. Educational and psychological measurement, 19(3), 413-420.
2. Mahoney, Michael W., and Petros Drineas. "CUR matrix decompositions for improved data analysis." Proceedings of the National Academy of Sciences 106.3 (2009): 697-702.

- For the results in Fig 4d, specify how $p(z|r)$ is computed in the FlowFA model (maybe earlier in section 2.1). Line 258 implies a MAP estimate is used, but it is not clear.
- For the sentence on lines 127-129, are only the nonlinearities shared but affine parameters different per-neuron? or all all nonlinearities and affine parameters shared? If the later, did you consider a model where each neuron has an independent set of parameters? Was overfitting a problem? Do you get roughly the same transformation for every neuron anyway?


This is somewhat tangential to the main thrust of the paper, but the sentence that starts on line 297 in the Discussion contrasts healthy, functional brains and brains with neurological and psychological disorders. People with neurological or psychological disorders can have healthy, functional brains. For example, rates of depression in graduate students in the US have greatly increased during the pandemic, but the cause probably won't be found in the brains of the grad students.
https://www.nature.com/articles/d41586-020-02439-6

I would rephrase this sentence to lessen the contrast. Suggestion:
"Accurate models of neural variability such as the one presented here can lead to deeper scientific insights and understanding of how brains perceive and compute with sensory information, and can eventually also provide insights into neurological and psychological disorders."

Minor comments
- and this cannot -> and thus cannot
limitations_and_societal_impact: The authors provide some discussion of the limitations. I think the limitations of diffeomorphisms could be more clear. Is another way of stating it that the diffeomorphisms cannot map a distribution with 1 peak (Gaussian) to a distribution with potentially multiple peaks (neural response)?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In the paper "A Flow-based latent state generative model of neural population responses to natural images", the authors propose a model that can take into account stimulus driven stimulus-conditioned neural activity. The model is based on normalizing flows for the marginals and on a factor analysis model for modelling covariances and is validated on simulated data as well as data recorded from mouse visual cortex.
main_review: Originality:
To the best of my knowledge, the factor analysis approach combined with normalizing flows for the marginals is novel. That said, many non-cited models have previously been proposed, explicitly taking stimulus driven and stimulus-conditioned neural activity into account. For instance, Keeley, Aoi, Yu, Smith and Pillow, bioRxiv 2020 explicitly model signal and noise components in a Gaussian process factor analysis framework. Sorochynskyi, Deny, Marre and Ferrari, PLoS Comput Biol 2021 used a maximum entropy model and explicitly take noise correlations into account as well.

Quality:
The flow transformations are compared against two fixed transformations: square-root and Anscombe. The alternatives being fixed, it is not surprising that the flow model outperforms these fixed transformations. A comparison to a more flexible transformation such as a kernel density estimator would have been more insightful. Figure 5 suggests that even the ZIG model does a better job of modelling the marginals of the analyzed dataset.
Similarly, the comparisons to the control models are not fair. Contrary to the proposed model, both the independent Poisson and the ZIG model assume independence (line 137).
Tractable maximum entropy models (e.g. O'Donnel et al. Neural Comput 2017) or the dichotomized Gaussian model with appropriate marginals (Macke et al., Neural Comput 2009) would have been better contenders.

Clarity:
The clarity of the manuscript is fine but could be improved. In line 87, we learn that neural responses are represented as real numbers but it is not clear what these numbers mean until much later in the manuscript. Sometimes the model is called FAFlow (e.g. Fig 5), other times FlowFA (e.g. line 88).

Significance:
Models for taking both stimulus driven and stimulus-conditioned activity into account are important tools in systems neuroscience. Given the lacking comparisons, the value of the proposed model is unclear.
limitations_and_societal_impact: Limitations of the work with respect to the ZIP marginals are discussed to some extend in the Discussion but important points are left out. The FlowFA model assumes a stimulus-independent Gaussian dependence structure which is transformed by marginal stimulus-dependent nonlinearities. How flexible is the resulting dependence structure? Is the flow transformation of line 129 limiting? Such questions are not addressed in the manuscript. Societal implications with respect to animal experiments are briefly mentioned in the Discussion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose an algorithm based on a combination of a DNN-based model (that account for stimulus-variability) and a flow-based model (accounting for inherent neural variability) to model the behavior of neural responses. In this paper, the algorithm, called FA-flow (compared with ZIG, Poisson and ZA-Fixed) is trained on pairs of image/neural recording from bi-photon microscope on 2 mices. The authors claim is that their algorithm perform similarly to current state of the model neural response while additionally yielding latent states that exhibit meaningful relations to anatomical and functional properties.
main_review: ## General comments :
This article is in general interesting, and bring new Machine Learning concepts (i.e. the normalizing Flow) into computational Neuroscience. The main claim of the article, that is adding a flow on top of a DNN-based model allow to model both types of neural variability (i.e. stimuli variability and inherent neural variability) is not properly demonstrated. Nevertheless, the experiments conducted by the authors demonstrate the FA-Flow algorithm produce a relatively good fit of neural responses, but some of the conclusion made by the authors are not support by quantitative results (i.e. RFs mappings) or suffer from low statistical significance (see below for more details). In addition, some of the conclusion made by the authors are complicated to grasp (especially the conclusion from the Fig 4b and c).  My score : 5. 

## More specifics comments :
Section Methods : This section is clear and well written. Minor question : The authors are using 1000 V1 neurons to train the model on scan 1, and they report a total number of V1 neurons of 2867. For scan 2 the selection process is described (this is random) but  not for scan 1. Could please describe the neuron’s selection process for V1 neurons on scan 1 ?

### Section Results : 
#### Section 3.1 : 
I think a crucial experiment is missing in this section. One of the main claim of the authors is that flexible latent model better capture intrinsic correlated neural activity, and this claim is not demonstrated for their FA-Flow model. On my understanding the authors have all the datas to conduct such an experiment : some of the images are repeated 10 times (see line 172), thus one can see on these specific images how the FA-Flow model is fitting the intrinsic neural variability compared to the other models. Such an experiments would support the main claim of the article.

Figure 3 : Line 236, the authors state : « with Increasing latent dimension the FlowFA model … approaches the ZIG model in terms of likelihood’.  I do not see this improvement of log-likelihood with the increase of latent dimensions. As this is far from being obvious, the authors should conduct statistical test to support their claim.

Figure 4a : We see qualitatively that the FAFlow algorithm allows deducing a relatively good mapping between the RF position in the image space and neurons position in the cortex. But without further quantification of the quality of the mapping, it is hard to draw a solid conclusion. Is it possible to quantify the quality of the mapping ?

Figure 4b and c : I have trouble understanding the claim made by the authors : ’the effect of some latent states vary systematically accros brain area, as a function of RF location' (line 255). Could you please further explain how you are drawing this conclusion. Also a legend showing the meaning of the colors would help to better understand these 2 subfigures.

Figure 4d : Without further comparison it is complicated to say that ’some of the inferred latent state correlated strongly with pupil dilatation' (see line 259). For example in scan 1, the latent 3 accounts for only 41% of the variability (r2=0.41) which is usually not high enough to consider a good fit. Further statistical tests need to be conducted to be able to conclude on the significance of the prediction (and ideally a comparison with other r2 given in the literature).

limitations_and_societal_impact: The authors have not addressed the societal impact of their work
review_title: null

Meta-review: This paper introduces a system identification model for stimulus-driven and stimulus-conditioned fluctuations. They propose an architecture based on factor analysis and a flow model, and validate their performance on synthetic and mouse visual calcium imaging data. The paper addresses an important question in neuroscience, it is well written, introduces a new ML approach to computational neuroscience, and the authors have done a great deal to address all the concerns of the reviewers and convince them of the points they intended to make. Given this, I recommend this paper for acceptance and ask the authors to make all the modifications they promised in the final manuscript. 

Decision: Accept (Spotlight)