Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends AutoLFADS to deal with missing data. The proposed solution has two main elements: (i) the generative loss is simply zeroed-out at those (neuron, time) pairs where data is missing, and (ii) missing data are zero-filled before going into the encoder for amortized inference. I am not entirely convinced that this contribution crosses Neurips' high bar on technical & conceptual innovation. Element (i) above is probably the first thing that would cross one's mind for dealing with missing data in this particular case -- not sure the use of the fancy name "Selective backpropagation through time" is justified, where it simply corresponds to ignoring what the generator does at points where no data is available. Moreover, I do not see a clear a priori justification for using zeroes in element (ii). Finally, this hack is so simple that it doesn't strike me as a big modification of the existing AutoLFADS codebase (though I do appreciate the huge coding effort for the applications!). Overall, I therefore find this contribution rather incremental. Nevertheless, the case for tackling the problem of missing data is well made, and the experiments on a number of datasets are both impressive and convincing: the method just works, and I can see that it will be used in practice. Also, the paper very nicely shows that missing data, even when a lot is missing, isn't necessarily a deal-breaker for accurate reconstruction of latent dynamics.
main_review: Beyond the review summary, here a few points the authors might want to consider:

- Perhaps Bayesian methods (e.g. GPFA) ought to be given more credit in the introduction / prior art for their natural principled approach to dealing with missing data.

- Unpacking a point made in the summary: it is not clear to me why replacing missing data by zeros in the input to the encoder is "the" right thing to do... Do the authors agree that this choice was arbitrary? Could one consider replacing missing data by the mean spike count (or calcium) of the relevant neuron instead? Better yet, shouldn't these values be treated as variational parameters? I can see that they could affect the ELBO.

- Have the authors looked at how GPFA performs with missing data? I am curious to know how the rate at which cross-validated prediction errors deteriorates with increasing fraction of dropped samples compares with AutoLFADS'.

- Section 2 is a greatt overview of related work; I learned a lot there about refs I didn't know about!
 
----

Minor things:

- The first paragraph of section 2 is rather obscure / premature; e.g. what does "similar loss function to that considered in this work" refer to? Having already gone through Figure 1, the reader might reasonably expect that the "loss function considered in this work" already involves the masking sketched in Figure 1C, yet the paragraph ends "all of the above works limit their focus to [...] where the observations are fully sampled".

- typo: Possion emissions model → Poisson emission model

- "KL penalties described above": only one KL penalty is described (?)



==== Post-rebuttal update =====

Many thanks to the authors for their response, which consolidates my impressions. I had scored the paper at "marginally above acceptance threshold", already balancing the lack of technical innovation (which would normally lead to a clear reject in my mind) with the convincing experimental results and interesting conceptual framing of the study. To explain my reasoning very bluntly: does a neuroscientist with a missing data problem really need this paper? Yes and no. Yes, for the encouraging proof of concept that missing data isn't that bad a problem. No, because there is no substantial technical insight required to replace missing data with zeros and ignoring the reconstruction loss at these points. I am therefore reluctant to increase my score, out of fairness for the countless papers that make highly innovative algorithmic contributions yet will be rejected this year in this extremely competitive field -- I hope the authors will understand this stance! In any case, great work to be proud of.
 
 
limitations_and_societal_impact: Yes, limitations and societal impact are well addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce selective backpropagation through time (SBTT), and use this technique to improve the spatiotemporal resolution of AutoLFADS (a sequential autoencoder). This approach allows accurately inferring a low-dimensional dynamical system when samples are not recorded at all times. They demonstrate how this approach could be useful for Neuropixel recordings, when a large population of neurons is recording with a limited bandwidth (so that not all neurons are recorded at all times). They also demonstrate how this could be useful with Calcium imaging, in order to incorporate the more precise times that neurons are recorded during scanning (rather than assuming all neurons were recorded at once). 
main_review: Pros:

-This is a novel idea - I’m not aware of anyone who was trying to address these challenges. In fact, I hadn’t really thought about them as being challenges, so I think it could lead to a lot of new work in computational neuroscience.

-The manuscript is very thorough and high quality - there are demonstrations on simulated data and multiple neuroscience applications.

-The manuscript is clearly written, especially given the amount of information the authors are providing.


Cons:

-I don’t have any major concerns. I think the broad significance of the approach is TBD, as it’s hard to know whether this would be used with Neuropixels as proposed. But I am more confident this approach would be used in the near future for calcium imaging data. I encourage the authors to make this tool very user friendly to increase adoption of their method.


Minor comments:

For Fig. 2b,c:
-Can you provide a citation for how you calculated pseudo-R2 (there are different ways of defining it). And similarly, can you provide a citation for coefficient of determination (unfortunately, i’ve also seen this defined multiple ways).

It can be a little challenging to keep the meaning of the acronyms in my head. A couple recommendations (that you can take or leave):
-Just call A-ZIG-FR “A-ZIG”.
-Instead of “A”, use “ALFADS” (so ALFADS-ZIG). I realize “A” is shorter, but I would forget that A referred to AutoLFADS.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a training strategy, selective backpropagation through time (SBTT), for the deep structure to learn latent dynamics from irregularly or sparsely sampled time series data. The learned latent dynamics then can be used to infer activity for missing samples based on observations. The experimental results have demonstrated the effectiveness of the SBTT with various rates of dropped samples; the SBTT can recover some of missing information.
main_review:  The whole idea is interesting, but the authors should improve some parts of the draft before further consideration.
1)	The author should explain the details of how does the SBTT work with SVAE.
2)	What is the relationship between the two equations in lines 137 and 128?
3)	The experiments do not compare the effectiveness with other state-of-the-art methods, e.g. transformers, deepset and neural differential equations. Also, it would be better that the author provides recovered samples learned by other state-of-the-art methods to demonstrate the effectiveness of SBTT.
4)	The SBTT might need to be tested with other structures to show its scalability rather than just SVAE.

limitations_and_societal_impact: The authors should improve some parts of the draft mentioned in the above further consideration.
review_title: null

Meta-review: The reviewers and I agree that this paper presents a novel approach to extracting fine-timescale dynamics from seemingly slow calcium imaging data by leveraging the 2p laser scan pattern. From a purely technical perspective, the idea is straightforward: model the data at a much higher sampling rate but with a ton of missing data. This leads to some tricky inference problems since amortized variational inference with missing data isn't solved, but the authors find that simply zeroing out missing data before passing it to the encoder works well enough. The thorough analyses, impressive experimental results, and clarity of the text make this paper particularly notable and exciting for the NeurIPS crowd.
Decision: Accept (Poster)