Deep inference of latent dynamics with spatio-temporal super-resolution using selective backpropagation through time
Authors: Feng Zhu
Andrew R Sedler
Harrison A Grier
Nauman Ahad
Mark A. Davenport
Matthew Kaufman
Andrea Giovannucci
Chethan Pandarinath
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computational neuroscience
Systems neuroscience
Neural population dynamics
intermittent sampling
Electrophysiology
Calcium imaging
Brain-computer interfaces
Neuroscience
Neuroprosthetics
Neural coding
Motor control
Sequential autoencoders

Abstract: Modern neural interfaces allow access to the activity of up to a million neurons within brain circuits. However, bandwidth limits often create a trade-off between greater spatial sampling (more channels or pixels) and the temporal frequency of sampling. Here we demonstrate that it is possible to obtain spatio-temporal super-resolution in neuronal time series by exploiting relationships among neurons, embedded in latent low-dimensional population dynamics. Our novel neural network training strategy, selective backpropagation through time (SBTT), enables learning of deep generative models of latent dynamics from data in which the set of observed variables changes at each time step. The resulting models are able to infer activity for missing samples by combining observations with learned latent dynamics. We test SBTT applied to sequential autoencoders and demonstrate more efficient and higher-fidelity characterization of neural population dynamics in electrophysiological and calcium imaging data. In electrophysiology, SBTT enables accurate inference of neuronal population dynamics with lower interface bandwidths, providing an avenue to significant power savings for implanted neuroelectronic interfaces. In applications to two-photon calcium imaging, SBTT accurately uncovers high-frequency temporal structure underlying neural population activity, substantially outperforming the current state-of-the-art. Finally, we demonstrate that performance could be further improved by using limited, high-bandwidth sampling to pretrain dynamics models, and then using SBTT to adapt these models for sparsely-sampled data.

Deep inference of latent dynamies with spatio-temporee super-resolution using selective backpropagation through time
Feng Zhu Andrew R. Sedler* Harrison A. Grier', Nauman Ahad', Mark A. Davenport",
Matthew T. Kaufman 366 Andrea Giovannucci 788.9 and Chethan Pandarinath 10,11
Neurrssiencce Graduate Program, Emory University CCetter for Machine Learning, Georgia Tech
Computational Neuroscience Graduate Program, The University of Chicago SScoool of Electrical and Computer Engineering, Georgia Tech Dept. of Organismal Biology and Anatomy. The University of Chicago
Neurosciennce Institute, The University of Chicago
Joint Dept. of Biomedical Engineering, UNC Chapel Hill and NC State University UNC Chapel Hill Neuroscience Center
'NC State/UNC Closed-Loop Engineering for Advanced Rehabilitation (CLEAR)
'CCoulter Dept. of Biomedical Engineering, Emory University and Georgia Tech DDept. of Neurosurgery, Emory University
Abstract
Modern neural interfaces allow access to the activity of up million neurons within brain circuits. However, bandwidth limits often create trade-off between greater spatial sampling (more channels sampling. Here we demonstrate that is possible to obtain spatio-tempol super-resolution in neuronal time series by exploiting relationships among neurons, embedded latent low-dimensional population dynamics. Our novel neural net- work training strategy, selective backpropagation through time (SBTT), enables learning of deep generative models of latent dynamics from data which the set observed variables changes at each time step. The resulting models able infer activity for missing samples by combining observations with learned latent dynamics. We test SBTT applied sequential autoencoders and demonstrate more efficient and higher-fidelity characterization of neural population dynamics clectrophysiological and calcium imaging data. In electrophysiology. SBTT enables accurate inference of neuronal population dynamics with lower interface bandwidths, providing an avenue sigmificant power savings for implanted neu- roelectronic interfaces. In applications to two-photon calcium imaging, SBTT accurately uncovers high-frequency temporal structure underlying neural popula- tion activity, substantially outperforming the current state-of tfthe-rrrr Finally, we demonstrate that performance could be further improved by using limited, high- bandwidth sampling to pretrain dynamics models, and then using SBTT to adapt these models for sparsely-sampeed data.
pixels) and the temporal frequency
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
* Contributed equally. Corresponding authors: @chu23@emory.edie {asedler, shaedler,chethan)de Shethha)

---

Introduction
Modern systems neuroscientists have access to the activity of many thousands to potentially millions of neurons via multi-photon calcium imaging and high-density silicon probes [H]] Such interfaces provide qualitatively different picture of brain activity than was achievable even a decade ago.
However, neural interfaces increasingly face a trade-off- - the number of neurons that can be accessed (capacity) often far greater than the number that is simultaneously monitored (bandwidth). For example, with 2-photon calcium imaging (2p: Fig. 1a, top), hundreds to thousands of neurons are serially scanned by a laser that traverses the field of view, resulting in different neurons being sampled at different times within imaging frame. As consequence, trade-off exists between the size of the field-of-view (and hence the number of neurons monitored), the sampling frequency, and the signal-to-noise with which each neuron is sampled. Whereas current analysis methods treat 2p data as if all neurons within a field-of-view were sampled at the same time at the imaging frame rate, the fact that each neuron is sampled at staggered, known times within the frame could be employed to increase the time resolution.
Electrophysiological interfaces face similar trade-offs (Fig, 1a, bottom). With groundbreaking high-density probes such as Neuropixels and Neuroseeker (35), simultaneous monitoring of all recording sites is either not currently possible or limits the signal-to-noise ratio, so users typically monitor selected subset of sites within given recording session. For example, Neuropixels 2.0 probes contain up to 5120 electrodes, 384 of which can be recorded simultaneously 4 In other situations, power constraints might make it preferable to restrict the number of channels that are simultaneously monitored, such as in wireless fully-implanted applications where battery life and heat dissipation are key challenges [68]] As newer interfacing strategies provide pathway to hundreds of thousands of channels for revolutionary brain-man machine interfaces [9] [0), neural data processing strategies that can leverage dynamic deployment of recording bandwidth might allow substantial power savings.
Solutions to these space-time trade-offs may come from the structure of neural activity itself. large body of work suggests that the activity of individual neurons within large population is not independent, but instead is coordinated through a rnver-dimensionl,, latent state that evolves with stereotyped temporal structure (Fig. lb). We can represent the state at time as a vector æ. E R that evolves according to dynamics captured by a function such that 1 R f(x)). Rather than
a
2P imaging
High S,
Fully observable
Latent states
Space
S/T
trade-olls
Frarre
High-density e-phys.
Low high T
Observed
Unobserved
activity
C
Sparse
Zero-filled
input
input
IC Encoder
Initial conditions
Generator
Factors
Rates
Sparse
input data
Time-varying
inputs
Encoder
Controller
Reconstruction
Figure 1: Exploiting space-time trade-offs in neural interfaces using SBTT. (a) 2-photon calcium imaging (top), individual neurons serially scanned at a low frame rate, resulting in staggered sample times. In modern clectrophysiologicl recordings (bottom), bandwidth or power constraints prevent simultaneous monitoring of all recording sites. (b) Observed neuronal activity reflects latent, low-dimensional dynamics (captured by the function f). (c) SBTT applied to a sequential autoencoder for inferring latent dynamics from neural population activity.

---

directly observing the latent state Z;. we observe neural activity that we represent as y, E R where Y, = h(xx)) for some function h. Due both the fact that f imposes significant amount of structure on the trajectory of the at is and the fact that we typically expect the dimension D) of a to be far smaller than the number of possible observations N, one might expect that it should be possible to estimate the Ey 's without observing every neuron at every time step (i.e., measuring only some of the elements of each yr). just just as we generally infer latent states from only a fraction of the neurons in given area. If so, principled exploitation of the space-time trade-off' of neural interfaces might achieve higher-fidelity or more bandwidth-efficie characterization of neural population activity.
To our knowledge, no methods have demonstrated inference of dynamics from data in which the set of neurons being monitored changes dynamically at short intervals. To address this challenge, we introduce selective backpropagation through time (SBTT; Fig, 1c), a method to train deep generative models of latent dynamics from data where the identity of observed variables varies from sample to sample. Here we explore applications of SBTT to state space modeling of neural population activity that obeys low-dimensional dynamics.
This paper is organized as follows. Section 2 provides an overview of related work. Section 3 details SBTT and its integration with sequential autoencoders for modeling neural population dynamics. Section 4 demonstrates the effectiveness of this solution in achieving more efficient and higher-fidelig inference of latent dynamics in applications to electrophyssological and calcium imaging data.
2 Related work
There is long and rich literature on methods for system identification, particularly in the case of linear dynamical systems. The last several years have witnessed a burst of activity in establishing a more robust theoretical understanding of when and how well these methods work. Particularly relevant to our approach, [1]] shows that under suitable conditions on the dynamical system, performing gradient descent on the reconstruction loss of observed data can provably recover the parameters of the system despite the nonconvexity of the problem. Additional guarantees are provided in [12 which make varying assumptions on the underlying dynamics and the observation function, the existence of an observable control input, and the stochasticity of the dynamical system. Adversarial noise models are further considered in [1]. We emphasize, however, that all of the above works limit their focus to linear dynamical systems where the observations are fully sampled, i.c., where all of y, Ha, œ; is measured for all t.
In the case of a linear observation model (y, = Hæ;) but where we observe only a subset of the elements of each y! the problem is reminiscent of the low-rank matrix completion problem [19] Specifically, by letting Y and X denote the matrices whose columns are given by the y: and at respectively, we can write Y = H. If D < N, this is a low-rank matrix, and hence could be recovered from a random sampling of O(D log N) elements of each column of Y [9]. However, this strategy essentially assumes that there is no relationship between the x; to obtain significant improvements by exploiting the dynamical structure among the o, imposed by f. Indeed, in 20 21 the authors show that if the dynamics f are known, then it is possible to significantly reduce the sampling requirements. However, the question of learning such an f from undersampled observations has again not been addressed in this literature.
would expect
In some application domains, there have been hints in this direction. In particular, in the related contexts of recommendation systems [22 23 and student knowledge tracking [24 (5) there have been successful empirical efforts aimed at learning dynamical systems for modeling how user preferences/Knowlldee change over time. While such approaches have also had to confront the issue of missing observations (items that are not rated or questions that are not answered), they are aided by the existence of rich sources of additional metadata (e.g., tags) that lead to fundamentally different approaches than what we take here.
Within our application domain, variety of methods have been developed to infer latent dynamical structure from neural population activity on individual trials, including those based on Gaussian processes [9]. linear 32 and switching linear dynamical systems B33 35). and nonlinear dynamical systems such as recurrent neural networks (3..99, hidden Markov models 40), neural
ODEs 41, and transformers 42). Variants of these methods accommodate cases where the particular
observed neurons change over long time periods (e-g., over the course of days) [36 43 44 but
these are not appropriate for cases where neurons are intermittently sampled on short timescales. As
3

---

described below, several of these methods would be amenable to using SBTT to adapt to intermittent sampling, as SBTT should be applicable to any neural network architecture that learns weights via backpropagation through time.
3 Selective backpropagation through time
3.1 Overview
SBTT is a learning rule for updating the weights of a neural network that allows backpropagation of loss for the portions of data that are present while preventing missing data from corrupting the gradient signal. The technique optimizes the model to reconstruct observed data while extrapolating to the unobserved data. The implementation of SBTT is related to other approaches that augment network inputs and cost functions to reflect different subsets of the data matrix across samples, in particular coordinated dropout (37, masked language modeling [45] and DeepInterpolation [46]. Though not designed for missing data, these previous approaches split fully-observed data into two portions portion that provided the input to the network, and portion that used to compute loss at the output. SBTT uses a similar strategy to accommodate missing data, by zero-filling missing input points and aggregating only losses for observed data points at the output. Though prior work has adopted similar strategy to handle missing data [47] the contribution of SBTT is integrating strategy with models with temporal component to learn dynamical system. To demonstrate SBTT, we provide code for a basic experiment using a sequential autoencoder and Lorenz dataset battps://githir com/ snel peel-repo/aee settt demo).
3.2 Illustration with simple linear dynamical system
We begin by describing our approach in the context of a simple linear dynamical system. In the case where we have no (observable) inputs, can model a linear dynamical system as
xr+ Aæ. + wt
yr = Hæ, + Zt-
Here, x E R represents hidden state, y RR- represents our observations, and w, and z, represent noise. The matrix A models the dynamics of the hidden state, and H models the observation function of our system. In this setting, our task to learn the parameters A and H given the observations Yo Yo..... 1r-1 well as the initial system state To-
SBTT is variation of standard back paok-propagation where loss terms attributed to missing observations are ignored when computing back back-propagation updates. Concretely speaking, consider a linear recurrent network that can learn this linear model using least squares loss
C Σ
ll H T
If the observation vector y/ contains a missing entry at index i, the least squares loss would not contain the (y) (Hzz)')² term, where the superscript represents the ith index of vector. If o, = Hæ, is taken to be the output of the recurrent network at time step t, then the loss with respect to the outputs of the network is
ac 1 -0: - y.).
0o: SBTT requires that loss terms, and subsequently loss gradients, related to missing observations are ignored. This means that elements in the gradient vector (1) are ignored and set to 0 at indices i where the corresponding observations, yi. are missing. This gradient is then back-propagated through time to obtain gradients with respect to model parameters A and H as shown below
0c T-1 ac
OL T-1 ac
T 0H do
x x__11
0A
dæ,
OL where dz,
is recursively computed using back- propppropagttion through time:
ac AT
OL
H OL
0x.
8x1+1
0o:

---

These parameters can then be updated using gradient descent.
3.3 Integration with a deep generative model of neural population dynamics
Here we will demonstrate the use of SBTT with a recently developed framework for inferring nonlinear latent dynamics from neural population recordings. This framework, Latent Factor Analysis via Dynamical Systems (LFADS), is a sequential variational auto-encoder (SVAE), detailed in [36]. LFADS models single-trial latent dynamics by learning the initial state of the dynamical system, the dynamical rules that govern state evolution, and any time-varying inputs that cannot be explained by the dynamics (i.e., in the case of non-autonomous dynamical system). Briefly, bidirectional RNN encoder operates on the neural spiking sequence y() and produces a conditional distribution over initial condition z, (()) A Kullback-Leibler (KL) divergence penalty is applied regularizer for divergence between the uninformative prior P(z) and Q(z(y(1)). The initial condition is then drawn from Q(zly (1)) and mapped to an initial state for a generator RNN, which learns to approximate the dynamical rules underlying the neural data. A controller RNN takes as input the state of the generator at each time step. along with a time-varying encoding of (t) (produced by a second bidirectional RNN encoder), and injects time-varying input u(t) into the generator. Similar to z, u(t) is drawn from a parameterized time- varying distribution of (t))))y(tt) produced by the controller. A second KL penalty is applied between P(u(t)) and ((u(t)ly(t)) At each time step, the generator state evolves with input from the controller and the controller receives delayed feedback from the generator. The generator states are linearly mapped to factors, which are in turn mapped to the firing rates of the neurons using a linear mapping followed by an exponential nonlinearity. LFADS assumes Possion emission model for the observed spiking activity. The optimization objective combines the reconstruction cost of the observed spiking activity (i.e., the Poisson likelihood of the observed spiking activity given the rates produced by the generator network), the KL penalties described above, and L2 regularization penalties on the weights of the recurrent networks. During training, network weights are optimized using stochastic gradient descent and backpropagation through time.
The first step in applying SBTT to LFADS is to zero-fill the missing data before feeding it into the initial condition (IC) and controller input (CI) encoders. After passing the data through the remaining hidden layers, we use the resulting rate estimates to compute reconstruction loss (Poisson negative log- (egliikeliood) for each observed
titroont and aggregate by taking the mean. The modified reconstruction loss is combined with other losses as in the standard LFADDS model. The network only optimizes for reconstruction of observed data and is free to interpolate at unobserved points.
Throughout this paper we use population-basee training along with coordinated dropout, together known as AutoLFADS, to optimize our models [37 39 48). This framework essential for achieving reliably
LFADS models, regardless of dataset statistics. Hyperparameters, search ranges, and training details are given in the supplement.
Experiments
4.1 High performance with limited bandwidth on primate electrophysiological recordings
A key target application of AutoLFADS with SBTT is to enable reduced sampling of electrodes: either to enable recording from larger populations of electrodes with limited bandwidth (such as with Neuropixels), or to reduce power consumption (such as for fully-imppantable brain-machine interfaces). To investigate the performance of AutoLFADS models trained with SBTT, we started with large and cell-characterized dataset containing electrophy sioooggicaa recordings from macaque primary motor and dorsal premotor cortex (MI/PMd) 49 50] The data were collected during a delayed reaching task, in which the monkey made both straight and curved reaches from a center position, around virtual barriers (the maze), to one of 108 possible target positions. The dataset consisted of 2296 trials with 202 sorted units aligned to movement onset in a window from 250 ms before to 450 ms after this point. Spike counts were binned at 10 ms (70 bins). We held out 50 randomly selected units from modeling to use for evaluation of inferred latent factors. We simulated various missing data scenarios for the remaining 152 units by randomly masking a fraction of the observations at each time step for each trial (Fig. 2a, top).
5

---

a
Fraction of dropped samples
0.0
0.4
0.7
0.9
(152)
Inferred Impul
(152)
200 ma
1.00
AutoLFADS
GPFA-normaized pRc
Decoding FP
0.75
2-
GalA
0.25
NET
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Fraction dropped samples
Fraction dropped samples
Figure 2: SBTT allows inference of latent dynamics from M1/PMd electrophysiology data with sparse observations. (a) Spike count input and inferred rate output of LFADS for the same example trial with increasingly sparse observations. Masked data are shown in white, observed zeroes are shown in light purple, and nonzero spike counts are shown in darker shades. Units are sorted by timing of firing rate peaks for the fully sampled model. (b) Accuracy of linear hand velocity decoding from inferred latent factors. (c) Quality of GLM fits from inferred latent factors to 50 held-out units. p/t² values for each held-out unit are normalized to the corresponding values achieved by the GPFA baseline. Points denote the median across all units. Shaded areas depict the 25th and 75th quantiles.
For each of the masked datasets, we used AutoLFADS with SBTT to robustly train neural dynamics models. Latent factors and firing rates were inferred for all time steps, despite the missing (masked) observations. Even with 70% dropped samples, the inferred firing rates showed structure comparable to the model of fully observed data (Fig. 2a, bottom).
To determine whether the models were able to capture biologically relevant information from sparsely sampled data, we evaluated the inferred latent factors in terms of their ability to predict hand velocity (Fig. 2b) and the spiking activity of held-out units (Fig. 2c). As a recognizable bascline, we trained Gaussian Process Factor Analysis (GPFA) model (40 latent dimensions, 20 ms bins) on the fully observed dataset [26 51]. GPFA is commonly-used and versatile method for extracting latent structure from neural population activity, and these parameters have been validated on this dataset in prior work [36]. We trained simple linear decoders to predict hand velocity from the inferred latent factors with an 80 ms delay (50/50, trial-wise train-test split), and evaluated using the coefficient of determination, averaged over and y-dimensions. For AutoLFADS with SBTT, decoding performance showed minimal decline until around 80% of data had been dropped, with some models outperforming the GPFA baseline using as little 15% of the original data (Fig, 2b). To measure how well the models captured the population structure, we trained generalized linear models (GLMs) 152 53) to predict the spikes for the held out units and evaluated fit quality using pseudo-R R2 (pR²). Similar to the decoding results, we found that AutoLFADS with SBTT captured population structure significantly better than fully observed GPFA, and that the information content of the factors declined slowly until about 80% missing samples (Fig. 2c). More detail on the R² and 2R² metrics can be found in Supplement Section E.
To evaluate the importance of modeling latent dynamics accurate inference with sparsely observed data, we also trained NDT with selective backpropagation the same datasets [2). We found that decoding performance from inferred firing rates declined faster than for AutoLFADDs with SBTT, but NDT still outperformed GPFA with up 40% missing data (Fig. 2b).

---

a
ALFADS-SBI
7 Hz
15 Hz
ALFADS
Latent recovery (FF)
Smoothing
Smoothing ALFADS ALFADS-SBTT Smoothing
ALFADS ALFADS-SBTT
WAy MA MN
0.8-
0.6-
MAY My MA Mr wWW wWM wM
0.4-
0.2
0
4
7 10 1315 20
Frequency of Lorenz Z (Hz)
Figure 3: SBTT improves inference of high-frequency dynamics from simulated 2P data with known dynamical structure. (a) True and inferred Lorenz latent states (X/Y/Z dimensions) for a single example trial from Lorenz systems simulated at two different frequencies (7Hz and 1 15Hz). Black: ground truth. Colored: inferred. (c) Performance in estimating the Lorenz Z dimension as a function of Lorenz speed was quantified by variance explained (R²²) for all three methods. The speed of the Lorenz dynamics was quantified based on the peak location of the power spectra of Lorenz Z. dimension, with a sampling frequency of 100Hz.
4.2 Recovery of high frequency features in simulated 2P calcium imaging data
high-frequency features of neural responses are generally assumed to be lost in 2P imaging due to limited scanning speeds and indicator kinetics. We hypothesized that some of the loss is actually due to standard 2P data processing, which discards information regarding sub-frame sampling time of individual neurons, and that SBTT could recover some of this information. The inherently staggered sampling of neurons due to raster scanning can be treated as time series with missing values and higher temporal resolution than the frame rate. We tested SBTT on both simulated and real calcium imaging data. In both cases, we adapted AutoLFADD to better account for the statistics of deconvolved calcium activity (AutoLFADS- ZIG, see supplement) by substituting the underlying Poisson emission model with Zero-Inflated Gamma distribution (54). In our experiments compared three methods: AutoLFADS-ZUI with SBTT (ALFADS- SBTT), a standard frame-resolution version of AutoLFADS- ZIG without SBTT (ALFADS), and Gaussian smoothing of deconvolved calcium activity.
We generated artificial 2P data from a population of simulated neurons (278 neurons) whose firing rates were linked to the state of an underlying Lorenz system [28] 55 (see supplement). To assess the ability to reconstruct latent dynamics at different frequencies, we simulated Lorenz systems with different speeds. For each Lorenz system we report the Z dimension power spectrum peak, which contains the most concentrated and highest frequencies. Fluorescence traces were simulated from the spike trains using an order autoregressive model followed by a non- non-linearity and injected with 4 sources of noise (see supplement). Firing rates were simulated with a sampling frequency of 100Hz, and a "location" was randomly chosen for each simulated neuron, such that sampling times for different neurons were staggered to simulate 2p laser scanning sampling times. This produced fluorescence traces with one of three possible associated phases (0, 11.2mms) and overall sample rate 33 Hz. We deconvolved neural activity from the fluorescence traces using the OASIS algorithm [56) implemented in the CalmAn package (57).
For AAFADS-SBTT we used the sub-frame phase information to generate internittently-sampled data. In contrast, for both ALFADS and Gaussian smoothing, we discarded phase information and collapsed samples into a single time bin per frame, as is standard in 2p imaging data processing. To evaluate the performance in recovering the ground truth Lorenz states, we trained mapping from the output of each method (i.e., the inferred event rates from ALFADS-SBD -BATT and ALFADS, and smoothed deconvolved events by Gaussian smoothing; signals were interpolated to 100 Hz for the latter methods) to the ground truth Lorenz states using cross-validate ridge regression. We used R? between the true and inferred Lorenz states as a metric of performance.
The true and predicted Lorenz states for two example trials are illustrated in Fig. 3a. The performance of smoothing and A-FR dropped substantially for higher Lorenz state frequencies. while A-SBTT maintained reasonable estimates (R² xs 0.8) up to 15Hz (Fig. 3a & b) and never dropped below 0.4 in the range of tested frequencies.
7

---

4.3 Improved representation of hand kinematics in mouse 2P calcium imaging data
We next applied SBTT to real 2P calcium imaging data collected from motor cortex in mouse performing a forelimb water grab task. The dataset comprised 475 trials in which the mouse was cued by tone reach to left or right spout and retrieve a droplet of water with its right forepaw. Pyramidal cells expressing the GCaMP6f calcium indicator were imaged with a two-photon microscope 31 Hz frame rate, and a subset of 439 modulated neurons within the field-of-view (FOV) were considered for analysis (FOV shown in Fig. 4a, left; example calcium traces in Fig, 4a, right). The mouse's forepaw position was tracked 3D at 150 Hz with stereo cameras and DeepLabCut (58). Calcium events were deconvolved with OASIS [56.5]. 57].
2P data for ALFADS-SAA SBTT were processed analogously to the simulations, using neuron locations within the FOV to inform the intermittent sampling times. Trials represented a window spanning 200 ms before 800 after the mouse's 's reach onset. This resulted in 100 time points per trial for ALFADS-SBTT, and 31 time points per trial for ALFADS and Gaussian smoothing. For both ALFADS-SBTT and ALFADS, trials were split into 80/20 train/validation.
To compare representations inferred by ALFADS-SBTT and ALFADS, we first evaluated how closely the single-trial event rates inferred for each neuron resembled that neuron's peri-stimulus time histogram (PSTH). PSTHs were calculated by taking the average of the Gaussian-smootiee veconvolved events across trials within each experimental condition. Because the mouse's reaches were not stereotyped to each spout (i.e., left or right), we subgrouped trials into 4 finer conditions based on forepaw Z position during the reach. ALFADS-SBTT single-trial event rates were more strongly correlated with neurons' PSTHs compared to those inferred by ALFADS (Fig. 4b).
We next decoded the mouse's single-trial forepaw kinematics (position and velocity) based on each model's output. Decoding was performed using ridge regression with 5-fold cross validation. We used R2 between the true and predicted hand positions and velocities as a metric of performance. R2 was averaged across XYZ behavioral dimensions and all 5 folds of the test sets. Decoding using ALFADS-SBTT inferred rates outperformed results from smoothing deconvolved events, from the ALFADS inferred rates (Fig. 4c). Because the improvement of decoding performance for position is modest, we further assessed how the improvement was distributed a function of temporal frequency. We computed the coherence between the true and decoded positions for each method (Fig. 4d). Consistent with the simulations, ALFADS-SAS SBTT predictions showed higher coherence with true position than predictions from other methods, with improvements more prominent at higher frequencies (5-15Hz).
4.4 Using bandwihhh observations to improve performance in low-bandwidth conditions
In implantable or wireless applications, using the device's full interface bandwidth might incur significant power costs, which would burden users with frequent battery recharging. However, may be possible to leverage high- bandwidh recordings from limited time periods to learn models of latent dynamics, and then switch minimize ongoing power use. Such an approach is enabled by the stability of latent dynamics over
bandwidhh modes for subsequent long- term operation, in order to
months to years 36 44 59
After training AutoLFADS models on the fully sampled data, we retrained the initial condition and controller input encoders using SBTT on each of the sparsely sampled datasets. The weights for the rest of the network remained fixed. In this way, the dynamical rules learned from the fully sampled data are maintained, while the mappings from data to the initial conditions and controller inputs are adapted for sparse data. Retraining the encoding networks in this way (Fig, 5, "Retrained sparse") maintained performance to high levels of missing data, outperforming AutoLFADS trained on fully observed data but run with missing data (Fig. 5, "Trained full") or training directly on pparsely-samples data (Fig. 5, "Trained sparse", same as in Fig. 2b). These results show that dynamics models are learned most accurately on fully observed data, but that the learned dynamics can be used to model sparsely sampled data if models are adapted to the sparser domain using SBTT.
We tested these ideas on the same electrophysiological dataset described in section 4.1
8

---

b
FOV
Calcium Traces
ALFADS-S0 SBTT
100 pm
51
83
0.5
77
B 1AFIF
50
29
0.5 ALFADS
ALFADS-SBTT ALFADS Smoothing
0.88
0.92
Position
0.75
Docoding
0.6
Coheremoe
0.8
0.52
0.6
0.4
0:36
0.4
0.2
10 15
Position
Velociy
Frequency (Hz)
Figure 4: SBTT improves inference of latent dynamics from mouse 2P calcium imaging data. (a) Left: an example field-of-view (FOV), colored by neurons. Right: calcium traces (dF/F) from single trial for 5 example neurons. (b) Performance of capturing empirical PSTHs quantified by computing the correlation coefficient r between the inferred single-trial event rates and empirical PSTHs, comparing ALFADS ALFADS-SBI Decoding performance was quantified by computing the R² between the true and decoded position (left) and velocity (right) across all trials. (d) Quality of reconstructing the kinematics across frequencies was quantified by measuring coherence between the true and decoded position for all three methods.
Each point represents individual neuron. (c)
a
b
Fraction of drapped samples during inference
D.0
0.8
0.9
1.0
(152)
0.8
Docoding FR
Retrained sparse Trained
Trained full Trained sparse
(182)
0.2
Refrained sparse
0.0
Fraction dropped samples
200 me
Figure 5: Retraining full-data LFADS encoders on sparse data improves decoding performance. (a) Hand velocity decoding performance function of dropped samples (as in Fig. 2b). "Trained full" indicates training fully observed data and inference sparse data. "Trained sparse" indicates training and inference on sparse data. "Retrained sparse" indicates training on fully observed data, followed by encoder retraining and inference sparse data. (b) Spike count input and inferred rate output of LFADS. Conventions are as in Fig. 2a.

---

5 Discussion
We introduced SBTT, a novel approach for learning latent dynamics from irregularly or sparsely sampled time series data. In experiments on real electrophysioloyy data from macaque motor cortex, we show that models trained with SBTT learn biologically relevant neural dynamics with up to 80% masked training data. On data from a synthetic 2P calcium imaging simulation, we show that models trained with SBTT capture high frequency features of the latent dynamics that are not captured at frame resolution. We also showed improved behavioral decoding performance on real 2P imaging data from mouse M1. Finally, we demonstrate that retraining the early layers of full-data model on sparse datasets using SBTT can substantially improve decoding performance at the most challenging sparsity levels, outperforming models trained on the sparse data alone. Taken together, these results clearly show that SBTT is valuable technique for training models with irregularly sparsely sampled time series data.
5.1 Limitations
Though we made an effort to characterize performance across multiple potential applications, it remains untested how this approach would generalize to other experimental settings (microscopes, calcium indicators, expression levels), model systems, and brain areas or tasks with more complex or higher-dimensional dynamics [39]. but we are optimistic that these propertics will extend to AutoLFADS models that use SBTT in other settings. Applications to brain-machine interfaces await incorporation neural network-basee dynamics models into closed-loop, real time systems. We also note that hardware implementations of intermittent sampling for electrophysiology largely unexplored, and might incur time or power costs when switching between channels. This might change the point at which intermittent sampling is beneficial from power performance perspective. We hope that this work indicates new directions for future generations recording hardware that focus high interface capacities and rapid switching between contacts.
still
5.2 Broader impact
Our results could pave the way to substantially decreased power consumption for fully-implantable wrain-machins interfaces. Ultimately, this should result in more reliable and less burdensome assistive devices for people with disabilities. Further, expanding the information that can be gathered through given recording bandwidth has scientific implications, and could enable neuroscientists to ask new questions via larger-scale studies of the brain.
Like any resource- inteurcce-teeivveeeeee technology, this technique has the potential to increase inequity by only benefiting those who can afford the most advanced neural interfaces. Efforts to deploy such technologies should weigh input from ethicists to ensure that everyone benefits from these scientific innovations [60
Acknowledgments and Disclosure of Funding
We thank M. Rivers and R. Vescovi for help with the real-time camera setup, and D. Sabatini for contributions to the behavioral control software. This work was supported by the Emory Neuromod- ulation and Technology Innovation Center (ENTICe), NSF NCS 1835364, NIH Eunice Kennedy Shriver NICHD K 1HD003994 the Simons Foundation as part of the Simons-Emory International Consortium Motor Control (CP), the Alfred P. Sloan Foundation (CP, MTK), NSF Graduate Research Fellowship DGE-2039655 (ARS), NSF NCS 1835390, The University of Chicago, the Neuroscience Institute at The University of Chicago (MTK), and Beckman Young Investigators Award (AG). The work was also supported by the following collaborative awards (PI: Prof. Ellen Hess, Emory): NIH NINDS R21 NS11631. Imagine, Innovate and Impact (I3) Funds from the Emory School of Medicine and through the Georgia CTSA NIH UL1-TR003378, and pilot grant from the Emory Udall Center of Excellence for Parkinson's Research. The authors declare no competing interests.
10

---

References
[1] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometrr of population responses in visual cortex. Nature,551177666 571(7765): 361-365, 2019.
[2] Jeffery Demas, Jason Manley, Frank Tejera, Hyewon Kim, Francisca Martinez Traub, Brandon Chen, and Alipasha Vaziri. High-speed, cortex-wide volumetric recording of neuroactivity at cellular resolution using light beads microscopy. bioRviv, 2021.
[3] James J Jun, Nicholas A Steinmetz, Joshua H Siegle, Daniel J Denman, Marius Bauza, Brian Barbarits, Albert K Lee, Costas A Anastassiou, Alexandru Andrei, Çagatay Aydin, et al. Fully integrated silicon probes for high-density recording of neural activity. Nature, 551(7679): 232-236, 2017.
[4] Nicholas A Steinmetz. Cagatay Aydin. Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, et al. Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science, 372(6539), 2021.
[5] Bogdan C Raducanu, Refet Yazicioglu, Carolina M Lopez, Marco Ballini, Jan Putzeys, Shiwei Wang, Alexandru Andrei, Veronique Rochus, Marleen Welkenhuysen, Nick van Helleputte, et al. Time multiplexed active neural probe with 1356 parallel recording sites. Sensors, 17(10):
2388, 2017.
[6] Henrique Miranda, Vikash Gilja, Cindy A Chestek, Krishna Shenoy, and Teresa Meng. Hermesd: A high-rate long-range wireless transmission system for simultaneous multichannel neural recording applications. IEEE Transactions on Biomedical Circuits and Systems, 4(3):
181-191, 2010.
[7] David A Borton, Ming Yin, Juan Aceros, and Arto Nurmikko. An implantable wireless neural interface for recording cortical circuit dynamics in moving primates. Journal of neural engineering, 10(2):026010, 2013.
[8] John Simeral, Thomas Hosman, Jad Saab, Sharlene N Flesher, Marco Vilela, Brian Franco, Jessica Kelemen, David M Brandman, John G Ciancibello, Paymon G Rezaii, et al. Home use of percutaneous wireless intracortical brain-computer interface by individuals with tetraplegia. IEEE Transactions on Biomedical Engineering, 2021.
[9] Kunal Sahasrabuddhe, Aamir A Khan, Aditya P Singh, Tyler M Stern, Yeena Ng, Aleksandar Tadić, Peter Orel, Chris LaReau, Daniel Pouzzner, Kurtis Nishimura, et al. The argo: high channel count recording system for neural recording vivo. Journal of Neural Engineering, 18 (1):015002, 2021.
[10] Elon Musk et al. An integrated brain-machine interface platform with thousands of channels. Journal of medical Internet research, 21(10):e16194, 2019.
[11] Moritz Hardt, Tengyu Ma, and Benjamin Recht. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research, 19(29): 2018. URL http://jmmr
prg/papers/v19/116 11111/9166666111111 4tm.
[12] Max Simchowitz, Horia Mania, Stephen Tu, Michael I. Jordan, and Benjamin Recht. Learning without mixing: Towards sharp analysis of linear system identification. In Proc. Conference on Learning Theory (COL.T), volume 75 of Proceedings of Machine Learning Research, pages
439-473. PMLR, July 2018
[13] Elad Hazan, Holden Lee, Karan Singh, Cyril Zhang, and Yi Zhang. Spectral filtering for general linear dynamical systems. In Adv. in Neural Information Processing Systems (NeurIPS), Montréal, Canada, December 2018.
[14] Samet Oymak and Necmiye Ozay. Non-asymptotic identification of Iti systems from a single trajectory. 2019 American Control Conference (ACC), pages 5655-5661, 2019. doi: 10. 23919/ACC.2019 14438.
11

---

[15] Anastasios Tsiamis and George J. Pappas. Finite sample analysis of stochastic system identi- fication. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 1654-3654, 2019. doi: 10. DD99CCDDC40024221199999999
[16] Holden Lee and Cyril Zhang. Robust guarantees for learning autoregressive filter. In Aryeh Kontorovich and Gergely Neu, editors. Proceedings of the 31st International Conference on Algorithmic Learning Theory, volume 117 of Proceedings of Machine Learning Research, pages 190-517, San Diego California, USA, February 2020. PMLR. URLhttpp htttp:///rroceedings mlr. wttppp//vvi/eeeeea........ html
[17] Max Simchowitz, Ross Boczar, and Benjamin Recht. Learning linear dynamical systems with semi-parametric least squares. In Alina Beygelzimer and Daniel Hsu, editors, Proc. Conference on Learning Theory (COLT), volume 99 of Proceedings of Machine Learning Research, pages 2714-2802, Phoenix, USA. June 2019. PMLR. URL http::/ppoccedding mlr mwr.mress. press/
r999ssimchowitz19a..oo
[18] Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Jacob Abernethy and Shivani Agarwal, editors, Proc. Conference on Learning Theory (COLT), volume 125 of Proceedings of Machine Learning Research, pages 0920-3436. PMLR, July
2020. httptttt::///ppt
mlr pwwss//e2....//////ppppppppppppe
v125/ htt////mccoiittt220 tz20a. html
[19] Mark A. Davenport and Justin Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE J. of Selected Topics in Signal Processing, 10(4))608-6) 622600886622000 June 2016.
[20] Liangbei Xu and Mark Davenport, Dynamic matrix recovery from incomplete observations under an exact low-rank constraint. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 29. Curran Associates, Inc., 2016. URL https::/ hww.:ee.eeiiii....ee cttccc///pppa/cc cc/pp/e//cc/ppp ff116669544411100006666666666888 Paper pdf
[21] Liangbei Xu and Mark A. Davenport. Simultaneous recovery of series of low-rank matrices by locally weighted matrix smoothing. In 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pages 1-5, 2017. doi: 10.1 109/ CAMSAP.2017.835 1733111
[22] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. November 2015.
[23] Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J. Smola, and How Jing. Recurrent recommender networks. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 17., pages 495-503, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781 1450346757 doi: 10.1 00.1145/301861133 36689
[24] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc., 2015. htttttpss////eeeeeeeeeeeee nnguuiirppp 103-8833dd1bb--pp...... .pff
cc/ hccppaper/2015/// (aaeds/file//aaeee
[25] Liangbei Xu and Mark A. Davenport. Dynamic knowledge embedding and tracing. May 2020.
[26] Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen I Ryu, Krishna V Shenoy, and Maneesh Sahani. Gaussian-rrocess factor analysis for low-dimensional single-trial analysis of neural population activity. In Advances in neural information processing systems, pages 1881-1888, 2009.
[27] Lea Duncker, Gergo Bohner, Julien Boussard, and Maneesh Sahani. Learning interpretable continuous-time models of latent stochastic dynamical systems. In International Conference on Machine Learning, pages 1726 1734. PMLR, 2019.
[28] Yuan Zhao and II Memming Park. Variational latent gaussian process for recovering single-trial dynamics from population spike trains. Neural computation, 29(5); 1293-11113111,, 2017.
12

---

[29] Anqi Wu, Nicholas A Roy, Stephen Keeley, and Jonathan W Pillow. Gaussian process based non- linear latent structure discovery in multivariate spike train data. Advances neural information processing systems, 30:3496, 2017.
[30] Jakob H Macke, Lars Buesing, John P Cunningham, Byron M Yu. Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking in neural populations. In Advances in Neural Information Processing Systems 24: 25th conference on Neural Information Processing Systems (NIPS 2011), I), pages 1350-13555 2012.
[31] Yuanjun Gao, Evan Archer, Liam Paninski, and John P Cunningham. Linear dynamical neural population models through nonlinear embeddings. arXiv preprint 1oXxvv:605.0844 2016.
[32] Jonathan C Kao, Paul Nuyujukian, Stephen I Ryu, Mark M Churchland, John Cunningham, and Krishna V Shenoy. Single-trial dynamics of motor cortex and their applications to brain- machine interfaces. Nature communications, 6(1):1-12, 2015.
[33] Biljana Petreska, Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Maneesh Sahani. Dynamical segmentation of single trials from population neural data. Advances in neural information processing systems, 24:756-764 2011.
[34] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pages 914-922 PMLR, 2017.
[35] Joshua I Glaser, Matthew R Whiteway, John P Cunningham, Liam Paninski, and Scott W Linderman. Recurrent switching dynamical systems models for multiple interacting neural populations. bioRxiv, 2020.
[36] Chethan Pandarinath, Daniel J O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen Ryu, Leigh R Hochberg, al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-8 815, 2018.
[37] MR Keshtkaran and C Pandarinath. Enabling hyperparameter optimization sequential autoencoders for spiking neural data. Advances in Neural Information Processing Systems, 32, 2019.
[38] Qi She and Anqi Wu. Neural dynamics discovery via gaussian process recurrent neural networks In Uncertainty in Artificial Intelligence, pages 454-664. PMLR, 2020.
[39] Mohammad Reza Keshtkaran, Andrew Robert Sedler, Raeed H Chowdhury, Raghav Tandon, Diya Basrai, Sarah L Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandarinath. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. bioRxiv, 2021.
[40] Daniel Hernandez, Antonio Khalil Moretti, Ziqiang Wei, Shreya Saxena, John Cunningham, and Liam Paninski. A novel variational family for hidden nonlinear markov models. arXiv preprint arXiv:1811.02445 2018.
[41] Timothy Doyeon Kim, Thomas Zhihao Luo, Jonathan W. Pillow, and Carlos D. Brody. Inferring latent dynamics underlying neural population activity via neural differential equations. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[42] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. bioRxiv, 2021.
[43] Marcel Nonnenmacher, Srinivas C Turagaa, and Jakob H Macke. Extracting low-dimensional dy- namics from multiple large-scale neural population recordings by learning to predict correlations. arXiv preprint arXiv:1711.01e8 2017.
[44] Jonathan C Kao, Stephen I Ryu, and Krishna Shenoy. Leveraging neural dynamies to extend functional lifetime of mrainn interfaces. Scientific reports, 7(1):1-16, 2017.
13

---

[45] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1att::88800005 2018.
[46] Jerome Lecoq, Michael Oliver, Joshua H Siegle, Natalia Orlova, and Christof Koch. Removing independent noise in systems neuroscience data using deepinterpolation. bioRxiv, 2020.
[47] Cole Hurwitz, Kai Xu, Akash Srivastava, Alessio Buccino, and Matthias Hennig, Scalable spike source localization in extracellular recordings using amortized variational inference. Advances in Neural Information Processing Systems, 32:4724 4736, 2019.
[48] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.0984. 2017.
[49] Mark M Churchland, John P Cunningham, Matthew T Kaufman, Stephen I Ryu, and Krishna V Shenoy. Cortical preparatory activity: representation of movement or first cog in a dynamical machine? Neuron, 68(3):387-400, 2010.
[50] Mark M. Churchland, Matthew T. Kaufman, John P. Cunningham, Justin D. Foster, Paul Nuyujukian, Stephen I. Ryu, and Krishna V. Shenoy. Neural population dynamics during reach- ing dataset. htttps:/ddandiareee httpa///vereeeeeaaaaaee////////r 2021. [Data set]. DANDI archive.
[51] M. Denker, A. Yegenoglu, and S. Grün. Collaborative HPC-enabled workflows the HBP Collaboratory using the Elephant framework. In Neuroinformatics 2018, page P19, 2018. doi: 10. 10.12751/nnc..m2018..000
URL http:://t/abbttttttssscs node org conference/ 000.000000000---1------002222200
[52] Liam Paninski. Maximum likelihood estimation of cascade point-process neural encoding models. Network: Computation Neural Systems, 15(4):243-22 262, 2004.
[53] Mainak Jas, Titipat Achakulvisut, Aid Idrizović, Daniel Acuna, Matthew Antalek, Vinicius Marques, Tommy Odland, Ravi Garg, Mayank Agrawal, Yu Umegaki, Peter Folcy, Hugo Fernandes, Drew Harris, Beibin Li, Olivier Pieters, Scott Otterson, Giovanni De Toni, Chris Rodgers, Eva Dyer, Matti Hamalainen, Konrad Kording, and Pavan Ramkumar. Pyglmnet: Python implementation of elastic-net regularized generalized linear models. Journal of Open Source Software, 5947) 2020. doi: 10.21 105/josss 019559. URL https:////// 21 21105/joss 01959
[54] Xue-Xin Wei, Ding Zhou, Andres Grosmark, Zaki Ajabi, Fraser Sparks, Pengcheng Zhou, Mark Brandon, Attila Losonezy, and Liam Paninski. zero-inflated gamma model for deconvolved calcium imaging traces. arXiv preprint wxXiv:2006.0373, 2020.
[55] David Sussillo, Rafal Jozefowicz, LF Abbott, and Chethan Pandarinath. Lfads-latent factor analysis via dynamical systems. arXiv preprint 1608.066 2016.
[56] Johannes Friedrich, Pengcheng Zhou, and Liam Paninski. Fast online deconvolution of calcium imaging data. PLoS computational biology, 13(3):e 2017.
[57] Andrea Giovannucei, Johannes Friedrich, Pat Gunn, Jérémie Kalfon, Brandon L Brown, Sue Ann Koay, Jiannis Taxidis, Farzaneh Najafi, Jeffrey I. Gauthier, Pengcheng Zhou, Baljit S Khakh, David W Tank, Dmitri Chklovskii, and Eftychios A Pnevmatikakis. CalmAn an open source tool for scalable calcium imaging data analysis. eLife, 8:c38173, January 2019. ISSN 2050-084X. doi: 10.7554/cLife.38 173. URL Utttsss::://doi org/ 755.755///.....333 38173.
Johhanne
[58) Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. Deeplabeut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 21(9):1211-1289, 2018.
[59] Juan A Gallego, Matthew G Perich, Raeed H Chowdhury, Sara A Solla, and Lee E Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience, 23(2):260-27 2700000--200000 2020.
14

---

[60] Eran Klein, Tim Brown, Matthew Sample, Anjali R Truitt, and Sara Goering. Engineering the brain: ethical issues and the introduction of neural devices. Hastings Center Report, 45(6): 26-35, 2015.
[61] Sara Goering, Eran Klein, Laura Specker Sullivan, Anna Wexler, Blaise Aguera Arcas, Guo- qiang Bi, Jose M Carmena, Joseph J Fins, Phoebe Friesen, Jack Gallant, et al. Recommendations for responsible development and application of neurotechnologies. Neuroethics, pages 1-22, 2021.
15


Reviews:
[{"review_id": "oAiUI987r2Y", "reviewer": "Reviewer_ZS9v", "summary": "The authors proposed a training strategy, selective backpropagation through time (SBTT), for the deep structure to learn latent dynamics from irregularly or sparsely sampled time series data. The learned latent dynamics then can be used to infer activity for missing samples based on observations. The experimental results have demonstrated the effectiveness of the SBTT with various rates of dropped samples; the SBTT can recover some of missing information.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " The whole idea is interesting, but the authors should improve some parts of the draft before further consideration.\n1)\tThe author should explain the details of how does the SBTT work with SVAE.\n2)\tWhat is the relationship between the two equations in lines 137 and 128?\n3)\tThe experiments do not compare the effectiveness with other state-of-the-art methods, e.g. transformers, deepset and neural differential equations. Also, it would be better that the author provides recovered samples learned by other state-of-the-art methods to demonstrate the effectiveness of SBTT.\n4)\tThe SBTT might need to be tested with other structures to show its scalability rather than just SVAE.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should improve some parts of the draft mentioned in the above further consideration.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "foOPSN1LW8", "reviewer": "Reviewer_b2vn", "summary": "The authors introduce selective backpropagation through time (SBTT), and use this technique to improve the spatiotemporal resolution of AutoLFADS (a sequential autoencoder). This approach allows accurately inferring a low-dimensional dynamical system when samples are not recorded at all times. They demonstrate how this approach could be useful for Neuropixel recordings, when a large population of neurons is recording with a limited bandwidth (so that not all neurons are recorded at all times). They also demonstrate how this could be useful with Calcium imaging, in order to incorporate the more precise times that neurons are recorded during scanning (rather than assuming all neurons were recorded at once). ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n-This is a novel idea - I’m not aware of anyone who was trying to address these challenges. In fact, I hadn’t really thought about them as being challenges, so I think it could lead to a lot of new work in computational neuroscience.\n\n-The manuscript is very thorough and high quality - there are demonstrations on simulated data and multiple neuroscience applications.\n\n-The manuscript is clearly written, especially given the amount of information the authors are providing.\n\n\nCons:\n\n-I don’t have any major concerns. I think the broad significance of the approach is TBD, as it’s hard to know whether this would be used with Neuropixels as proposed. But I am more confident this approach would be used in the near future for calcium imaging data. I encourage the authors to make this tool very user friendly to increase adoption of their method.\n\n\nMinor comments:\n\nFor Fig. 2b,c:\n-Can you provide a citation for how you calculated pseudo-R2 (there are different ways of defining it). And similarly, can you provide a citation for coefficient of determination (unfortunately, i’ve also seen this defined multiple ways).\n\nIt can be a little challenging to keep the meaning of the acronyms in my head. A couple recommendations (that you can take or leave):\n-Just call A-ZIG-FR “A-ZIG”.\n-Instead of “A”, use “ALFADS” (so ALFADS-ZIG). I realize “A” is shorter, but I would forget that A referred to AutoLFADS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "ZOfL_NiQrA", "reviewer": "Reviewer_SHsa", "summary": "This paper extends AutoLFADS to deal with missing data. The proposed solution has two main elements: (i) the generative loss is simply zeroed-out at those (neuron, time) pairs where data is missing, and (ii) missing data are zero-filled before going into the encoder for amortized inference. I am not entirely convinced that this contribution crosses Neurips' high bar on technical & conceptual innovation. Element (i) above is probably the first thing that would cross one's mind for dealing with missing data in this particular case -- not sure the use of the fancy name \"Selective backpropagation through time\" is justified, where it simply corresponds to ignoring what the generator does at points where no data is available. Moreover, I do not see a clear a priori justification for using zeroes in element (ii). Finally, this hack is so simple that it doesn't strike me as a big modification of the existing AutoLFADS codebase (though I do appreciate the huge coding effort for the applications!). Overall, I therefore find this contribution rather incremental. Nevertheless, the case for tackling the problem of missing data is well made, and the experiments on a number of datasets are both impressive and convincing: the method just works, and I can see that it will be used in practice. Also, the paper very nicely shows that missing data, even when a lot is missing, isn't necessarily a deal-breaker for accurate reconstruction of latent dynamics.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Beyond the review summary, here a few points the authors might want to consider:\n\n- Perhaps Bayesian methods (e.g. GPFA) ought to be given more credit in the introduction / prior art for their natural principled approach to dealing with missing data.\n\n- Unpacking a point made in the summary: it is not clear to me why replacing missing data by zeros in the input to the encoder is \"the\" right thing to do... Do the authors agree that this choice was arbitrary? Could one consider replacing missing data by the mean spike count (or calcium) of the relevant neuron instead? Better yet, shouldn't these values be treated as variational parameters? I can see that they could affect the ELBO.\n\n- Have the authors looked at how GPFA performs with missing data? I am curious to know how the rate at which cross-validated prediction errors deteriorates with increasing fraction of dropped samples compares with AutoLFADS'.\n\n- Section 2 is a greatt overview of related work; I learned a lot there about refs I didn't know about!\n \n----\n\nMinor things:\n\n- The first paragraph of section 2 is rather obscure / premature; e.g. what does \"similar loss function to that considered in this work\" refer to? Having already gone through Figure 1, the reader might reasonably expect that the \"loss function considered in this work\" already involves the masking sketched in Figure 1C, yet the paragraph ends \"all of the above works limit their focus to [...] where the observations are fully sampled\".\n\n- typo: Possion emissions model → Poisson emission model\n\n- \"KL penalties described above\": only one KL penalty is described (?)\n\n\n\n==== Post-rebuttal update =====\n\nMany thanks to the authors for their response, which consolidates my impressions. I had scored the paper at \"marginally above acceptance threshold\", already balancing the lack of technical innovation (which would normally lead to a clear reject in my mind) with the convincing experimental results and interesting conceptual framing of the study. To explain my reasoning very bluntly: does a neuroscientist with a missing data problem really need this paper? Yes and no. Yes, for the encouraging proof of concept that missing data isn't that bad a problem. No, because there is no substantial technical insight required to replace missing data with zeros and ignoring the reconstruction loss at these points. I am therefore reluctant to increase my score, out of fairness for the countless papers that make highly innovative algorithmic contributions yet will be rejected this year in this extremely competitive field -- I hope the authors will understand this stance! In any case, great work to be proud of.\n \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations and societal impact are well addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and I agree that this paper presents a novel approach to extracting fine-timescale dynamics from seemingly slow calcium imaging data by leveraging the 2p laser scan pattern. From a purely technical perspective, the idea is straightforward: model the data at a much higher sampling rate but with a ton of missing data. This leads to some tricky inference problems since amortized variational inference with missing data isn't solved, but the authors find that simply zeroing out missing data before passing it to the encoder works well enough. The thorough analyses, impressive experimental results, and clarity of the text make this paper particularly notable and exciting for the NeurIPS crowd.
Decision: Accept (Poster)