SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Liu
d1543@cam.ac.uk
Máté Lengyel
Department of Engineering Department of Cognitive Science
University of Cambridge Central European University
m.lengyel@eng.cam.ac.uk
Neural spiking activity is generally variable, non-stationary, and exhibits complex dependencies on covariates, such as sensory input or behavior. These dependencies have been proposed to be signatures of specific computations, and so characterizing them with quantitative rigor is critical for understanding neural computations. Approaches based on point processes provide a principled statistical framework for modeling neural spiking activity. However, currently, they only allow the instantaneous mean, but not the instantaneous variability, of responses to depend on covariates. To resolve this limitation, we propose a scalable Bayesian approach generalizing modulated renewal processes using sparse variational Gaussian processes. We leverage pathwise conditioning for computing nonparametric priors over conditional interspike interval distributions and rely on automatic relevance determination to detect lagging interspike interval dependencies beyond renewal order. After systematically validating our method on synthetic data, we apply it to two foundational datasets of animal navigation: head direction cells in freely moving mice and hippocampal place cells in rats running along a linear track. Our model exhibits competitive or better predictive power compared to state-of-the-art baselines, and outperforms them in terms of capturing interspike interval statistics. These results confirm the importance of modeling covariate-dependent spiking variability, and further analyses of our fitted models reveal rich patterns of variability modulation beyond the temporal resolution of flexible count-based approaches.
Analyses of spike time [49, 76, 86] and count [12, 26, 57] statistics have revealed neural responses in vivo to be structured but generally variable or noisy [24, 44, 70, 78]. To model this stochastic aspect of neural spike trains, probabilistic approaches based on temporal point processes have been widely applied. This in turn has been a major driver of point process theory development [42] for capturing spiking variability structure with statistical models [74]. The study of neural computation underlying naturalistic behavior in particular involves non-stationary spike trains, which presents a significant challenge as apparent spiking variability is a result of both irreducible "intrinsic" neural stochasticity as well as dependencies on behavioral covariates that can themselves vary on multiple time scales.
Different approaches have been proposed for handling non-stationary spike trains, starting with the classical log Cox Gaussian process [14, 55] to allow variations in the local intensity or firing rate while modeling independent spikes. Dependencies on previous spikes can be captured to first order with renewal processes, and these models have been extended to non-stationary cases through modulation of the hazard function with some time-dependent function [43, 79] or through rescaling interspike intervals with a covariate-dependent rate function [2, 6]. Another approach based on Hawkes processes and spike-history filters [45, 83, 91] introduces conditional point processes that go
beyond the first order Markov assumption. Approaches based on recurrent networks [51, 90] and neural ODEs [10, 41] can in theory capture arbitrarily long dependencies on past spikes and input covariates, but provide more limited descriptive interpretability.
However, not only the rate but also the variability of spiking encodes task-relevant information [34, 60], and bears signatures of the underlying computations [11]. Importantly, this variability has stimulus- and state-dependent structure [12, 21, 48, 67]. In statistical modeling language, this corresponds to heteroscedastic or input-dependent observation noise. Such structure reflects computations performed in the underlying neural circuit, and thus characterizing it from data in a flexible and robust manner is critical for advancing theories of neural computation. The classical approaches reviewed above do not attempt to characterize such covariate-dependent changes in variability. Flexible count models have been introduced to more faithfully capture variability at the count level [28], and recent work has extended this to the general case of input-dependent variability [48]. Count approaches however are limited in the resolution of the analysis set by the time bin size. In addition, the resulting count statistics strongly depend on the chosen time bin size [48, 73].
While firing rates are routinely modeled as input-dependent, extending point process models with input-dependent variability has not been widely explored in the literature. Rate-rescaled and modulated renewal processes rely on fixed base renewal densities. Allowing the shape parameters of the renewal density to vary with covariates in the corresponding hazard function is one potential approach, but this still relies on a commitment to a particular parametric family of renewal densities. Spike-history filters in conditional point processes are conventionally fixed and thus do not directly model input-dependent spiking variability, though dependence on observed and unobserved covariates [91] and switching filters based on discrete states [23] have been considered. Recent work has moved away from parametric filters to nonparametric Gaussian processes [18], which can be extended to flexibly model dynamic filters as functions of external covariates with a spatio-temporal Gaussian process. However, any modulation of the filter will no longer permit fast convolutions, and such models will be computationally expensive as the filter needs to be recomputed every time step. The direct nonparametric estimation of conditional intensity functions based on maximum likelihood has been explored [13, 82], but scalable Bayesian approaches have remained absent.
Contribution To enable flexible modeling as well as modulation of instantaneous point process statistics for analyzing neural spike train variability, we introduce the Bayesian nonparametric nonrenewal process (NPNR). NPNR builds on sparse variational Gaussian processes and defines a nonparametric prior over conditional interspike interval distributions, generalizing modulated renewal processes with nonparametric renewal densities and spike-history dependencies beyond renewal order. In particular, our point process can flexibly model modulations of not only spiking intensity but also variability. We validate our model using parametric inhomogeneous renewal processes, recovering conditional interspike interval distributions and identifying renewal order in spike-history dependence. On neural data from mouse thalamus and rat hippocampus, our method has competitive predictive power while being superior in capturing interspike interval statistics from the non-stationary data. In particular, our method provides instantaneous measures of spike train variability that are modulated by covariates, and shows rich variability patterns in both datasets consistent with previous studies at coarser timescales. We provide a JAX [4] implementation of our method as well as established baseline models within a scalable general variational inference scheme.
We start with a brief overview of the theoretical foundations and related point process models, as well as their combination with Gaussian processes to introduce non-stationarity.
2.1 Temporal point processes
Statistical modeling of events that occur stochastically in time is handled by the general framework of temporal point processes [59, 72]. Denoting the number of events that occurred until time  $t$  by  $N(t)$ , a temporal point process model is completely characterized by its conditional intensity function (CIF)
$$
\lambda (t | \mathcal {H} _ {t}) = \lim  _ {\delta t \rightarrow 0} \frac {\mathbb {E} [ N (t + \delta t) - N (t) | \mathcal {H} _ {t} ]}{\delta t} \tag {1}
$$
$^{1}$ Code available at https://github.com/davindicode/nonparametric-nonrenewal-process
where  $\lambda(t)\delta t$  is the probability to emit a spike event in the infinitesimal interval  $[t, t + \delta t)$  conditioned on  $\mathcal{H}_t$ , the spiking history before  $t$ . We can write the point process likelihood for a single neuron spike train consisting of an ordered sequence of  $S$  spike events at times  $t_i$  as [5]
$$
p \left(t _ {1}, \dots , t _ {S} \mid \lambda (\cdot)\right) = \left[ \prod_ {i = 1} ^ {S} \lambda \left(t _ {i} \mid \mathcal {H} _ {t _ {i}}\right) \right] e ^ {- \int_ {0} ^ {T} \lambda \left(t ^ {\prime} \mid \mathcal {H} _ {t ^ {\prime}}\right) \mathrm {d} t ^ {\prime}} \tag {2}
$$
In neuroscience applications, one often wants to describe modulation of the point process statistics with some time-varying covariates  $\pmb{x}(t)$ , such as animal head direction or body position, which leads to a generalized CIF  $\lambda(t|\mathcal{H}_t, \pmb{x}_{\leq t})$ . Several classes of models have been proposed that are defined by particular restrictions on the functional form of  $\lambda(t|\mathcal{H}_t, \pmb{x}_{\leq t})$ .
2.1.1 Inhomogeneous renewal processes
Renewal assumption The statistical model in Eq. 2 describes dependencies between all spikes. One common simplification is the renewal assumption: interspike intervals (ISIs)  $\Delta^{(i)} = t_{i + 1} - t_i$  are drawn i.i.d. from an interval distribution called the renewal density  $g(\Delta ;\theta)$  with parameters  $\theta$ . This induces a Markov structure  $p(t_i|t_{i - 1},t_{i - 2},\ldots) = p(t_i|t_{i - 1})$  in the spike train likelihood
$$
p \left(t _ {1}, \dots , t _ {S}; \theta\right) = p \left(t _ {1}\right) p \left(t _ {2} \mid t _ {1}\right) \dots p \left(t _ {S} \mid t _ {S - 1}\right) = p \left(t _ {1}\right) \prod_ {i = 1} ^ {S - 1} g \left(u _ {i}; \theta\right) \tag {3}
$$
Common renewal densities used for neural data are the exponential (equivalent to a Poisson process), gamma, and inverse Gaussian distributions [2, 6].
Hazard function modulation Non-stationary point processes need to model changes in statistics with time, and combined with the renewal assumption one obtains inhomogeneous renewal processes. A classical approach that dates back to Cox [14] is to modulate the hazard function (Appendix A)
$$
\lambda (t \mid t _ {i}) = h (\tau) \cdot \rho (t) \tag {4}
$$
with time since last spike  $\tau = t - t_i$  and the modulation factor  $\rho(t)$ . In our context, this can be replaced by some function of covariates  $\rho(\boldsymbol{x}_t)$  [43]. A multiplicative interaction between  $\rho$  and  $h$  as above is typically considered, though this framework allows general parametric forms [71, 79].
Rate-rescaling Another approach that has been widely applied in the neuroscience community is rate-rescaling [2, 6], closely related to time-rescaling (see Appendix B.3). Here, modulation is achieved with a rate function  $r(\pmb{x}_t) \geq 0$  that transforms time  $t$  into rescaled time  $\hat{t}$
$$
\tilde {t} (t) = \int^ {t} r \left(\boldsymbol {x} _ {t ^ {\prime}}\right) \mathrm {d} t ^ {\prime} \tag {5}
$$
This maps all spike times  $t_i$  to rescaled times  $\tilde{t}_i$ , and will be one-to-one as long as  $r(t) > 0$ . By modeling the rescaled ISIs  $\tilde{\Delta}^{(i)} = \tilde{t}_{i + 1} - \tilde{t}_i$  as drawn from a stationary renewal density  $g(\cdot)$ , we obtain an inhomogeneous renewal process from a homogeneous one. The CIF becomes dependent on the covariate path since last spike  $\mathcal{P}_k = \{\pmb{x}(u)|u\in (t_i,t])\}$ , see Appendix B.3.
2.1.2 Conditional point processes
Conditional Poisson processes The renewal assumption ignores correlations between ISIs, which generally are observed in both the peripheral and central nervous system [1, 25] and can be computationally relevant for signal detection and encoding [8, 9, 22, 69]. Going beyond Markovian dependencies, a tractable approach, similar to Hawkes processes [51], is to introduce a causal linear filter  $h(t)$  that is convolved with spikes and added to the log CIF, giving conditional Poisson processes
$$
\log \lambda (t | \mathcal {H} _ {t}, \boldsymbol {x} _ {t}) = h * y (t) + f (\boldsymbol {x} _ {t}), \quad \text {w i t h} y (t) = \sum_ {i} \delta (t - t _ {i}) \tag {6}
$$
where  $*$  denotes temporal convolution. These models are closely linked to mechanistic integrate-and-fire models [52, 85] and have a long history in the neuroscience literature [18, 31, 35, 47, 62, 66, 83], appearing as generalized linear models (GLMs) and spike response models (SRMs).
Conditional renewal processes An even more expressive model can be obtained by replacing the Poisson spiking process with a rate-rescaled renewal process. This results in a conditional renewal process, where the rate function has history dependence  $r(t|\mathcal{H}_t,\boldsymbol{x}_t)$  as in Eq. 6.
2.2 Gaussian process modulated point processes
Gaussian processes (GPs) represent a data-efficient alternative to neural networks, which have been widely used to model the CIF [59, 72, 90]. When combining GPs with point process likelihoods, the resulting generative model leads to doubly stochastic processes for event data. Placing a Gaussian process prior [87] over the log intensity function leads to the classic log Cox Gaussian processes [14, 55], and in the same spirit one can modulate renewal hazard functions [79] or perform rate-rescaling [16] with GPs. Such constructions form the basis of many widely used Bayesian neural encoding models for spike trains, both for modeling single neuron responses [15, 16, 68] as well as population activity [19, 93]. Combining the flexibility offered by renewal and conditional point processes with GP rate or modulation functions within a variational framework has been impeded by the fact that the original papers were built on a GLM framework with parametric covariate mappings [6, 7, 18, 65]. To provide a fair comparison of NPNR to these baselines, we implement a general scalable variational inference framework for the construction and application of such models (see Appendix B for details on the baseline models).
We now introduce the nonparametric non-renewal (NPNR) process and present the approximate Bayesian inference scheme used for model fitting, noting connections to related works in the literature. Our NPNR model provides a nonparametric generalization of modulated renewal processes beyond renewal order, and adds suitable inductive biases for neural spike train data. It implicitly defines a flexible prior over conditional ISI distributions that can be computed using pathwise conditioning, which enables one to analyze spiking variability modulation with minimal parametric constraints. Furthermore, the Bayesian framework provides an elegant data-driven approach to inferring the lagging ISI order of the spike-history dependence.
3.1 Generative model
Conditional intensity surface priors To obtain flexible modulated point process models, we directly model the CIF, or more precisely its logarithm, of the form
$$
\lambda (t | \mathcal {H} _ {t}, \boldsymbol {x} _ {\leq t}) = \lambda (t | t _ {i}, t _ {i - 1}, \dots , \boldsymbol {x} _ {t}) \tag {7}
$$
where  $t_i$  is the most recent spike at current time  $t$ . First considering the renewal case, we note the spatio-temporal structure in the log CIF using time since last spike  $\tau = t - t_i$  is
$$
\log \lambda (t | \mathcal {H} _ {t}, \boldsymbol {x} _ {t}) = \log \lambda (t | t _ {i}, \boldsymbol {x} _ {t}) = f (\tau , \boldsymbol {x} _ {t}) \tag {8}
$$
which suggests placing a spatio-temporal GP prior on the log CIF to describe a log intensity surface
$$
f (\tau , \boldsymbol {x} _ {t}) \sim \mathcal {G P} \left(m (\tau , \boldsymbol {x}), k _ {t} \left(\tau , \tau^ {\prime}\right) \cdot k _ {x} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)\right) \tag {9}
$$
This generalizes the parametric forms of modulation considered in previous approaches [43, 79], in particular allowing modulation of the effective instantaneous renewal density by covariates  $\boldsymbol{x}_t$ . We can introduce lagging ISIs covariates  $\Delta_k(t)$  with lag  $k$  as depicted in Fig. 1A to extend the model to a non-renewal process
$$
\log \lambda (t | t _ {i}, t _ {i - 1}, t _ {i - 2}, \dots) = f (\tau , \Delta_ {1} (t), \Delta_ {2} (t), \dots) \tag {10}
$$
and for a maximum ISI lag  $K$  we denote the lagging ISIs  $\Delta_t = [\Delta_1(t),\dots,\Delta_K(t)]$  to obtain
$$
\log \lambda (t | \mathcal {H} _ {t}, \boldsymbol {x} _ {t}) = f (\tau , \boldsymbol {\Delta} _ {t}, \boldsymbol {x} _ {t}) \tag {11}
$$
Inductive biases for neural data For neural spiking data, there are biological properties to consider for building a more realistic prior. Firstly, neurons have refractory periods immediately following a spike, though in practice neural recordings may not respect this due to contamination in spike sorting [38]. Another potentially useful inductive bias is that changes in the spiking intensity of
neurons fluctuate mostly at shorter ISI timescales [32], whereas at longer delays they tend to be temporally smoother. The latter suggests non-stationary GP kernels to be more suitable for modeling the spike-history dependencies [18]. However, non-stationary kernels do not allow straight-forward use of random Fourier features for evaluating GP posterior function samples at many locations with pathwise conditioning [88, 89]. This in particular is needed to compute the conditional ISI distributions  $g(\tau | \dots)$  in Eq. 15, see Appendix B.5 for details. To achieve the desired non-stationarity for modeling Eq. 11 while maintaining the ability to draw samples using pathwise conditioning, we apply time warping on  $\tau$  from  $[0, \infty) \rightarrow [0, 1]$  with some warping timescale  $\tau_w$
$$
\tilde {\tau} = 1 - e ^ {- \tau / \tau_ {w}} \quad \Longleftrightarrow \quad \tau = - \tau_ {w} \log (1 - \tilde {\tau}) \tag {12}
$$
and place a stationary Gaussian process prior over the warped temporal dimension  $f(\tilde{\tau}, \dots) \sim \mathcal{GP}$  with a temporal kernel  $k(\tilde{\tau}, \tilde{\tau}') = k(\tilde{\tau} - \tilde{\tau}')$ . This transformation is monotonic (see Fig. 1B), and hence we can easily compute the transformation on the CIF
$$
\lambda (t | \dots) = \lambda (\tilde {t} | \dots) \left| \frac {\mathrm {d} \tilde {\tau}}{\mathrm {d} \tau} \right| = e ^ {f (\tilde {\tau}, \dots)} \frac {e ^ {- \tau (\tilde {\tau}) / \tau_ {w}}}{\tau_ {w}} \tag {13}
$$
Similarly, we apply time warping to the  $\Delta_{k}$  dimensions on which we also place stationary kernels  $k(\tilde{\Delta},\tilde{\Delta}^{\prime}) = k(\tilde{\Delta} -\tilde{\Delta}^{\prime})$ . We note that unlike spike-history filters in conditional point processes (Eq. 6) which do not change with inputs, the resulting coupling to past activity in Eq. 11 is dependent on covariates  $\pmb{x}_t$ . This allows one to capture spiking variability modulation via the conditional ISI distribution perspective discussed below. The refractory nature of real neurons can be addressed by the mean function
$$
m (\tilde {\tau}, \boldsymbol {x}) = m (\tilde {\tau}) = a _ {m} \cdot e ^ {- \tilde {\tau} / \tau_ {m}} + b _ {m} \tag {14}
$$
with parameters  $a_{m},\tau_{m},b_{m}$  , where refractory periods can be modeled with large negative  $a_{m}$
Conditional ISI distributions Instead of looking at the CIF, we can view the model as a prior over conditional ISI distributions as depicted in Fig. 1C using the relation (see Appendix A)
$$
g \left(\tau \mid \boldsymbol {\Delta} _ {t}, \boldsymbol {x} _ {\left(t _ {i}, t \right]}\right) \propto \lambda \left(t \mid \mathcal {H} _ {t}, \boldsymbol {x} _ {t}\right) \cdot e ^ {- \int_ {t _ {i}} ^ {t} \lambda \left(t ^ {\prime} \mid \mathcal {H} _ {t ^ {\prime}}, \boldsymbol {x} _ {t ^ {\prime}}\right) \mathrm {d} t ^ {\prime}} \tag {15}
$$
where one drops the dependence on  $\mathcal{H}_t$  in  $g$  for the modulated renewal case. If one fixes the lagging ISIs and picks a constant covariate path  $g(\tau |\Delta_{*},\boldsymbol{x}_{*})$ , this can be interpreted as an instantaneous ISI distribution of a neuron at the conditioned inputs  $\Delta_{*}$  and  $\boldsymbol{x}_{*}$ . Moments of the conditional ISI distribution are computed using Gauss-Legendre quadratures in warped time (Appendix B.5)
$$
\mathbb {E} _ {g (\tau)} \left[ \tau^ {m} \right] = \int_ {0} ^ {\infty} g (\tau) \tau^ {m} \mathrm {d} \tau = \int_ {0} ^ {1} \left| \frac {\mathrm {d} \tau}{\mathrm {d} \tilde {\tau}} \right| g \left(\tau (\tilde {\tau})\right) \tau (\tilde {\tau}) ^ {m} \mathrm {d} \tilde {\tau} \tag {16}
$$
and this can be used to compute tuning curves of spike train statistics, such as the mean ISI  $\mathbb{E}[\tau]$  and coefficient of variation  $\mathrm{CV} = \sqrt{\mathrm{Var}[\tau]} / \mathbb{E}[\tau]$  [58], as a function of  $\pmb{x}_{*}$ . This approach generalizes the homogeneous case considered in the literature, and in particular allows one to compute instantaneous measures of non-stationary spike train variability that are otherwise non-trivial to estimate [58, 75].
3.2 Inference
Temporal discretization The generative model is formulated as a continuous time model. In practice, neural and behavioural data are typically recorded with finite temporal resolution at small regular intervals  $\Delta t$ . The cumulative intensity integral has to be approximated by a sum, though note that directly modeling the cumulative hazard function [59] elegantly avoids this for purely temporal point processes. Spike times are now discretized as a binary vector  $\mathbf{y} = [y_{1},\dots,y_{T}]$  where  $y_{t} = 1$  if  $t$  has a spike event, zero otherwise. Overall, this discretizes the point process likelihood Eq. 2 as
$$
p \left(t _ {1}, \dots , t _ {S} \mid \lambda (t)\right) \quad \rightarrow \quad p (\boldsymbol {y} \mid \boldsymbol {\lambda}) = \prod_ {t = 1} ^ {T} \lambda_ {t} ^ {y _ {t}} e ^ {- \sum_ {t = 1} ^ {T} \lambda_ {t} \Delta t} \tag {17}
$$
where we have  $T$  time steps in total. Note that the discretization scheme implies we do not have observations at  $\tau = 0$ , since the time step immediately after an observed spike has  $\tau = \Delta t$ .
B
Figure 1: Schematic of our proposed model. (A) Time since last spike  $\tau$  and lagging ISIs  $\Delta$  for an observed spike train (top row) alongside covariates  $x$ . (B) Illustration of the time warping procedure. We fix the warping parameter  $\tau_w$  to the empirical mean ISI, which leads to more uniform distributions  $\tilde{\tau} \in [0,1]$  suitable for a stationary GP kernel. (C) Prior samples from the generative model for two values of  $a_m$  characterized by the lack and presence of a refractory period. The transformation Eq. 15 links the log CIF (top rows) with conditional ISI distributions (bottom rows).
Variational lower bound We use stochastic variational inference [39] with batches obtained from consecutive temporal segments and sparse variational GPs [37], giving the loss objective
$$
\mathcal {L} = \sum_ {n = 1} ^ {N} \left(\mathbb {E} _ {q \left(\boldsymbol {f} _ {n} \mid \boldsymbol {u} _ {n}\right)} \left[ - y _ {n t} f _ {n t} + \Delta t \sum_ {t = 1} ^ {T} e ^ {f _ {n t}} \right] + D _ {\mathrm {K L}} \left(q \left(\boldsymbol {u} _ {n}\right) \mid p \left(\boldsymbol {u} _ {n}\right)\right)\right) \tag {18}
$$