Title: Bypassing spike sorting: Density-based decoding using spike localization from dense multielectrode probes
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural decoding

brain-computer interfaces

spike sorting

variational inference

generative models
Abstract: Neural decoding and its applications to brain computer interfaces (BCI) are essential for understanding the association between neural activity and behavior. A prerequisite for many decoding approaches is spike sorting, the assignment of action potentials (spikes) to individual neurons. Current spike sorting algorithms, however, can be inaccurate and do not properly model uncertainty of spike assignments, therefore discarding information that could potentially improve decoding performance. Recent advances in high-density probes (e.g., Neuropixels) and computational methods now allow for extracting a rich set of spike features from unsorted data; these features can in turn be used to directly decode behavioral correlates. To this end, we propose a spike sorting-free decoding method that directly models the distribution of extracted spike features using a mixture of Gaussians (MoG) encoding the uncertainty of spike assignments, without aiming to solve the spike clustering problem explicitly. We allow the mixing proportion of the MoG to change over time in response to the behavior and develop variational inference methods to fit the resulting model and to perform decoding. We benchmark our method with an extensive suite of recordings from different animals and probe geometries, demonstrating that our proposed decoder can consistently outperform current methods based on thresholding (i.e. multi-unit activity) and spike sorting. Open source code is available at https://github.com/yzhang511/density_decoding.

Bypassing spike sorting: Density-based decoding using spike localization from dense multielectrode probes

Abstract

Neural decoding and its applications to brain computer interfaces (BCI) are essential for understanding the association between neural activity and behavior. A prerequisite for many decoding approaches is spike sorting , the assignment of action potentials (spikes) to individual neurons. Current spike sorting algorithms, however, can be inaccurate and do not properly model uncertainty of spike assignments, therefore discarding information that could potentially improve decoding performance. Recent advances in high-density probes (e.g., Neuropixels) and computational methods now allow for extracting a rich set of spike features from unsorted data; these features can in turn be used to directly decode behavioral correlates. To this end, we propose a spike sorting-free decoding method that directly models the distribution of extracted spike features using a mixture of Gaussians (MoG) encoding the uncertainty of spike assignments, without aiming to solve the spike clustering problem explicitly. We allow the mixing proportion of the MoG to change over time in response to the behavior and develop variational inference methods to fit the resulting model and to perform decoding. We benchmark our method with an extensive suite of recordings from different animals and probe geometries, demonstrating that our proposed decoder can consistently outperform current methods based on thresholding (i.e. multi-unit activity) and spike sorting. Open source code is available at https://github.com/yzhang511/density_decoding.

1 Introduction

Decoding methods for large-scale neural recordings are opening up new ways to understand the neural mechanisms underlying cognition and behavior in diverse species (Urai et al., 2022). The emergence of high-density multi-electrode array (HD-MEA) devices introduced a tremendous increase in the number of extracellular channels that can be recorded simultaneously (Jun et al., 2017; Steinmetz et al., 2021), leading to scalable and high-bandwith brain computer interfaces (BCI) systems (Musk et al., 2019; Paulk et al., 2022).

Traditional neural decoding methods assume that spiking activity has already been correctly spike-sorted. As a result, these methods are not appropriate for situations where sorting cannot be performed with high precision. Despite intensive efforts towards automation, current spike sorting algorithms still require manual supervision to ensure sorting quality (Steinmetz et al., 2018). Even after careful curation, current spike sorters suffer from many sources of errors including erroneous spike assignment (Deng et al., 2015). The dense spatial resolution of HD probes makes some known issues of spike sorting even more evident. With the increased density of the recording channels, the

probability of visibly overlapping spikes (spike collisions) is higher (Buccino et al., 2022). Even for the same HD dataset, different spike sorters have low agreement on the isolated units and can find a significant number of poorly sorted and noisy units (Buccino et al., 2020). Consequently, only single units that pass quality control metrics are included in many neural coding studies (IBL et al., 2022).

Because the spike-sorting problem remains unresolved, alternative approaches that do not rely on sorted single-units for decoding have been proposed. A popular choice is multi-unit threshold crossing that uses spiking activity on each electrode for decoding (Fraser et al., 2009; Trautmann et al., 2019). However, this approach ignores the fact that the signal on each electrode is a combination of signals from different neurons, thus making inefficient use of the data (Todorova et al., 2014). Ventura (2008) proposed a spike-sorting free decoding paradigm that estimates neuronal tuning curves from electrode tuning curves and then infers the behavior of interest using the estimated tuning curves and newly observed electrode spike trains. More recently, Chen et al. (2012); Kloosterman et al. (2014); Deng et al. (2015); Rezaei et al. (2021) developed spike feature decoding methods that use marked point processes to characterize the relationship between the behavior variable and features of unsorted spike waveforms. However, these decoders based on state-space models make explicit assumptions about the underlying system dynamics which reduce their flexibility in capturing complex relationships in the data. Moreover, these methods mainly utilize simple waveform features for decoding such as the maximum amplitude on each electrode and do not take advantage of HD spike features such as the estimated spike location.

To leverage the spatial spread and density of HD probes, Hurwitz et al. (2019); Boussard et al. (2021) developed spike localization methods. These methods estimate the source location of a detected spike; this is a low-dimensional feature that is informative about the firing neuron's identity. We propose a probabilistic model-based decoding method that scales to HD-MEA devices and utilizes these novel localization features in conjunction with additional waveform features. We use a mixture of Gaussians (MoG) model to encode the uncertainty associated with spike assignments in the form of parametric distributions of the spike features. Unlike traditional MoG models with a fixed mixing proportion, our method allows the mixing proportion to depend on the behavior of interest and change over time. This is motivated by the theory that behavioral covariates that modulate neurons' firing rates also contain information about spike identities and that such tuning information should be incorporated into spike sorting and neural decoding in order to obtain unbiased and consistent tuning function estimates (Ventura, 2009). To infer the functional relationship between spike features and behavioral correlates, we employ automatic differentiation variational inference (ADVI) (Kucukelbir et al., 2017) and coordinate ascent variational inference (CA VI) (Blei et al., 2017), which enable us to perform efficient and accurate inference while considering the behavior-modulated MoG model.

We apply our method to a large number of HD recordings and decode various types of behavioral correlates. Experimental results show that our decoder consistently outperforms decoders based on multi-unit threshold crossings and single-units sorted by Kilosort 2.5 (Pachitariu et al., 2023). We further validate the robustness of our method by applying it to recordings with different levels of sorting quality, HD probes with varying geometry, and recordings from multiple animal species. Consistent with previous results, our findings indicate that relying solely on 'good' units, as determined by sorting quality metrics, leads to information loss and suboptimal decoding performance. This observation motivates our transition to a spike sorting-free decoding framework which enables us to extract more information from the spiking activity and improve decoding performance.

2 Method

Consider an electrophysiological recording comprised of K trials, where each trial is divided into T equally spaced time bins. Let { s itk } n tk i =1 , s itk ∈ R D denote a set of spike features, where i indexes the i -th spike, n tk represents the number of spikes collected in the t -th time bin of the k -th trial, and D is the dimension of spike features. For example, the spike feature s itk = ( x itk , z itk , a itk ) ∈ R 3 includes the spike location along the x- and z-axis of the probe, and its maximum peak-to-peak (max ptp) amplitude. Let y k ∈ R T be the observed time-varying behavior in the trial k , e.g., the speed of a rotating wheel controlled by a mouse. When the behavior in the trial k does not vary over time, it can take on either a binary ( y k ∈ { 0 , 1 } ) or scalar ( y k ∈ R ) value, e.g., the mouse responds to a

scalar-valued stimulus by making a binary decision.

The proposed decoding method comprises an encoder and a decoder model. During the training of the model, the encoder captures the relationship between the observed spike feature distribution and the observed behavior. During the testing phase, the decoder utilizes the learned relationship from the encoder to predict the unobserved behavior based on newly observed spike features. In the following section, we present a general formulation of the encoding-decoding paradigm and provide more detailed implementations in the supplementary materials.

time

T

c)

encoder

decoder

decoder

decoded behavior

encoder

behavior

b)

…

…

…

…

spike features

components

C

firing rates

time

T

trials

K

components

C

trials

K

behavior

decoder

weight matrix

estimated

behavior

from

thresholding

train trials

test trials

training time

test time

neural

recordings

spike features

1 cm

amplitude (s.u.)

30

22

14

7

0

spike localization features

100 um

NP1

a)

（wn)x

500

1000

1500

z (um)

2500

..--3000.

3500

x（um)

-40,

400

600

---800

z (um)

1200

1400

1600

0c

C

0c

C

ytk

入ctk

（Tctk）

ytk

入ctk

Tctk

Zitk)

Zitk)

(Sitk)

Sitk

ntk

ntk

Zitk

Sitk

ntk

Zitk

Sitk

ntk

Encoder. The multivariate spike feature distribution is modeled using a mixture of Gaussian (MoG). The encoder generative model is as follows:

where λ ( · ) is a function that describes the firing rate's dependence on behaviors y k , while p ( θ c ) represents a general prior on θ c encompassing the model parameters for the mixture component c . Intuitively, the behavior-dependent λ governs the mixing proportion π of the MoG, which determines the specific component c from which a spike feature s is generated. As λ varies over time in response to y , spikes that are spatially close and share similar waveform features may originate from different MoG components at different time points within a trial. In our implementation, we parameterize λ using a generalized linear model (GLM), but alternative models such as neural networks (NN) can also be used; see the supplementary material for the GLM configuration.

We employ variational inference (VI) to learn the unknown quantities. In the standard MoG setting (Blei et al., 2017), our goal is to infer the spike assignment z which indicates the latent component from which the observation s originates. However, unlike the standard MoG, our spike assignment z is influenced by the firing rates λ of the neurons which are modulated by the behavior y . Consequently, learning the association between λ and y necessitates the estimation of the unknown model parameters θ . Our objective is to simultaneously learn both the latent variables z and model parameters θ based on the observed spike features s and behavior y . To accomplish this, we posit a mean-field Gaussian variational approximation

for the posterior p ( z, θ | s, y ) . Subsequently, we employ the CAVI or ADVI methods to maximize the evidence lower bound (ELBO) and compute updates for z and θ . Analogous to the standard Expectation-Maximization (EM) algorithm for MoG, the proposed CAVI and ADVI procedures consist of an E step and a M step. The E step, which updates z , closely resembles that of the ordinary MoG, while the M step, responsible for updating θ , differs. CA VI utilizes coordinate ascent to find θ that maximizes the ELBO, while ADVI employs stochastic gradient ascent for θ updates. For detailed information on the CAVI and ADVI model specifications, refer to Supplement 1 and 2.

Decoder. The decoder adopts the same generative model in Equation 1-3 as the encoder with two distinctions: 1) y k is unobserved and considered a latent variable we aim to estimate, i.e., y k ∼ p ( y k ) , where p ( y k ) is a general prior, and 2) the model parameters θ are obtained from the encoder and kept constant. In practice, the choice of prior relies on the nature of y k . For instance, if y k is binary, we can sample from a Bernoulli distribution while a Gaussian process prior can capture the temporal correlation between time steps if y k ∈ R T . The posterior p ( z, y | s ) is approximated using a mean-field Gaussian variational approach

We employ standard CAVI or ADVI methods to infer z and decode y .

Robust behavior decoding. In practice, we found that direct decoding of y using the approximated posterior q ( y ) in Equation 5 was not robust, leading to decoding results of inconsistent quality across different datasets. Although the factorization described in Equation 5 may not fully exploit the available information for predicting y , it is useful for learning about the spike assignment z . To enhance decoding robustness, we compute a weight matrix from the MoG outputs as input to the final behavior decoder. The weight matrix, denoted as W , has dimensions K × C × T and entries W kct := ∑ n tk i =1 q ( z ikct ) , which capture the posterior probability of assigning spike i collected at time t of the trial k into the component c . In scenarios such as spike sorting or multi-unit thresholding, spikes are assigned deterministically to one of C sorted single units or thresholded channels. In this case, W has one-hot rows, and each entry W kct represents the number of spikes belonging to trial

k , time t and unit (channel) c . To obtain q ( z ikct ) , we rely on estimating the posterior π ctk , which requires either the observed y tk or the estimated ˆ y tk . During model training, we can substitute the observed y tk into Equation 1 to calculate the posterior π ctk for the train trials. At test time, we use the estimated ˆ y tk obtained from multi-unit thresholding along with the learned encoder parameters θ to calculate the posterior π ctk for the test trials. With the posterior π ctk in hand, we then estimate q ( z ) to compute the weight matrix W for both the train and test trials, which serves as input to the final behavior decoder. The choice of the behavior decoder depends on the user's preference. For instance, we can use logistic regression as the behavior decoder for binary y k , and ridge regression for y k ∈ R T that exhibit temporal variations. Additional information regarding the selection of the behavior decoder can be found in Supplement 5.

3 Experiments

We conducted experiments using both electrophysiological and behavior data obtained from the International Brain Laboratory (IBL) (IBL et al., 2021). The electrophysiological recordings were acquired using Neuropixels (NP) probes that were implanted in mice performing a decision-making task. Each recording comprises multiple trials with several behavioral variables recorded during each trial including the choice, face motion energy, and wheel speed; see Figure 2 (a) for details. Each trial has a duration of 1.5 seconds and is divided into 30 time bins of 50 milliseconds length. The NP probe spans multiple brain regions; an example of the brain parcellation can be seen in Figure 3. To prepare the recordings for decoding, we first applied IBL's standard destriping procedure (Chapuis et al., 2022) to reduce artifacts. Then, we used a subtraction-based spike detection and denoising method described in Boussard et al. (2023). After preprocessing, we computed the spike locations (Boussard et al., 2021) to acquire spike features for decoding and then utilized registration techniques (Windolf et al., 2022) to correct for motion drift in the recorded data. Further details about data preprocessing can be found in Supplement 4. In all experiments, we used spike locations along the width and depth of the NP probe, and maximum peak-to-peak amplitudes of spikes for decoding. We selected this set of spike features based on empirical evidence from our experiments, which showed their good decoding performance. Furthermore, previous studies (Boussard et al., 2023; Hilgen et al., 2017) have also recognized that these features were highly informative about unit identity. Figure 1(a) illustrates the spike localization and waveform features that were utilized for decoding.

We evaluate the performance of our decoding method by comparing it to the following baselines: (1) Spike-thresholded decoders which utilize the spiking activity on each electrode after a voltage-based detection step. (2) Spike-sorted decoders that utilize all single-units found using Kilosort (KS) 2.5 (Pachitariu et al., 2016). (3) Spike-sorted decoders based on 'good' units which consist of KS units that have passed IBL's quality control procedure (IBL et al., 2022). The parameters used for KS were tuned across multiple IBL datasets as described by IBL's spike sorting white paper (Chapuis et al., 2022).

To assess the quality of decoding, we perform 5-fold cross validation (CV) and compute relevant decoding metrics. The coefficient of determination ( R 2 ) is used to evaluate continuous behavior decoding (e.g., motion energy and wheel speed) while accuracy is used for discrete behaviors (e.g., choice). To demonstrate the efficacy of our approach in a wide range of settings, we conduct the following experiments.

Varying levels of spike sorting quality. Our objective is to compare the proposed decoding method to spike-sorted decoders using datasets which have varying levels of spike sorting quality. We apply our method to two datasets with high sorting quality ('good' sorting) and two with low sorting quality ('bad' sorting). The quality of sorting is assessed using IBL's quality metrics (Chapuis et al., 2022). Although motion registration has been performed, we find that the recordings that have 'bad' sortings are more affected by motion drift then the recordings which have 'good' sortings; see supplementary materials.

Different brain regions from 20 datasets. To demonstrate the efficacy of our method across many different datasets, we decode 20 IBL datasets (IBL et al., 2022). In these datasets, mice perform a behavioral task while NP1 probes record activity from multiple brain regions. These brain regions

time (sec)

good vs. bad sorting example sessions

motion energy

b)

d)

choice

(left / right)

stimulus

a)

R2

ours

thresh 'good'

KS

all

KS

accuracy

R2

wheel speed

time (sec)

time (sec)

cumulative absolute drift (um/sec)

missed spikes (%)

R2

motion

energy

wheel speed

'good'

'bad'

motion energy

ours

thresh 'good'

KS

all

KS

wheel speed

ours

thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

'good'

'bad'

choice

ours

thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

'good'

'bad'

c)

motion energy

time (sec)

wheel speed

time (sec)

wheel

speed

observed

all KS

ours

0.7

0.6

0.5

0.4

0.3

0.7

0.6

0.5

0.4

0.3

0.0          2.5

log(contamination (%))

0.7

0.6

0.5

0.4

0.3

25       50       75

20 sessions

R2

R2

R2: 0.491  R2: 0.696

R2: 0.377  R2: 0.474

R2: 0.401  R2: 0.519

R2: 0.150  R2: 0.270

motion

energy

25       30      35

N

2

×

10

15

M

5

1.5 0

1.5

all KS (good") ×

(peq) SX Ie

ours (*bad")

0.6

中

0.6

0.9

白

白

0.5

6白白

8

Q

0.5

空空

0.4

0.8

0.4

0.3

0.7

。

20

10

o

8

0

1.5 0

1.5 0

1.5 0

1.5 0

1.5

Q

10

5

0

1.5 0

1.5 0

1.5 0

1.5 0

1.5

2.5

ox

0.0

0

1.5 0

1.5 0

1.5 0

1.5 0

1.5

8

2

M

?

Fo

1.5 0

1.5 0

1.5 0

1.5 0

1.5

are repeatedly targeted across all the datasets. To explore how behaviors are linked to specific brain regions, we use spikes that are confined to a particular area of the mouse brain for decoding. We collect decoding results from the posterior thalamic nucleus (PO), the lateral posterior nucleus (LP), the dentate gyrus (DG), the cornu ammonis (CA1) and the anterior visual area of the visual cortex (VISa).

Different probe geometry. Our method is capable of decoding electrophysiological data from a variety of HD probes. To demonstrate this, we apply our method on Neuropixels 2.4 (NP2.4) and Neuropixels 1.0 in nonhuman primates (NP1-NHP) datasets. The NP2.4 and NP1-NHP recordings are preprocessed using an identical pipeline as employed for NP1; see Supplement 4 for details. For spike sorting, the NP2.4 and NP1 adopt the same KS parameters as outlined in IBL's spike sorting procedure. Different KS parameters are utilized for NP1-NHP probes which are detailed in Trautmann et al. (2023).

NP2.4: Each NP2.4 probe consists of four shanks and a total of 384 channels (Steinmetz et al., 2021). NP2.4 probes are more dense (i.e., more channels in a given area) than NP1. The mice were trained in accordance with the IBL experiment protocols to perform a visual decision-making task. The behavioral correlates we decode are choice, motion energy, and wheel speed.

NP1-NHP: NP1-NHP is designed for nonhuman primate species, such as macaques. The NP1-NHP probe maintains the same number of channels as NP1 (384), but its overall length is extended, resulting in a sparser configuration compared to NP1 (Trautmann et al., 2023). During the experiment, the macaque underwent training in a sequential multi-target reaching task (Marshall et al., 2022). The behavioral correlate we decode is the monkey's arm force. The probe was implanted in the macaque's motor cortex.

Comparison to a state-of-the-art clusterless decoder. Although the lack of available code for prior methods make comprehensive comparisons difficult, we benchmark our density-based decoder against a state-of-the-art clusterless decoder on datasets from both HD probes and multiple tetrodes. We compare our method to the clusterless point process decoder of Denovellis et al. (2021), which utilizes a marked point process to connect spike features with behaviors. For more details of this comparison, see Section 9 of the supplementary materials.

To decode the binary choice variable, we utilize the CA VI algorithm described in Supplement 2. For continuous behaviors like motion energy, wheel speed, and arm force, we employ the ADVI algorithm outlined in Supplement 1. We specify the maximum number of iterations as a hyperparameter in the CAVI model as it requires analytical updates for the model parameters. Running the CAVI encoder and decoder for fewer than 50 iterations yields satisfactory decoding outcomes. As for the ADVI algorithm, we implement it in PyTorch and update the model parameters using the Adam optimizer with a learning rate of 0.001 and a batch size of 6. The ADVI model is run for 1000 iterations. Open source code is available at https://github.com/yzhang511/density_decoding.

4 Results

Varying levels of spike sorting quality. The performance of our method in comparison to the spike-thresholded and spike-sorted decoders for both the 'good' and 'bad' sorting examples is summarized in Figure 2. For the 'good' sorting examples, our method has the highest decoding performance for motion energy and wheel speed. For choice decoding, our approach is comparable to decoding based on all KS single-units and better than decoders based on multi-unit thresholding and 'good' KS units. For the 'bad' sorting sessions, the gap in decoding performance between our method and other decoders is more pronounced. Example traces are illustrated in Figure 2 (c), which demonstrate that the behavior traces decoded by our method closely match the observed traces compared to decoded traces from the sorted decoders. In Figure 2 (d), we quantify the relationship between sorting quality and decoding quality using data from 20 IBL sessions. For all three quality metrics, the performance of our decoder and the spike-sorted decoder decreases as the quality of the sorting decreases. Despite this decrease in performance, our method consistently has better performance than the spike-sorted decoder even in the presence of significant motion drift as well as a when there is a large fraction of missed spikes or contaminated units.

All

PO

LP

DG

CA1

VISa

thresh

'good'

KS

all

KS

ours

thresh

'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

All

PO

LP

DG

CA1

VISa

R2

accuracy

All

PO

LP

DG

CA1

VISa

wheel speed

choice

motion energy

VISa

CA1

LP

PO

DG

thresh

'good'

KS

all

KS

ours

thresh

'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

R2

thresh

'good'

KS

all

KS

ours

thresh

'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

thresh 'good'

KS

all

KS

ours

a)

b)

All

R2

PO

LP

DG

CA1

VISa

All

R2

1e6

1e6

1e6

1e6

1e6

1e6

PO

LP

DG

CA1

VISa

0.4

0.2

0.4

0.2

0.9

十

中

0.8

0.7

0.6

×

all KS

×

good KS

×

ours

0.85

0.3

0

0

500

0

200

0

200

0

100

0

100

0

200

s#components

# components

s#components#components

#components

0.85

xx

0.3

2

0

0

i

0.0

# spikes

macaque

brain

45 mm

10 mm

NP1

NP1-NHP

R2

0.6

0.4

all

KS

ours

thresh good

KS

motion

energy

observed

all KS

ours

b)

a)

NP2.4

R2

0.4

0.2

all

KS

ours

thresh good

KS

wheel speed

accuracy

0.80

0.75

0.70

all

KS

ours

thresh good

KS

choice

all KS

ours

thresh

R2

arm force

arm force

time (s)

R2: 0.307  R2: 0.658

宁

0.6

口

0.4.

5

0

0.2

0

9.850

9.85

Different brain regions from 20 datasets. The decoding results across various brain regions for 20 IBL sessions are summarized in Figure 3. Overall, our approach consistently achieves higher R 2 values compared to other competing methods in decoding both motion energy and wheel speed across the five recorded brain regions. Notably, decoders based on 'good' KS units exhibit poor performance across all recorded brain regions when compared to decoders based on all KS units. This observation highlights the importance of utilizing all available information for decoding behaviors rather than solely relying on 'good' units based on sorting quality metrics. The scatter plots in Figure 3 (b) indicate a general trend where decoding quality tends to increase when more components (i.e., KS units for the spike-sorted decoders and MoG components for our method) are available for decoding. However, our method outperforms spike-sorted decoders even with a limited number of components.

Different probe geometry. The decoding results for the NP2.4 and NP1-NHP geometries are illustrated in Figure 4. For NP2.4, our approach significantly outperforms other competing methods when decoding motion energy and wheel speed, while again, the approaches are more comparable when decoding the discrete choice variable (our method performs slightly worse than the spikesorted decoder). For NP1-NHP, Figure 4 demonstrates that our method achieves better decoding performance ( R 2 ≈ 0 . 6 ) compared to the spike-thresholded ( R 2 ≈ 0 . 2 ) and spike-sorted baselines ( R 2 ≈ 0 . 3 ). 'Good' KS units are not available in this scenario (the IBL quality criteria were not applied to this primate dataset) and are therefore not included in the results.

Comparison to a state-of-the-art clusterless decoder. We compare our method to a state-of-theart clusterless point process decoder (Denovellis et al., 2021) in Table 1. Our method has higher decoding performance than the point process decoder on both HD and simulated tetrode datasets for all behavior variables. This performance improvement is likely due to the increased flexibility of our decoder compared to state-space models that make stronger assumptions about the dynamics of the decoded signals.

Computation time. In Figure 5, we provide a computation time comparison relative to real-time. Our decoding step operates at a sub-real-time pace (0.3 times real-time). The total time after preprocessing for our method is close to real-time.

(real-time)

1.5

1.2

0.9

工

0.6

time

0.3

0.0

preprocessing spike subtraction/denoising

localization

registration

density-decoding

totalafterpreprocess

sDenovellisetal'21

5 Discussion

In this work, we introduce a probabilistic model-based neural decoding method that relates spike feature distributions to behavioral correlates for more accurate behavior decoding. Our method is designed for high-density recording devices such as Neuropixels probes, utilizing novel HD spike features (i.e., spike locations) and maximum peak-to-peak amplitudes of the spikes. We further develop an efficient variational approach to perform inference in this model. We benchmark our method across a comprehensive set of HD recordings with varying levels of sorting quality, different probe geometries, and distinct brain regions. We demonstrate that our decoding method can consistently outperform spike-thresholded decoders and spike-sorted decoders across a wide variety of experimental contexts. This motivates a shift towards a spike-feature based decoding paradigm that avoids the need for spike sorting while also achieving comparable or superior decoding performance to approaches relying on well-isolated single-units.

While our method shows promising results, it is essential to explore avenues for improvement. Two potential improvements to our method include utilizing deep learning-based models to capture complex functional associations between firing rates and behaviors and also introducing dependencies among the mixture components to account for correlated neuronal firing patterns. An interesting extension of this work would be to apply this spike-feature based paradigm to unsupervised learning of neural dynamics which would enable us to estimate MoG 'firing rates' conditioned on time-varying latent variables. By addressing these challenges and expanding the scope of our method, we can advance our understanding of neural activity and its relationship with behavior in both the supervised and unsupervised settings.

6 Broader Impact

Neuroscience research aims to uncover how much behavior information can be decoded from neural signals in specific brain regions. To do this fairly across regions without bias from spike sorting quality, we need decoding methods that do not rely on spike sorting. However, we should recognize the current decoders' limitations to avoid drawing erroneous conclusions from the decoding results. Our generative model, which improves predictive performance by conditioning on external variables, is also relevant to the machine learning community.

References

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association , 112(518):859-877, 2017.

Julien Boussard, Erdem Varol, Hyun Dong Lee, Nishchal Dethe, and Liam Paninski. Threedimensional spike localization and improved motion correction for neuropixels recordings. Advances in Neural Information Processing Systems , 34:22095-22105, 2021.

Julien Boussard, Charlie Windolf, Cole Hurwitz, Hyun Dong Lee, Han Yu, Olivier Winter, and Liam Paninski. Dartsort: A modular drift tracking spike sorter for high-density multi-electrode probes. bioRxiv , page 553023, 2023.

Alessio P Buccino, Cole L Hurwitz, Samuel Garcia, Jeremy Magland, Joshua H Siegle, Roger Hurwitz, and Matthias H Hennig. Spikeinterface, a unified framework for spike sorting. Elife , 9: e61834, 2020.

Alessio Paolo Buccino, Samuel Garcia, and Pierre Yger. Spike sorting: new trends and challenges of the era of high-density probes. Progress in Biomedical Engineering , 2022.

Mayo Faulkner Chapuis, Kenneth D Harris, Julia M Huntenburg, Cole Hurwitz, Hyun Dong Lee, Liam Paninski, Cyrille Rossant, Noam Roth, Nicholas A Steinmetz, Charlie Windolf, et al. Spike sorting pipeline for the international brain laboratory. channels , 10:6, 2022.

Zhe Chen, Fabian Kloosterman, Stuart Layton, and Matthew A Wilson. Transductive neural decoding for unsorted neuronal spikes of rat hippocampus. pages 1310-1313, 2012.

Xinyi Deng, Daniel F Liu, Kenneth Kay, Loren M Frank, and Uri T Eden. Clusterless decoding of position from multiunit activity using a marked point process filter. Neural computation , 27(7): 1438-1460, 2015.

Eric L Denovellis, Anna K Gillespie, Michael E Coulter, Marielena Sosa, Jason E Chung, Uri T Eden, and Loren M Frank. Hippocampal replay of experience at real-world speeds. Elife , 10:e64505, 2021.

George W Fraser, Steven M Chase, Andrew Whitford, and Andrew B Schwartz. Control of a brain-computer interface without spike sorting. Journal of neural engineering , 6(5):055004, 2009.

Joshua I Glaser, Ari S Benjamin, Raeed H Chowdhury, Matthew G Perich, Lee E Miller, and Konrad P Kording. Machine learning for neural decoding. Eneuro , 7(4), 2020.

Gerrit Hilgen, Martino Sorbaro, Sahar Pirmoradian, Jens-Oliver Muthmann, Ibolya Edit Kepiro, Simona Ullo, Cesar Juarez Ramirez, Albert Puente Encinas, Alessandro Maccione, Luca Berdondini, et al. Unsupervised spike sorting for large-scale, high-density multielectrode arrays. Cell reports , 18(10):2521-2532, 2017.

Daniel N Hill, Samar B Mehta, and David Kleinfeld. Quality metrics to accompany spike sorting of extracellular signals. Journal of Neuroscience , 31(24):8699-8705, 2011.

Cole Hurwitz, Kai Xu, Akash Srivastava, Alessio Buccino, and Matthias Hennig. Scalable spike source localization in extracellular recordings using amortized variational inference. Advances in Neural Information Processing Systems , 32, 2019.

International Brain Laboratory IBL, Kush Banga, Julius Benson, Niccolò Bonacchi, Sebastian A Bruijns, Rob Campbell, Gaëlle A Chapuis, Anne K Churchland, M Felicia Davatolhagh, Hyun Dong Lee, et al. Reproducibility of in-vivo electrophysiological measurements in mice. bioRxiv , pages 2022-05, 2022.

The International Brain IBL, Valeria Aguillon-Rodriguez, Dora Angelaki, Hannah Bayer, Niccolo Bonacchi, Matteo Carandini, Fanny Cazettes, Gaelle Chapuis, Anne K Churchland, Yang Dan, et al. Standardized and reproducible measurement of decision-making in mice. Elife , 10, 2021.

James J Jun, Nicholas A Steinmetz, Joshua H Siegle, Daniel J Denman, Marius Bauza, Brian Barbarits, Albert K Lee, Costas A Anastassiou, Alexandru Andrei, Ça˘ gatay Aydın, et al. Fully integrated silicon probes for high-density recording of neural activity. Nature , 551(7679):232-236, 2017.

Fabian Kloosterman, Stuart P Layton, Zhe Chen, and Matthew A Wilson. Bayesian decoding using unsorted spikes in the rat hippocampus. Journal of neurophysiology , 2014.

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic differentiation variational inference. Journal of machine learning research , 2017.

Jesse A Livezey and Joshua I Glaser. Deep learning approaches for neural decoding across architectures and recording modalities. Briefings in bioinformatics , 22(2):1577-1591, 2021.

Jeremy F Magland and Alex H Barnett. Unimodal clustering using isotonic regression: Iso-split. arXiv preprint arXiv:1508.04841 , 2015.

Najja J Marshall, Joshua I Glaser, Eric M Trautmann, Elom A Amematsro, Sean M Perkins, Michael N Shadlen, LF Abbott, John P Cunningham, and Mark M Churchland. Flexible neural control of motor units. Nature neuroscience , 25(11):1492-1504, 2022.

Elon Musk et al. An integrated brain-machine interface platform with thousands of channels. Journal of medical Internet research , 21(10):e16194, 2019.

Marius Pachitariu, Nicholas Steinmetz, Shabnam Kadir, Matteo Carandini, et al. Kilosort: realtime spike-sorting for extracellular electrophysiology with hundreds of channels. BioRxiv , page 061481, 2016.

Marius Pachitariu, Shashwat Sridhar, and Carsen Stringer. Solving the spike sorting problem with kilosort. bioRxiv , pages 2023-01, 2023.

Angelique C Paulk, Yoav Kfir, Arjun R Khanna, Martina L Mustroph, Eric M Trautmann, Dan J Soper, Sergey D Stavisky, Marleen Welkenhuysen, Barundeb Dutta, Krishna V Shenoy, et al. Large-scale neural recordings with single neuron resolution using neuropixels probes in human cortex. Nature Neuroscience , 25(2):252-263, 2022.

Mohammad Reza Rezaei, Kensuke Arai, Loren M Frank, Uri T Eden, and Ali Yousefi. Real-time point process filter for multidimensional decoding problems using mixture models. Journal of neuroscience methods , 348:109006, 2021.

Nicholas A Steinmetz, Christof Koch, Kenneth D Harris, and Matteo Carandini. Challenges and opportunities for large-scale electrophysiology with neuropixels probes. Current opinion in neurobiology , 50:92-100, 2018.

Nicholas A Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, et al. Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science , 372(6539): eabf4588, 2021.

Sonia Todorova, Patrick Sadtler, Aaron Batista, Steven Chase, and Valérie Ventura. To sort or not to sort: the impact of spike-sorting on neural decoding performance. Journal of neural engineering , 11(5):056005, 2014.

Eric M Trautmann, Sergey D Stavisky, Subhaneil Lahiri, Katherine C Ames, Matthew T Kaufman, Daniel J O'Shea, Saurabh Vyas, Xulu Sun, Stephen I Ryu, Surya Ganguli, et al. Accurate estimation of neural population dynamics without spike sorting. Neuron , 103(2):292-308, 2019.

Eric M Trautmann, Janis K Hesse, Gabriel M Stine, Ruobing Xia, Shude Zhu, Daniel J O'Shea, Bill Karsh, Jennifer Colonell, Frank F Lanfranchi, Saurabh Vyas, et al. Large-scale brain-wide neural recording in nonhuman primates. bioRxiv , pages 2023-02, 2023.

Anne E Urai, Brent Doiron, Andrew M Leifer, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience , 25(1):11-19, 2022.

Valérie Ventura. Spike train decoding without spike sorting. Neural computation , 20(4):923-963, 2008.

Valérie Ventura. Traditional waveform based spike sorting yields biased rate code estimates. Proceedings of the National Academy of Sciences , 106(17):6921-6926, 2009.

Charlie Windolf, Angelique C Paulk, Yoav Kfir, Eric Trautmann, Samuel Garcia, Domokos Meszéna, William Muñoz, Richard Hardstone, Irene Caprara, Mohsen Jamali, et al. Robust online multiband drift estimation in electrophysiology data. bioRxiv , pages 2022-12, 2022.

Supplementary Material

1 Decoding using automatic differentiation variational inference (ADVI)

In the method section of our paper, we describe the general encoding-decoding paradigm. In this section of the supplementary material, we delve into a specific case that focuses on decoding continuous behaviors, y k ∈ R T , that exhibit temporal variations. We introduce the use of ADVI, which allows us to model the relationship between firing rates of MoG components and behavior correlates using a generalized linear model (GLM).

1.1 ADVI-based encoder

Building upon the general model specification outlined in Equations 1-3, we can describe the generative model of the encoder as follows:

where b and β are the unknown model parameters corresponding to θ in Equation 1, and are sampled from a standard normal prior distribution. The latent spike assignment z depends on the mixing proportion π of the MoG, which is influenced by the behavior y through the firing rate λ . Intuitively, we can interpret λ ctk as the 'firing rate' of component c at time t in trial k , while b c and β c describe the 'baseline firing rate' and 'behavior-modulated firing rate' of component c , respectively.

To learn the latent variables z , b and β , we posit a mean-field Gaussian variational approximation

for the posterior distribution p ( z, b, β | s, y ) . Obtaining exact updates for b and β is challenging due to the normalization term for π in Equation 7. Therefore, we employ ADVI to maximize the

ELBO and utilize stochastic gradient ascent to update b and β . ADVI requires that the model be differentiable with respect to the parameters, and with a normal prior, the latent variables b and β reside in the real coordinate space and cause no issues with differentiability. The variational approximations for b and β are

A drawback of the parameterization in Equations 6-8 is that the spike assignment variables z are discrete and not compatible with ADVI. An alternative, equivalent parameterization that addresses these problems is to marginalize over z . The marginalized model is

Under this parameterization, the ELBO for the encoder is

1.2 ADVI-based decoder

The decoder adopts the same generative model as the encoder described in Equations 6-8 with two exceptions: 1) The latent variable y tk is assumed to have a standard normal prior, i.e., y tk ∼ N ( y tk ; 0 , 1) , assuming independence at each time step t . Alternatively, a Gaussian process prior can be chosen to capture temporal correlations between time steps. 2) The parameters b and β are learned from the ADVI-based encoder and kept fixed in the decoder.

The mean-field Gaussian variational approximation for the posterior distribution p ( z, y | s ) is

where

To enable the use of ADVI, we can marginalize out the discrete latent variable z , thereby transforming the MoG model into a differentiable form. Under the marginalized MoG parametrization in Equations 12-14, the ELBO of the decoder is

2 Decoding using coordinate ascent variational inference (CA VI)

We present a specific scenario for decoding binary variables, y k ∈ { 0 , 1 } , where we derive exact updates for the variational variables using CA VI.

2.1 CAVI-based encoder

Extending the general model described in Equations 1-3, the generative model of the encoder can be defined as follows:

where z and λ are the latent variables that we aim to learn. The behavior-dependent firing rates of each component c at time t vary based on the binary variable y , such that the components switch between two behavioral states characterized by firing rates λ ct 1 and λ ct 0 .

The log-likelihood of the encoder can be written as

where Λ t 1 = ∑ c ′ λ c ′ t 1 and Λ t 0 = ∑ c ′ λ c ′ t 0 . To approximate the posterior p ( z | s, y ) , we employ the mean-field variational approximation q ( z ) = ∏ c,t q ( z ct ) . The ELBO of the encoder is

The exact update for q ( z ) is obtained by maximizing the ELBO with respect to q ( z ) , which leads to the following update equation:

̸

where q -ct means ∏ c ′ = c ∏ t ′ = t q ( z c ′ t ′ ) . Then,

̸

denotes the unnormalized posterior probability of assigning spike i collected at time t in trial k to component c , while E [ z ictk ] = ρ ictk / ∑ c ′ ρ ic ′ tk := r ictk represents the normalized posterior probability.

After fixing q ( z ) , the term in the ELBO which depends on λ , µ and Σ can be expressed as

We derive the update for λ by setting the gradients ∇ λ ct 1 L and ∇ λ ct 0 L to 0:

̸

̸

̸

̸

̸

̸

Consider the gradient with respect to the η c parameter,

̸

The closed-form updates for µ c and Σ c are

2.2 CAVI-based decoder

The CAVI-based decoder employs the same generative model as the CAVI-based encoder, with the exception that the behavior-dependent firing rates λ ct 1 and λ ct 0 are learned by the encoder and kept fixed, and the behavior y is treated as an unknown latent variable. We sample y from a prior distribution, y k ∼ Bernoulli ( ϕ ) , where ϕ is a variational parameter that represents the probability that y k = 1 . The log-likelihood can be expressed as follows

We use the factorization q ( z, y ) = q ( z ) q ( y ) = ∏ c,t q ( z ct ) q ( y ) to approximate the posterior distribution p ( z, y | s ) . The ELBO of the decoder can be defined as

The exact updates for q ( z ) and q ( y ) that guarantee an increase in the ELBO are

where

and E [ z ictk ] = ρ ictk / ∑ c ′ ρ ic ′ tk := r ictk are the unnormalized and normalized posterior probabilities of assigning spike i collected at time t in trial k to component c , respectively. The unnormalized posterior probabilities of y k = 1 and y k = 0 are

and E [ y k ] = y ∗ k 1 / ( y ∗ k 1 + y ∗ k 0 ) := ν k represents the normalized posterior probability of y k = 1 .

After fixing q ( z ) and q ( y ) , the term in the ELBO which depends on ϕ , µ and Σ can be written as

Considering the gradient of the ELBO with respect to the ϕ parameter, we obtain its update:

The updates for η c are computed in a similar manner as described in Equations 36-39.

3 MoG initialization

We employ the following procedure to intialize the MoG model used in both the ADVI-based and CAVI-based algorithms: 1) According to Figure 1 (a), the spike feature distribution is highly multimodal. To determine the appropriate number of modes, we utilize isosplit (Magland and Barnett, 2015) to cluster the spike features. This step helps in splitting the set of spike features into distinct clusters. 2) For each identified cluster, we compute the mean and variance of the spike features belonging to that cluster, which serve as the parameters for the corresponding Gaussian component. This automatic selection of the number of MoG components and the initialization of means and covariance matrices facilitate the initialization of the ADVI-based and CA VI-based algorithms.

4 Data preprocessing

We provide a brief overview of our data preprocessing pipeline, which involves the following steps.

Destriping. During the data collection process, we encounter line noise due to voltage leakage on the probe. This translates into large 'stripes' of noise spanning the whole probe. To mitigate the impact of these noise artifacts, we apply a destriping procedure (Chapuis et al., 2022).

Subtraction-based spike detection and denoising. We employ the iterative subtraction-based procedure for spike detection and collision-correction described in Boussard et al. (2023).

Spike localization. We employ the method of Boussard et al. (2021) to estimate the location of each denoised spike.

Drift registration. Probe motion (or drift) in the electrophysiology data poses a challenge for downstream analyses. Decentralized registration (Windolf et al., 2022) is applied to track and correct for motion drift in the high-density probe recordings.

'bad' sorting example

b)

'good' sorting example

a)

3500

3000

(wn)

2500

2000

depth

1500

1000

500

0

50

100

150

200

250

50

100

150

200

250

time (s)

time (s)

5 Behavior decoder

To decode binary behaviors, such as the mouse's left or right choices, we utilize L 2 -penalized logistic regression. For decoding dynamic behaviors, such as wheel speed, we employ a sliding-window algorithm to aggregate the entries of the weight matrix, W kct , over time. Within the time window [ t -δ, t + δ ] , where δ is the window size, we stack 2 δ weight matrix entries, { W kct } C c =1 , for time point t in trial k . This aggregated weight matrix is then used as input for ridge regression to predict the behavior y tk at time t . The window size δ and the regularization strength are model hyper-parameters, set through cross-validation to achieve the optimal decoding performance.

6 Model interpretation

In this section, we provide visualizations to gain insights into the effectiveness of our proposed decoder. We quantify the posterior entropy of each spike assignment in Figure 7 (a). Spike assignments with higher entropy correspond to a spread of posterior probabilities among multiple MoG components. In contrast, traditional spike sorting or thresholding methods result in deterministic spike assignments, leading to lower entropy and empirically reduced decoding performance. In Figure 7 (b), we compare the trial-averaged weight matrices ( W ) used for decoding between spike-sorted, spike-thresholded, and our proposed decoders.

(spike-thresholded)

a)

b)

(spike-sorted)

KS unit

time (s)

component

(ours)

time (s)

time (s)

channel

Wkct

entropy:1.0 bits0.3 bits

1

0.8 bits

0.4 bits

0

0

0

2.00

10

5

10

1.75

0

20-

10

5060

2030

4050

8090

1.50

component component component component

30

20

1.25

entropy (bit)

3

40

50

30

1.00

0.75

2

60

30

40

0.50

1

0

80

70

X

35

40

50

0.25

0.00

0

10

20

30

40

50

0

0.5

1

1.5

0

0.5

1

1.5

0

0.5

1

1.5

amplitude (s.u.)

7 Decoding across brain regions

In addition to the previously mentioned five brain regions (PO, LP, DG, CA1, VISa) depicted in Figure 3, we expanded our analysis to include two additional brain regions situated in the cerebellum: the arbor vitae (ARB) and the ansiform cruciform lobule (ANCR). We specifically include the cerebellum in our analysis due to the frequent occurrence of spike sorting issues in this area. In Figure 8 and 9, we present a comparison between our decoder and the spike-sorted decoder that utilizes all KS units across all the brain regions studied. According to Figure 8, our method consistently outperforms the decoder that relies on all KS units across all brain regions and the majority of IBL sessions. Furthermore, Figure 9 specifically demonstrates that our method consistently achieves superior decoding performance in the recorded regions of the cerebellum, where spike sorting quality issues are commonly encountered. This highlights the robustness and reliability of our method, particularly in challenging recording conditions.

All

PO

LP

DG

CA1

VISa

motionenergy

wheelspeed

choice

0.81

0.6 /

our accuracy

1.0

2

0.6

z 1no

0.4

uno

0.4

0.2

0.8

0.2-

0.0

0.6

0.2

0.4

0.6

0.8

0.0

0.20.4

0.6

0.6

0.8

1.0

KS R2

KS R2

KS accuracy

zJno

0.6

z Ino

0.4

0.8

0.2

0.2

0.6

0.0

0.0

0.4

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.4

0.60.81.0

KS R2

KS R2

KS accuracy

0.6

0.5

our accuracy

0.9

2

0.4

z uno

0.3

0.7

uno

0.2

#+#-

0.1

0.5

0.0

<+--+

-0.1

0.3

0.0 0.2 0.4 0.6

-0.1 0.1

0.3

0.5

0.3

0.5

0.7

0.9

KS R2

KS R2

KS accuracy

0.9

our accuracy

z uno

0.6

F

O

0.4

0.7

0.4

0.2

0.2

0.5

0.0

0.0

0.3

0.00.2 0.40.6

0.0

0.2

0.4

0.3

0.5

0.7

0.9

KS R2

KS R2

KS accuracy

our accuracy

0.9

1

z Ino

0.4

0.6

O

0.4

0.7

0.2

0.2

0.5

×

0.0

0.0

0.3

T

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.3

0.5

0.7

0.9

KS R2

KS R2

KS accuracy

1.1

our accuracy

0.6

z uno

0.4

0.9

0.4

×

0.2

0.2

0.7

0.0

0.0

0.5

0.00.2 0.4 0.6

0.0

0.2

0.4

0.5

0.7

0.9

1.1

>

KS R2

KS R2

KS accuracy

All

ARB

ANCR

motion energy

wheel speed

choice

0.7

accuracy

2

0.6

0.5

×

×

uno

our

0.9

0.4

0.3

#

our

0.2

0.1

0.7

0.2

0.4

0.6

0.1

0.3

0.5

0.7

0.7

0.9

1.1

KS R2

KS R2

KS accuracy

motion energy

wheel speed

choice

0.6

accuracy

0.6

0.9

2

R

0.4

Xx

十

召

0.4

0.7

our

our

0.2

0.2

×

our

0.5

0.0

0.0

0.3

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.6

0.3 0.5 0.7 0.9

KS R2

KS R2

KS accuracy

motion energy

wheel speed

choice

0.5 1

1.1

0.6

×

accurac

2

2

×

uno

0.4

our

0.3

0.9

our

0.2

0.1

0.7

7

0.2

0.4

0.6

0.1

0.3

0.5

0.7

0.9

1.1

KS R2

KS R2

KS accuracy

8 Ablation study

We conduct an ablation study to investigate the importance of various components in our decoding paradigm. Specifically, we examine how the integration of dynamic (behavior-dependent) mixing proportions in the MoG and higher-dimensional spike features influence decoding performance. Additionally, we analyze how the inclusion criteria for spike waveforms can affect decoding outcomes.

Effects of dynamic MoG mixing proportion. Table 3 presents a comparison between the ordinary MoG with a fixed mixing proportion (referred to as 'fixed π ') and our proposed model with a dynamic mixing proportion (referred to as 'dynamic π '). The results indicate that this approach leads to improved decoding performance compared to using a fixed mixing proportion in the MoG model.

Motion energy (

R

2

)

Wheel speed (

R

2

)

Choice (accuracy)

Fixed

π

Dynamic

π

Fixed

π

Dynamic

π

Fixed

π

Dynamic

π

All

0.664 (

±

0.034)

0.742

(

±

0.028)

0.470 (

±

0.062)

0.564

(

±

0.045)

0.948 (

±

0.038)

0.957

(

±

0.036)

PO

0.365 (

±

0.031)

0.488

(

±

0.046)

0.520 (

±

0.019)

0.670

(

±

0.015)

0.844 (

±

0.016)

0.861

(

±

0.035)

LP

0.145 (

±

0.015)

0.464

(

±

0.054)

0.114 (

±

0.032)

0.342

(

±

0.027)

0.917 (

±

0.026)

0.931

(

±

0.022)

DG

0.280 (

±

0.033)

0.492

(

±

0.042)

0.221 (

±

0.040)

0.381

(

±

0.035)

0.669 (

±

0.084)

0.722

(

±

0.050)

CA1

0.407 (

±

0.021)

0.538

(

±

0.038)

0.308 (

±

0.051)

0.428

(

±

0.030)

0.621 (

±

0.062)

0.626

(

±

0.064)

VISa

0.488 (

±

0.046)

0.490

(

±

0.047)

0.318 (

±

0.056)

0.364

(

±

0.074)

0.857 (

±

0.065)

0.874

(

±

0.066)

Effects of higher-dimensional spike features. In Table 4, we provide a comparison of decoding performance using two different sets of spike features. The first set includes spike location along the width and depth dimensions of the probe (denoted as x and z ) as well as the maximum peak-topeak amplitude of the spike (denoted as a ). The second set includes the first and second principal components (PCs) of the spike waveforms (denoted as u 1 and u 2 ) in addition to x, z and a . The spike features are visually represented using scatter plots in Figure 10. We report the mean and standard deviation of the decoding accuracy ( R 2 ) obtained from a 5-fold CV on a single session. Table 4 provides insights regarding the inclusion of additional waveform PC features for decoding. The findings suggest that the incorporation of these additional PC features does not contribute to significant improvements in decoding performance.

Motion energy (

R

2

)

Wheel speed (

R

2

)

Choice (accuracy)

(

x, z, a

)

(

x, z, a, u

1

, u

2

)

(

x, z, a

)

(

x, z, a, u

1

, u

2

)

(

x, z, a

)

(

x, z, a, u

1

, u

2

)

All

0.531

(

±

0.026)

0.529 (

±

0.026)

0.484

(

±

0.042)

0.478 (

±

0.045)

0.917 (

±

0.019)

0.917 (

±

0.019)

PO

0.462

(

±

0.038)

0.457 (

±

0.039)

0.469

(

±

0.061)

0.464 (

±

0.051)

0.853 (

±

0.019)

0.853 (

±

0.025)

LP

0.490

(

±

0.026)

0.489 (

±

0.029)

0.479

(

±

0.035)

0.473 (

±

0.013)

0.864

(

±

0.022)

0.849 (

±

0.036)

DG

0.335

(

±

0.028)

0.321 (

±

0.037)

0.273

(

±

0.030)

0.264 (

±

0.025)

0.675 (

±

0.038)

0.679

(

±

0.049)

CA1

0.449

(

±

0.027)

0.440 (

±

0.046)

0.329

(

±

0.044)

0.328 (

±

0.040)

0.755 (

±

0.045)

0.758

(

±

0.053)

VISa

0.270

(

±

0.021)

0.237 (

±

0.017)

0.225

(

±

0.024)

0.206 (

±

0.013)

0.725 (

±

0.048)

0.732

(

±

0.054)

4000

3500

5

3000

6

2500

maxptp amplitude (s.u.)

(um)

2000

N

1500

1000

500

25

0

T

-100

0

100

0

20

40

0

50

100-25

0

25

x(um)

maxptp amplitude (s.u.)

U1

U2

Effects of inclusion criteria for spike waveforms. To investigate whether the density-based decoder is performing better by using additional spikes that a spike sorter would miss, we conducted an experiment using different inclusion criteria for spike waveforms. We fitted our model using only spikes detected by Kilosort 2.5, and compared its performance to decoders using spike-sorted outputs and our subtraction-based spike detection on choice and motion energy decoding. The results are summarized in Table 5. As shown in the table, our decoder can achieve comparable or better decoding performance than the spike-sorted decoder when modeling the same spikes. This suggests that the gain in decoding performance can be attributed to the density-based approach, instead of the spike inclusion criteria.

9 Comparison to a state-of-the-art clusterless decoder

In this section, we outline the specific experimental setup for evaluating our density-based decoder in comparison with the clusterless point process decoder (Denovellis et al., 2021) on both multiple tetrodes and high-density (HD) probes data.

Application to tetrodes. We utilized the code provided in the GitHub repository 1 of the clusterless point process decoder (Denovellis et al., 2021) to generate simulated neural and behavioral data. This synthetic dataset was designed to mimic recordings from 5 tetrodes, with each tetrode containing 4 channels. Spike amplitudes from each channel of these multiple tetrodes were selected as the spike features for both decoding methods. The objective of the decoding was to estimate the animal's position from these simulated spike features. As the original simulated position was too straightforward to decode, we intentionally distorted it by blending it with real position data sourced from the GitHub repository 2 . Moreover, we introduced random Gaussian noise to the simulated position for added complexity.

Application to HD probes. We evaluated both decoders on decoding wheel speed and motion energy from spike features extracted from NP1 probes across three IBL datasets. To preprocess the data, we followed the pipeline outlined above, extracting spike localization features and maximum peak-to-peak amplitudes as common features for both decoders.

We only focus on continuous behaviors for decoding since the clusterless point process decoder is designed exclusively for continuous behaviors. Notably, due to its continuous time nature, the clusterless point process decoder directly decoded behaviors without time binning. In contrast, our density-based model required time binning of both behavioral and spike feature data into equal-sized time intervals. Consequently, we used the time-binned behaviors for decoding with the density-based approach. We employed 5-fold cross-validation to assess the decoding performance of both decoders. We used a random walk as the state transition for the clusterless point process decoder, and used the estimated variance from the animal's behavior to set the variance of the random walk. The clusterless point process decoder uses a grid-based approximation of the inferred behavior, discretizing the behavior space into place bins; see Section 'Decoding' in Denovellis et al. (2021). We determine place bin size based on the square root of the observed behavior variance. Denovellis et al. (2021) use kernel density estimation (KDE) to estimate the distributions of both the behavior variable and the spike features used for decoding. We set the KDE bandwidth that determines the amount of smoothing done for spike features to be 1.0, and the bandwidth for behavior to be the square root of the observed behavior variance; see Section 'Encoding - clusterless' in Denovellis et al. (2021).

10 Simulation for model validation

Weconducted simulations to illustrate the principles of our method. The simulation aimed to show that our encoding model can learn the relationship between spike features and behaviors. We performed two tasks, decoding a binary variable, y k , simulated from a Bernoulli distribution, and decoding a continuous variable, y k , simulated from a Gaussian process. To mimic the data-generating process, we selected Gaussian components with 'templates' extracted from a real dataset. The encoding model parameters, b and β , were also taken from learned parameters in the same dataset. Given b , β and y k , we simulated the 'firing rates' λ for each Gaussian component in the mixture, as described in the Method section of our paper. Next, we generated spike features based on these simulated 'firing rates,' and applied the encoding model to infer the behavior-dependent λ . Figure 11 displays the learned λ for each component c , time t , and trial k . The learned 'firing rates' closely resembled the simulated ones, indicating the model's ability to recover the primary associations between spike features and behaviors. With such associations, the decoding model can decode behaviors.

simulated

simulated

learned

a)

b)

time (second)

0                 1.0

time (second)

0                1.0

time (second)

0                1.0

component

component

time (second)

0                 1.0

learned

入ctk

0

0

0

0

15

10

20

20

20

10

5

5

40

40

40

40

11 Relationship to spike sorting

We conducted experiments to investigate the biological interpretation of our MoG units and the correspondence between single cells identified by KS and our MoG units. The agreement matrix between 'hard' KS spike assignments and 'soft' MoG assignments is shown in Figure 12. We calculated the conditional probability of spikes belonging to each MoG component, given that these spikes belong to the corresponding KS unit. Notably, KS units with large amplitudes are less likely to be split into multiple Gaussian components. This shows a reasonable correspondence between the Gaussian components and the spike-sorted units.

Prob.

Amplitude

(s.u.)

0-

10-

0.75

20-

0.50

MoGunits

0.25

30

0.00

40

50

30

-09

20

0

20

40

60

08

100

120

10

KSunits

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors demonstrate a novel approach to decoding behavioral variables from neural activity recording using dense silicon probes. Most approaches utilize a sort-then-decode line of attack – first channels are spike sorted, then decoders are trained on unambiguously identified single-units. Here, the authors present a probabilistic model that bypasses explicitly assigning waveforms to units; rather, the authors model spike features using a mixture of Gaussians, which is then leveraged for decoding. The authors go on to show that their method generally outperforms the current state-of-the-art in a variety of experiments spanning recordings in the mouse and in the monkey.\n\nSTRENGTHS:\nStrengths:\n\n1. The authors present an elegant solution a highly-challenging problem in neuroscience – decoding behavior in the presence of noisy and highly-overlapping signals. \n1. The results are generally compelling. Clearly, the author's algorithm outperforms Kilosort and multi-unit thresholding in a variety of decoding contexts.\n1. The core idea and implementation is fairly straightforward. Given this, I expect this contribution to be generally useful to the community.\n\nWEAKNESSES:\nWeaknesses:\n\nMajor:\n\n1. One potential issue with \"bypassing\" spike-sorting is noise and motion artifact. Spike-sorting, if carefully curated, can remove unwanted sources of contamination in extracellular electrophysiology. Here, I am somewhat concerned that this method could pick up and amplify contaminating signals that may contain behaviorally relevant information. For instance, Figure 3a shows that \"good\" KS units in CA1 and DG contain barely any information about motion energy; however, multi-unit thresholding, all KS, and the author's method find a surprising amount of information. Do the authors think this is reasonable? I am a bit worried since similar gains are not seen for choice decoding, which may not be as easy to decode from motion artifact. This could absolutely be real information uncovered by the model, but this issue is worth addressing head on. Do the authors think that their method could be picking up on any noise or motion artifact in the original recordings?\n\nMinor:\n\n1. Rhetorical point. Line 47 the authors mention prior attempts to \"bypass\" spike sorting as working with \"limited\" features such as maximum amplitudes and principal components of the spike waveform. Unless I missed something this language seems overly critical since the authors are only using peak-to-peak amplitudes and spike localization features (x, z). \n1. Related to the last point, did the authors consider waveform principal components or aspects of the waveforms other than amplitude? Spike width, e.g., is known to contain information about cell type.\n1. Line 97 typo\n\nQUESTIONS:\n1. Did the authors check to see if their technique picks up on noise or motion artifact in these recordings?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a way to decode behavior from neural recordings, bypassing an explicitly spike sorting step. They model individual spikes as coming form a mixture of gaussian distribution, and the assignment probability to different mixtures is used instead of a 'hard' assignment of each spike to a cell. Robust performance of this approach is shown across a large dataset of Neuropixels recordings.\n\nSTRENGTHS:\nWhile spike sorting is thought to incorporate the biological priors (ie., the fact that spikes come from cells, which carry information in a neural circuit), and promote robust decoding, deviating from the prior by developing a \"soft\" spike sorting approach seems to have helped in this paper. This is surprising and very impactful.\n\nWell written paper.\n\nWEAKNESSES:\nAll the results use very high-density neuropixel recordings. However, it is not clear if the method is a general replacement to spike sorting across different arrays, etc. \n\nThe encoding model depends only on behavior. However, there is a large amount of behavior-independent variability (ex. attention states) that are not incorporated in the encoding model.\n\nQUESTIONS:\n1. Could the paper describe the method as \"soft spike-sorting\"? This would help place the method in the right context. \n\n2. Spike sorting is necessary for biological interpretation of neural recordings. Does soft-spike sorting lend such interpretation? \n\n3. If one doesn't do spike sorting, one could use local field potentials, or other bandpass signals to supplement threshold crossings. One should compare with all those features to claim the superiority of the method. \n\n4. How crucial are the preprocessing steps? Were they developed on the same dataset to optimize spike sorting? If so then the results might be confounded. \n\n5. What are the computational costs of this method compared to spike sorting? \n\n6. Using the KS spike assignments, how 'pure' are the gaussians in the MOG? Are single cells split into different gaussians , or do single gaussians correspond to multiple cells? This could shed light on how important is it to deviate from \"hard\" spike sorting. \n\n7. While spike sorting gives stable decoding across time (inspite of template changes over time), does the current method present with similar stability? Can you learn a decoder from the beginning of a recording and apply it towards the end?\n\nLIMITATIONS:\nLimitations have been adequately addressed.\n\nQuestion for Area chair: The paper identified the source of the data as the data from International Brain Lab. It is not clear if the data is open-source. If not, then it clearly violates the double-blind policy of NeurIPS. Please check.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to decode animal behavior with a spike-sorting-free method that is well-suited for high-density recordings, by modeling the distribution of extracted spike features using a mixture of Gaussians (MoG) on uncertainty of spike assignments, and decoding using a generalized linear model (GLM). The authors benchmarked results on Neuropixel recordings from different brain regions, different animals, as well as different probe geometries, and showed that performance outperformed decoders based on multi-unit threshold crossings and single-units sorted by Kilosort.\n\nSTRENGTHS:\n1. The paper is clearly presented. The writings are easy to follow, and most of methods and results are described clearly. \n2. Benchmarking results from multiple brain regions, animals, and different versions of Neuropixel probes were conducted, with a significant improvement on decoding accuracy compared to baseline methods. \n3. Given that high-density recordings are getting more widely-used in the neuroscience field, this paper brings impacts with a better decoding tool for studies conducted with this kind of high-density probes.\n\nWEAKNESSES:\n1. While I recognize the paper providing extensive analysis results, the novelty of proposed density-based decoding on unsorted spike waveforms is limited from my perspective. First, the proposed algorithm on using MoG and GLM for decoding has been widely used on neural ensembles, or other types of neural signals like LFP or ECoG. For example, the [paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3972894/) also leveraged MoG for unsorted spike decoding, despite that it did leverage Bayesian decoding framework. This [paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9585446) proposed Gaussian mixture of model (GMM)-assisted PLS (GMMPLS) for decoding in BMI. Second, there are also other methods proposed for decoding behavior from cluster-less neural data. For example this [paper](https://www.biorxiv.org/content/10.1101/760470v1.full.pdf) proposed cluster-less HMM model for decoding. Another [paper](https://www.biorxiv.org/content/biorxiv/early/2021/08/28/2021.08.26.457795.full.pdf) proposed Gaussian-process multi-class regression decoding on neural population data. The concept of modeling spike distributions with MoG has been applied, and GLM decoding also has been commonly used for neural ensemble. I recognized that some of these papers are not specifically targeting unsorted spikes, and acknowledge this paper bring good impacts for the neuroscience community, but I am not sure if NeurIPS is the proper target given novelty of its proposed algorithm. To meet the bar for NeurIPS, I would like to see more originality algorithm improvements.\n2. Unclear whether baseline decoders are fair baselines. First, Kilsort depends on hyperparameters that need to tune properly for each dataset. From some of example decoded curves, I'm not sure if those baseline decoders were calibrated properly to data. In practice, we rarely worked with decoders with poor accuracy. This leads me to suspect the baseline decoders were not trained/tuned properly for comparison. Second, there are more SOTA decoders that have been used, including previous clusterless decoders on unsorted spiking data (see [paper](https://journals.physiology.org/doi/full/10.1152/jn.01046.2012)). Without a benchmark comparison to these decoders, it's hard to be convinced that the proposed algorithm is much superior than those other decoders. \n3. Unclear if the proposed method only worked better for high-density recordings from Neuropixel or not. Despite that the authors provided two different probe versions for different geometries, it's unclear whether this brings similar benefits for other types of neural recordings (e.g., multiple tetrode arrays, etc). Especially if the paper claims that this method would be better than previous clusterless decoders tested on tetrode recordings, it's important for the NeurIPS audience to know if the proposed method works more generally on decoding unsorted spikes, or somehow better tuned for the distribution from Neuropixel recordings.\n\nQUESTIONS:\n1. I'm not following the authors' arguments on why previous Bayesian decoding methods on unsorted spikes do not apply for HD probes. Specifically, this argument \"the aforementioned approaches are not suitable for HD probes as they rely on sampling-based Bayesian inference and are specifically designed for tetrodes\". It's true that sampling-based Bayesian inference makes certain assumptions on point process, but I don't quite see why those are specific for tetrodes. The original Bayesian inference paper used waveform features such as the maximum amplitudes and principal components for decoding, but that do not mean waveform features cannot be represented in other formats. Can authors elaborate this statement?\n2. What are the baseline decoders used with sorted neurons after sorted by Kilsort?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper develops a decoding method directly on ‘spike features’, without going through spike sorting. The authors model spike assignment uncertainty using a mixture of Gaussians model, and then perform variational inference to model the relationship between the spike features and behavior.

STRENGTHS:
The premise and models in this paper are sound, if not particularly novel. The inference methods are not particularly novel either. However, it is clearly shown that the models developed here are empirically quite powerful. The evaluations performed are very extensive, showing very clearly that their methods outperform standard spike sorting and decoding methods on large datasets in multiple species.

WEAKNESSES:
It is unclear why exactly previous clusterless methods fail on this kind of data. The authors may need to show this concretely to be fair to previous approaches that do not use spike sorting.

Simulation data may help to get intuition on the methodology, and to show validation of the methods.

QUESTIONS:
It may be a good idea to apply previously developed clusterless methods on this kind of data to show how / why they fail.

Simulated data will be helpful to the reader to better understand the limitations of the modeling approach.

LIMITATIONS:
The limitations are adequately discussed. Potential negative societal impact is adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12