LeMoNADe: Learned Motif and Neuronal Assembly Detection in calcium imaging videos

Neuronal assemblies , loosely defined as subsets of neurons with reoccurring spatiotemporally coordinated activation patterns , or `` motifs '' , are thought to be building blocks of neural representations and information processing . We here propose LeMoNADe , a new exploratory data analysis method that facilitates hunting for motifs in calcium imaging videos , the dominant microscopic functional imaging modality in neurophysiology . Our nonparametric method extracts motifs directly from videos , bypassing the difficult intermediate step of spike extraction . Our technique augments variational autoencoders with a discrete stochastic node , and we show in detail how a differentiable reparametrization and relaxation can be used . An evaluation on simulated data , with available ground truth , reveals excellent quantitative performance . In real video data acquired from brain slices , with no ground truth available , LeMoNADe uncovers nontrivial candidate motifs that can help generate hypotheses for more focused biological investigations . 1 INTRODUCTION . Seventy years after being postulated by Hebb ( 1949 ) , the existence and importance of reoccurring spatio-temporally coordinated neuronal activation patterns ( motifs ) , also known as neuronal assemblies , is still fiercely debated ( Marr et al. , 1991 ; Singer , 1993 ; Nicolelis et al. , 1997 ; Ikegaya et al. , 2004 ; Cossart & Sansonetti , 2004 ; Buzsáki , 2004 ; Mokeichev et al. , 2007 ; Pastalkova et al. , 2008 ; Stevenson & Kording , 2011 ; Ahrens et al. , 2013 ; Carrillo-Reid et al. , 2015 ) . Calcium imaging , a microscopic video technique that enables the concurrent observation of hundreds of neurons in vitro and in vivo ( Denk et al. , 1990 ; Helmchen & Denk , 2005 ; Flusberg et al. , 2008 ) , is best suited to witness such motifs if they indeed exist . In recent years , a variety of methods have been developed to identify neuronal assemblies . These methods range from approaches for the detection of synchronous spiking , up to more advanced methods for the detection of arbitrary spatio-temporal firing patterns ( Comon , 1994 ; Nicolelis et al. , 1995 ; Grün et al. , 2002a ; b ; Lopes-dos Santos et al. , 2013 ; Russo & Durstewitz , 2017 ; Peter et al. , 2017 ) . All of these methods , however , require a spike time matrix as input . Generating such a spike time matrix from calcium imaging data requires the extraction of individual cells and discrete spike times . Again , many methods have been proposed for these tasks ( Mukamel et al. , 2009 ; Pnevmatikakis & Paninski , 2013 ; Pnevmatikakis et al. , 2013 ; Diego et al. , 2013 ; Diego & Hamprecht , 2013 ; Pachitariu et al. , 2013 ; Pnevmatikakis et al. , 2014 ; Diego & Hamprecht , 2014 ; Kaifosh et al. , 2014 ; Pnevmatikakis et al. , 2016 ; Apthorpe et al. , 2016 ; Inan et al. , 2017 ; Spaen et al. , 2017 ; Klibisz et al. , 2017 ; Speiser et al. , 2017 ; Zhou et al. , 2018 ) . Given the low signal-to-noise ratios ( SNR ) , large background fluctuations , non-linearities , and strong temporal smoothing due to the calcium dynamics itself as well as that of calcium indicators , it is impressive how well some of these methods perform , thanks to modern recording technologies and state-of-the-art regularization and inference ( Pnevmatikakis et al. , 2016 ; Zhou et al. , 2018 ) . Still , given the difficulty of this data , errors in segmentation and spike extraction are unavoidable , and adversely affect downstream processing steps that do not have access to the raw data . Hence , properly annotating data and correcting the output from automatic segmentation can still take up a huge amount of time . In this paper , we propose LeMoNADe ( Learned Motif and Neuronal Assembly Detection ) , a variational autoencoder ( VAE ) based framework specifically designed to identify repeating firing motifs with arbitrary temporal structure directly in calcium imaging data ( see figure 1 ) . The encoding and decoding networks are set up such that motifs can be extracted directly from the decoding filters , and their activation times from the latent space ( see sec . 3 ) . Motivated by the sparse nature of neuronal activity we replace the Gaussian priors used in standard VAE . Instead we place Bernoulli priors on the latent variables to yield sparse and sharply peaked motif activations ( sec . 3.1 ) . The choice of discrete Bernoulli distributions makes it necessary to use a BinConcrete relaxation and the Gumbel-softmax reparametrization trick ( Maddison et al. , 2016 ; Jang et al. , 2017 ) to enable gradient descent techniques with low variance ( sec . 3.3 ) . We add a β-coefficient ( Higgins et al. , 2017 ) to the loss function in order to adapt the regularization to the properties of the data ( sec . 3.3 ) . Furthermore , we propose a training scheme which allows us to process videos of arbitrary length in a computationally efficient way ( sec . 3.4 ) . On synthetically generated datasets the proposed method performs as well as a state-of-the-art motif detection method that requires the extraction of individual cells ( sec . 4.1 ) . Finally , we detect possible repeating motifs in two fluorescent microscopy datasets from hippocampal slice cultures ( sec . 4.2 ) . A PyTorch implementation of the proposed method is released at https : //github.com/EKirschbaum/LeMoNADe . 2 RELATED WORK . Autoencoder and variational autoencoder Variational Autoencoders ( VAEs ) were introduced by Kingma & Welling ( 2014 ) and have become a popular method for unsupervised generative deep learning . They consist of an encoder , mapping a data point into a latent representation , and a decoder whose task is to restore the original data and to generate samples from this latent space . However , the original VAE lacks an interpretable latent space . Recent suggestions on solving this problem have been modifications of the loss term ( Higgins et al. , 2017 ) or a more structured latent space ( Johnson et al. , 2016 ; Deng et al. , 2017 ) . VAE have also been successfully used on video sequences . Li & Mandt ( 2018 ) learn a disentangled representation to manipulate content in cartoon video clips , while Goyal et al . ( 2017 ) combine VAEs with nested Chinese Restaurant Processes to learn a hierarchical representation of video data . Johnson et al . ( 2016 ) use a latent switching linear dynamical system ( SLDS ) model combined with a structured variational autoencoder to segment and categorize mouse behavior from raw depth videos . Unfortunately , this model is not directly applicable to the task of identifying motifs with temporal structure from calcium imaging data for the following reasons : Firstly , neuronal assemblies are expected to extend over multiple frames . Since in the model by Johnson et al . ( 2016 ) the underlying latent process is a relatively simple first-order Markovian ( switching ) linear process , representing longer-term temporal dependencies will be very hard to achieve due to the usually exponential forgetting in such systems . Secondly , in the model of Johnson et al . ( 2016 ) each frame is generated from exactly one of M latent states . For calcium imaging , however , most frames are not generated by one of the M motifs but from noise , and different motifs could also temporally overlap which is also not possible in the model by Johnson et al . ( 2016 ) . Closest to our goal of detecting motifs in video data is the work described in Bascol et al . ( 2016 ) . In this approach , a convolutional autoencoder is combined with a number of functions and regularization terms to enforce interpretability both in the convolutional filters and the latent space . This method was successfully used to detect patterns in data with document structure , including optical flow features of videos . However , as the cells observed in calcium imaging are spatially stationary and have varying luminosity , the extraction of optical flow features makes no sense . Hence this method is not applicable to the task of detecting neuronal assemblies in calcium imaging data . Cell segmentation and spike time extraction from calcium imaging data Various methods have been proposed for automated segmentation and signal extraction from calcium imaging data . Most of them are based on non-negative matrix factorization ( Mukamel et al. , 2009 ; Pnevmatikakis & Paninski , 2013 ; Pnevmatikakis et al. , 2013 ; 2014 ; Diego & Hamprecht , 2014 ; Pnevmatikakis et al. , 2016 ; Inan et al. , 2017 ; Zhou et al. , 2018 ) , clustering ( Kaifosh et al. , 2014 ; Spaen et al. , 2017 ) , and dictionary learning ( Diego et al. , 2013 ; Diego & Hamprecht , 2013 ; Pachitariu et al. , 2013 ) . Recent approaches started to use deep learning for the analysis of calcium imaging data . Apthorpe et al . ( 2016 ) and Klibisz et al . ( 2017 ) use convolutional neural networks ( CNNs ) to identify neuron locations and Speiser et al . ( 2017 ) use a VAE combined with different models for calcium dynamics to extract spike times from the calcium transients . Although many sophisticated methods have been proposed , the extraction of cells and spike times from calcium imaging data can still be prohibitively laborious and require manual annotation and correction , with the accuracy of these methods being limited by the quality of the calcium recordings . Furthermore , some of the mentioned methods are specially designed for two-photon microscopy , whereas only few methods are capable to deal with the low SNR and large background fluctuations in single-photon and microendoscopic imaging ( Flusberg et al. , 2008 ; Ghosh et al. , 2011 ) . Additional challenges for these methods are factors such as non-Gaussian noise , non-cell background activity and seemingly overlapping cells which are out of focus ( Inan et al. , 2017 ) . Neuronal assembly detection The identification of neuronal assemblies in spike time matrices has been studied from different perspectives . For the detection of joint ( strictly synchronous ) spike events across multiple neurons , rather simple methods based on PCA or ICA have been proposed ( Comon , 1994 ; Nicolelis et al. , 1995 ; Lopes-dos Santos et al. , 2013 ) , as well as more sophisticated statistical methods such as unitary event analysis ( Grün et al. , 2002a ; b ) . Higher-order correlations among neurons and sequential spiking motifs such as synfire chains can be identified using more advanced statistical tests ( Staude et al. , 2010a ; b ; Gerstein et al. , 2012 ) . The identification of cell assemblies with arbitrary spatio-temporal structure has been addressed only quite recently . One approach recursively merges sets of units into larger groups based on their joint spike count probabilities evaluated across multiple different time lags ( Russo & Durstewitz , 2017 ) . Another method uses sparse convolutional coding ( SCC ) for reconstructing the spike matrix as a convolution of spatio-temporal motifs and their activations in time ( Peter et al. , 2017 ) . An extension of this method uses a group sparsity regularization to identify the correct number of motifs ( Mackevicius et al. , 2018 ) . To the authors ’ knowledge , solely Diego & Hamprecht ( 2013 ) address the detection of neuronal assemblies directly from calcium imaging data . This method , however , only aims at identifying synchronously firing neurons , whereas the method proposed in this paper can identify also assemblies with more complex temporal firing patterns .

Summary: The paper proposes a VAE-style model for identifying motifs from calcium imaging videos. As opposed to standard VAE with Gaussian latent variables it relies on Bernouli variables and hence, requires Gumbel-softmax trick for inference. Compared to methods based on matrix factorization, the proposed method has the advantage of not requiring any preprocessing on the imaging videos. My main comments are as follows: