Synthesising Realistic Calcium Traces of Neuronal Populations Using GAN

1 INTRODUCTION . The ability to record accurate neuronal activities from behaving animals is essential for the study of information processing in the brain . Electrophysiological recording , which measures the rate of change in voltage by microelectrodes inserted in the cell membrane of a neuron , has high temporal resolution and is considered the most accurate method to measure spike activities ( Dayan & Abbott , 2001 ) . However , this method is not without shortcomings ( Harris et al. , 2016 ) . For instance , a single microelectrode can only detect activity from few neurons in close proximity , and extensive pre-processing is required to infer single-unit activity from a multi-unit signal . Disentangling circuit computations in neuronal populations of a large scale remains a difficult task ( Rey et al. , 2015 ) . On the other hand , calcium imaging monitors the calcium influx in the cell as a proxy of an action potential ( Berridge et al. , 2000 ) . Contrary to electrophysiological recordings , this technique yields data with high spatial resolution and low temporal resolution ( Grienberger & Konnerth , 2012 ) , and has become a powerful imaging technique to monitor large neuronal populations . With the advancements in these recording technologies , it has become increasingly easier to obtain high-quality neuronal activity data in vivo from live animals . However , due to ethical considerations , the acquired datasets are often limited by the number of trials or the duration of each trial on a live animal . This poses a problem for assessing analysis techniques that take into account higher-order correlations ( Brown et al. , 2004 ; Staude et al. , 2010 ; Stevenson & Kording , 2011 ; Saxena & Cunningham , 2019 ) . Even for linear decoders , the number of trials can be more important for determining coding accuracy than the number of neurons ( Stringer et al. , 2019 ) . Generative models of neuronal activity hold the promise of alleviating the above problem by enabling the synthesis of an unlimited number of realistic samples for assessing advanced analysis methods . Popular modelling approaches such as the maximum entropy framework ( Schneidman et al. , 2006 ; Tkačik et al. , 2014 ) and the latent variable model ( Macke et al. , 2009 ; Lyamzin et al. , 2010 ) have shown ample success in modelling spiking activities , though many of these models re- quire strong assumptions on the data and can not generalize to different cortical areas . To this end , GANs have shown tremendous success in synthesizing data across a vast variety of domains and data-types ( Karras et al. , 2017 ; Gomez et al. , 2018 ; Donahue et al. , 2019 ) , and are good candidates for modelling neuronal activities . Spike-GAN ( Molano-Mazon et al. , 2018 ) demonstrated that GANs can model neural spikes that accurately match the statistics of real recorded spiking behaviour from a small number of neurons . Moreover , the discriminator in Spike-GAN is able to learn to detect which population activity pattern is the relevant feature , and this can provide insights into how a population of neurons encodes information . Ramesh et al . ( 2019 ) trained a conditional GAN ( Mirza & Osindero , 2014 ) , conditioned on the stimulus , to generate multivariate binary spike trains . They fitted the generative model with data recorded in the V1 area of macaque visual cortex , and the GAN generated spike trains were able to capture the firing rate and pairwise correlation statistics better than the dichotomized Gaussian model ( Macke et al. , 2009 ) and a deep supervised convolution model . Nevertheless , the aforementioned deep generative models operate on spike trains which are discrete in nature , and back-propagation on discrete data remains a difficult task ( Caccia et al. , 2018 ) . For instance , Ramesh et al . ( 2019 ) used the REINFORCE gradient estimate ( Williams , 1992 ) to train the generator in order to perform back-propagation on discrete data . Still , gradient estimation with the REINFORCE approach yields large variance , which is known to be challenging for optimization ( Maddison et al. , 2016 ; Zhang et al. , 2017 ) . In addition , generating and training on binary spike trains directly introduces uncertainty as the generator has to learn the deconvolution process as well , making it an even more difficult task . In this work , we investigate the possibility of synthesising continuous calcium fluorescent signals using the GAN framework , as a method to scale-up or augment the amount of population activity data . In addition , modelling the calcium signals directly has several advantages ( a ) the generator needs to learn the deconvolution process when synthesising directly on binary spike trains , hence there is additional uncertainty , which is not present for calcium signals . ( b ) Calcium imaging signals have inherently more information about the neuronal activities than binary spike trains . ( c ) Based on calcium signals with known ground-truth , calcium deconvolution algorithms can be evaluated . Hence , We devised a workflow to synthesize and evaluate calcium imaging signals , then validate the method on artificial data with known ground-truth as well as mimicking real two-photon calcium ( Ca2+ ) imaging data as recorded from the primary visual cortex of a behaving mouse ( Pakan et al. , 2018 ; Henschke et al. , 2020 ) . 2 METHODS . 2.1 NETWORK ARCHITECTURE . The original GAN framework , introduced in Goodfellow et al . ( 2014 ) , plays a min-max game where the generator G attempts to generate convincing samples from the latent space Z , and the discriminator D learns to distinguish between generated samples and real samples X . In this work , we use the WGAN-GP ( Gulrajani et al. , 2017 ) formulation of the loss function without the need of incorporating any information of the neural activities into the training objective : LD = E z∼Z [ D ( G ( z ) ) ] − E x∼X [ D ( x ) ] + λ E x̃∼X̃ [ ( ‖ ∇x̃D ( x̃ ) ‖2 −1 ) 2 ] ( 1 ) where λ denotes the gradient penalty coefficient , x̃ = x+ ( 1− ) x̂ are samples taken between the real and generated data distribution . For learning calcium signal generation , we adapted the WaveGAN architecture ( Donahue et al. , 2019 ) , which has shown promising results in audio signal generation . In the generator , we used 1-dimensional transposed convolution layers to up-sample the input noise . We added Layer Normalization ( Ioffe & Szegedy , 2015 ) in between each convolution and activation layer , in order to stabilize training as well as to make the operation compatible with the WGAN-GP framework . To improve the model learning performance and stability , the calcium signals were scaled to the range between 0 and 1 by normalizing with the maximum value of the calcium signal in the data . Correspondingly , we chose sigmoid activation in the output layer of the generator and then re-scaled the signals to their original range before inferring their spike trains . The architecture of the discriminator in our model is largely a mirror of the generator , with the exception of the removal of Layer Normalization and instead of up-sampling the input with transposed convolution , we used a simple convolution layer . Samples generated using transposed convolution often exhibit the ” checkerboard ” artifacts described by Odena et al . ( 2016 ) , where the output exhibits repeated patterns ( usually very subtle to the eye ) due to a filter being applied unevenly to the receptive field . In the context of signal generation , the discrimination could exploit the periodic artifacts pattern and learn a naive policy to reject generated samples . Donahue et al . ( 2019 ) proposed the Phase Shuffle mechanism in the discriminator to address the aforementioned issue . The Phase Shuffle layer randomly shifts the activated units after each convolution layer within [ −n , n ] , in order to distort the periodic pattern . Hence , the resulting samples constitute a more challenging task for the discriminator . Figure A.4 shows a simple illustration of the Phase Shuffle operation . In our network , we incorporated the Phase Shuffle operation , as well as using a kernel size that is divisible by the stride size , as suggested in Odena et al . ( 2016 ) . We apply the Phase Shuffle operation after each convolution layer , which has led to a noticeable improvement in the generated samples . Table A.1 shows the exact architecture of our model . 2.2 MODEL PIPELINE . We devised a consistent model analysis pipeline to evaluate the quality of samples generated by the model , as well as its ability to generalize , in the context of neuronal population spiking activities . The complete model analysis pipeline is shown in Figure A.2 . As calcium imaging is largely being used as a proxy to monitor spiking activities , we have decided to evaluate and present the inferred spike trains instead of raw calcium signals . We used the Online Active Set method to Infer Spikes ( OASIS ) AR1 deconvolution algorithm ( Friedrich et al. , 2017 ) to infer spiking activities from calcium fluorescent signals . We apply OASIS to both the training data and generated data to ensure the potential bias in the deconvolution process applies to the two sets of data . We then trained both the generator and discriminator with the WGAN-GP framework ( Gulrajani et al. , 2017 ) , with 5 discriminator update steps for each generator update step . We used the Adam optimizer ( Kingma & Ba , 2014 ) to optimize both networks , with a learning rate of λ = 10−4 , β1 = 0.9 and β2 = 0.9999 . To speed up the training process , we incorporated Mixed Precision training ( Micikevicius et al. , 2017 ) in our codebase . The exact hyper-parameters being used in this work can be found in Table A.2 . After inferring the spike trains from the generated calcium signals , we then measure the spike train statistics and similarities using the Electrophysiology Analysis Toolkit ( Denker et al. , 2018 ) . Following some of the previous works in spike generation ( Macke et al. , 2009 ; Molano-Mazon et al. , 2018 ; Ramesh et al. , 2019 ) , we evaluate the performance of our model with the following statistics and similarities : ( a ) mean firing rate for evaluating single neuron statistics ; ( b ) pairwise Pearson correlation coefficient for evaluating pairwise statistics ; ( c ) pairwise van-Rossum distance ( Rossum , 2001 ) for evaluating general spike train similarity . Importantly , we evaluate these quantities across the whole population for each neuron or neuron pair and each short time interval ( 100 ms ) and compare the resulting distributions over these quantities obtained from training data as well as generated data . We therefore validate the whole spatiotemporal first- and second-order statistics as well as general spike train similarities . 2.3 DATA . 2.3.1 DICHOTOMIZED GAUSSIAN ARTIFICIAL DATA . In order to verify that CalciumGAN is able to learn the underlying distribution and statistics of the training data , we generated our own ground-truth dataset with pre-defined mean and covariance using the dichotomized Gaussian ( DG ) model ( Macke et al. , 2009 ) . The model uses a multivariate normal distribution to generate latent continuous random variables which are then thresholded to generate binary variables representing spike trains . The DG model has mean vector and covariance matrix as free parameters . To generate data from this model , we used the sample means and sample covariances obtained from real recorded data ( see Section 2.3.2 ) . In alignment with the recorded data , we generated correlated spike trains for N = 102 neurons with a duration of 899 seconds and at 24Hz , hence a matrix with shape ( 21576 , 102 ) . In order to obtain calcium-like signals c from spike trains s with length T , we convolved the generated spike trains with a calcium response kernel and added noise , as described in Friedrich et al . ( 2017 ) : st = gst−1 + st 1 ≤ t ≤ T ( 2 ) c = b+ s+ σu u ∼ N ( 0 , 1 ) ( 3 ) where g denotes a finite impulse response filter , b is the baseline value of the signal and σ is the noise standard deviation . In our work , we set g = 0.95 , σ = 0.3 and b = 0 . We scale the signal range to the unit interval . The data is then segmented using a sliding window along the time dimension with a stride of 2 and a window size of T = 2048 ( around 85 seconds in experiment time ) . We apply the segmentation procedure to both the signal and spike data , hence resulting in two matrices with shape ( 9754 , 2048 , 102 ) . Examples of signals and spikes generated from the DG model can be found in Figure A.1a .

Summary: The paper proposes to use a GAN framework to generate the realistic neuronal calcium signals, enabling to scale-up the neuronal population activity data. The solution is based on WAVEGAN architecture with Wasserstein distance to train on calcium fluorescent signals. The experiments are performed in comparison to artificial calcium signals with known ground-truth closely resembles the underlying data distribution. The accuracy of the approach, robustness of generated signals from the model are evaluated.