Modelling neuronal behaviour with time series regression: Recurrent Neural Networks on synthetic C. elegans data

1 INTRODUCTION . The study of the human brain is probably one of the greatest challenges in the field of neuroscience . Recent developments in experimental neuroscience have considerably increased the availability of novel recordings and reconstructions shedding further light into the structure and function of the brain as well as many other systems . But understanding the complexities behind the relations between structure and function as well as the behaviour of such systems across multiple scales in these neuronal collections is constrained by the methods available to study them . This challenge has raised interest in many related fields , such as electrophysiological analysis , imaging techniques , brain-related medicine , computational modelling and simulation , model reduction . Many of these efforts , while not directly providing specific information regarding structural or functional dynamics , do supply large volumes of recordings , measurements or simulations of observable input-output behaviour . The availability of these large datasets raises the question whether low complexity , data-driven , black-box models can be used to model such input-output relations with low error , avoiding the excessive inner detail which may not be known or available . To determine whether such an approach can be used for large , complex systems , one research direction is the study of smaller and simpler nervous systems , for which the underlying principles of network organization and information processing are easier to postulate . These organisms can become useful models to gain insight into the fundaments of neuronal dynamics and whole brain organization , to validate hypotheses , to develop and test modelling methods , simulation instruments and model reduction techniques . The hope is that the knowledge gained from these analyses and the techniques developed for these simpler organisms can later be used to model more complex systems . Caenorhabditis Elegans ( C. elegans ) belongs to this category of organisms and is quickly becoming one of the benchmarks in whole brain organization studies . C. elegans is a nematode ( roundworm ) of about 1 mm in length with a compact nervous system consisting of less than 1000 cells across all sexes and around 15000 connections ( Cook et al. , 2019 ) . This rather small nervous system allows the worm to solve basic problems such as feeding , predator avoidance and mate-finding . The relative simplicity of C. elegans allowed for its almost complete description from different perspectives and scales , from its genetics and genomics to the molecular biology , structural anatomy , neuronal function , circuits and behaviour . This information is available in comprehensive databases of genetics and genomics ( Hunt-Newbury et al. , 2007 ) , electron micrographs and associated data , online books and atlases of the neurobiology , structural and behavioural anatomy ( Jackson et al. , 2014 ) . Creating a realistic model that encapsulates all this information is not a trivial task . Open-source databases of digitally reconstructed neurons ( Gleeson et al. , 2019 ) , computational models ( Szigeti et al. , 2014 ) and collaborative solutions ( Cantarelli et al. , 2018 ) are opening the door for more flexible , multi-scale and multi-algorithm simulation environments for C. elegans and other complex biological systems . The underlying models are based on the connectome , the map of the neuronal connections in the brain . Usually described as a neuronal network , the connectome is a graph where the nodes are the neurons and the edges represent the synapses . The complete connectome of C. elegans contains 302 neurons for the adult hermaphrodite ( Varshney et al. , 2011 ) and 385 neurons for the male ( Cook et al. , 2019 ) , but for the latter the respective 3D reconstructions are not yet published . Digital reconstructions for the male are only available for the posterior nervous system of 144 neurons ( Jarrell et al. , 2012 ) . The more complex the organism , the more complicated the resulting model , needing more computationally demanding and potentially intractable simulations of its dynamic behaviour . This increased complexity stems from the detailed modelling of the internal structure . However , in many cases , especially of highly complex systems , this detail is not available since the internal mechanisms may not be well known or mapped or it may be simply impossible to examine and record . For that reason , frequently one is really only interested in the peripheral , or input-output behaviour . This motivates our efforts not only to place the focus more on observable input-output data , as well as to try and generate reduced models that avoid extraneous detail not necessary to explain these peripheral relations . In this work we propose a methodology for generating a reduced order model of the neuronal behaviour of organisms using only peripheral information . We use C. elegans as a proxy for our study . Realistic models of C. elegans , which take into account spatial distribution and biophysical properties of neuronal compartments have been reported in the literature ( Gleeson et al. , 2018 ) . We start with a similar model created in-house . Our model comprises the complete connectome of the adult hermaphrodite of C. elegans , with 302 multi-compartmental neurons and 6702 synapses ( Varshney et al. , 2011 ) . The model was validated ( Anonymous , 2021 ) against four scenarios described in related literature ( Kim et al. , 2019 ) : Forward Crawling Motion ( FCM ) for the full network , Ablation of AVB interneurons + FCM , Ablation of AVA interneurons + FCM and the Nictation behaviour . We reproduce here the FCM scenario , in which we apply stimulus on the touch sensitive sensory neurons and the interneurons known to be part of the forward movement circuit and we check the activity of the motor neurons associated with forward locomotion . Since we find strong activity in most of these neurons , we conclude that the worm moves forward . Based on synthetic data extracted from this high-fidelity model , we create a completely equation-free data-driven model assuming no prior knowledge of the original system ’ s structure and equations , using neural networks trained on datasets representing the system ’ s response to different input signals . The ultimate goal is to generate a reduced model to replace the original , detailed one . This reduced model should be able to reproduce with reasonably low RMSE the behaviour of the realistic model while having fewer degrees of freedom . In this work we focus on the issue of reduced RMSE , which we equate to fidelity in reproducing the system dynamics , showing that we can produce sufficiently accurate models for analysing the behaviour of the C. elegans nervous system using neural networks . 2 RELATED WORK AND CONTEXT . The connectome-based models mentioned above are often termed white-box models , as they are based on direct knowledge and access to the internal structure and parameters ’ values of the modelled system . These are distributed models , where each neuron has a 3D description and position in space and the synapses are associated with neuronal sections . Such models enable highly accurate simulation of the dynamic behaviour of organisms , but easily become extremely complex as they incorporate detailed structural and functional information of the system . While the white-box approach ensures access to and evaluation of inner parameters during simulation , it has been shown that the activity of complex networks of neurons can often be described by relatively few distinct patterns , which evolve on low-dimensional subspaces ( Karasözen , 2020 ) . This knowledge , together with the ever-present need to avoid potential numerical intractability in large-scale networks with many degrees of freedom , has generated renewed interest in applying model reduction , often also referred to as model compression , to these neuronal networks , including techniques such as Dynamic Mode Decomposition ( DMD ) ( Brunton et al. , 2016 ) , Proper Orthogonal Decomposition ( POD ) ( Kellems et al. , 2009 ) and Discrete Empirical Interpolation ( DEIM ) ( Lehtimäki et al. , 2019 ) . Depending on the level of morphological accuracy of the underlying models , reduction techniques can have any shade of grey from white-box to black-box , the latter assuming no preliminary knowledge of the system structure and building the model solely out of knowledge of its input-output behaviour . Black-box approaches are often built upon data-driven models , sometimes learning-based , which have the ability to grasp more naturally and more efficiently the complexity induced by the profound nonlinearities in the neuronal transmission of information . Machine-learning techniques are used to extract data-driven reduced order models for systems arising from differential equations describing the intrinsic dynamics ( Regazzoni et al. , 2019 ) and even to extract the governing equations of the estimated model ( Sun et al. , 2020 ) . It is therefore quite natural to consider using state of the art learning methods for developing reduced models of neuronal behaviour using data obtained from available recordings or even simulations obtained with more complex models . Especially designed to capture temporal dynamic behaviour , Recurrent Neural Networks ( RNNs ) , in their various architectures such as Long Short-Term Memory ( LSTMs ) and Gated Recurrent Units ( GRUs ) , have been extensively and successfully used for forecasting or detecting faults in multivariate time series data ( Massaoudi et al. , 2019 ) , ( Gallicchio et al. , 2018 ) , ( Yuan et al. , 2020 ) , ( Filonov et al. , 2016 ) . Bidirectional LSTMs were used to model genome data by Tavakoli ( 2019 ) , whereas a combination of CNNs and LSTMs generates a model for epileptic seizure recognition using EEG signal analysis in Xu et al . ( 2020 ) . An attempt to model the human brain activity based on fMRI using RNNs ( LSTMs and GRUs ) is reported in Güçlü & van Gerven ( 2017 ) . In recent years , deep network approaches were used to model realistic neural activity data ( Molano-Mazon et al. , 2018 ) , ( Bellec et al. , 2021 ) , ( Karampatziakis , 2010 ) . Few studies examined the behavioural output of network models of C. elegans using machine-learning techniques . RNNs are generated in a grey-box manner to study the chemotaxis behaviour ( Xu et al. , 2010 ) or to predict the synaptic polarities ( Lanza et al. , 2021 ) of C. elegans , yet these models only include a subset of the connectome . 3 METHODS . Given that the starting point is in fact represented by time series data obtained from simulations of the realistic connectome-based model , the modelling task is akin to a sequence to sequence conversion for which the most suitable neural network models are sequential ones . In this work we analyze the suitability of three of the most commonly used architectures for recurrent neural networks . We start with the least complex unit , the simple RNN , originally proposed in the 1980 ’ s to model sequence data ( Rumelhart et al. , 1986 ) , ( Werbos , 1988 ) , ( Elman , 1990 ) . The second model used for the recurrent layer is the LSTM unit , developed by Hochreiter & Schmidhuber ( 1997 ) and later improved with the introduction of the forget gate to adaptively release internal resources when necessary ( Gers et al. , 1999 ) . Finally we analyze its sibling , the GRU ( Cho et al. , 2014 ) . 3.1 RECURRENT NEURAL NETWORKS . RNNs ( Rumelhart et al. , 1986 ) , ( Werbos , 1988 ) , ( Elman , 1990 ) are a family of neural networks used for processing sequential data , particularly adept to processing a sequence of values x ( 1 ) , ... , x ( t ) , and in most cases capable to process sequences of variable length . RNNs appear from the relaxation of the condition on Feedforward Neural Networks ( FFNNs ) that neurons in a given layer do not form connections among themselves . Although RNNs , which are trained using Backpropagation Through Time ( BPTT ) ( Werbos , 1990 ) , seem to be a good model for sequential tasks , they are known to suffer mainly from two issues , vanishing and exploding gradients ( Bengio et al. , 1994 ) . Exploding gradients ( Bengio et al. , 1994 ) refer to a large increase in the norm of the gradient during training , which appears due to the explosion of long term components that can grow exponentially more than short term ones . This is the less common of the two problems and there are known solutions to handle it , such as the clipping gradient technique ( Pascanu et al. , 2012 ) . A harder to solve problem is the vanishing gradient issue ( Bengio et al. , 1994 ) , which refers to when long term components go exponentially fast to norm 0 , making it impossible for the model to learn the correlation between temporally distant events . In order to faithfully reproduce the dynamics of our system the simulations used for generating the datasets require the use of fine time steps , leading to long data sequences . This in turn implies that the response at a given time will depend on values which are far back in the sequence . This situation , however unavoidable , may lead the RNN to experience difficulties in learning our data resulting in a model with unacceptable RMSE .

Summary: Authors show how the nervous system of C. elegans can be modelled and simulated with data-driven models using different neural network architectures. Specifically, they target the use of state of the art recurrent neural networks architectures such as LSTMs and GRUs and compare these architectures in terms of their properties and their RMSE, as well as the complexity of the resulting models. Authors show that GRU models with a hidden layer size of 4 units are able to accurately reproduce the system’s response to very different stimuli.