Machine learning for neural decoding

Despite rapid advances in machine learning tools, the majority of neural decoding approaches still use traditional methods. Improving the performance of neural decoding algorithms allows us to better understand the information contained in a neural population, and can help advance engineering applications such as brain machine interfaces. Here, we apply modern machine learning techniques, including neural networks and gradient boosting, to decode from spiking activity in 1) motor cortex, 2) somatosensory cortex, and 3) hippocampus. We compare the predictive ability of these modern methods with traditional decoding methods such as Wiener and Kalman filters. Modern methods, in particular neural networks and ensembles, significantly outperformed the traditional approaches. For instance, for all of the three brain areas, an LSTM decoder explained over 40% of the unexplained variance from a Wiener filter. These results suggest that modern machine learning techniques should become the standard methodology for neural decoding. We provide a tutorial and code to facilitate wider implementation of these methods.  Neural decoding uses activity recorded from the brain to make predictions about variables in the outside world. For example, researchers predict movements based on activity in motor cortex [1, 2], predict decisions based on activity in prefrontal and parietal cortices [3, 4], and predict locations based on activity in the hippocampus [5, 6]. There are two primary purposes of decoding. First, it is an increasingly critical tool for understanding how neural signals relate to the outside world. It can be used to determine how much information the brain contains about an external variable (e.g., sensation or movement) [7-9], and how this information differs across brain areas [10-12], experimental conditions [13, 14], disease states , and more. Second, it is useful in engineering contexts, such as for brain machine interfaces (BMIs), where signals from motor cortex are used to control computer cursors , robotic arms [16], and muscles . Decoding is a central tool for neural data analysis. When predicting a continuous variable, decoding is simply a regression problem and when predicting a discrete variable, decoding is simply a classification problem. Thus, there are many methods that can be used for neural decoding. However, despite the recent advances in machine learning techniques, it is still common to use traditional methods such as linear regression. Using modern machine learning tools for neural decoding would likely significantly boost performance, and might allow deeper insights into neural function. Here, we first give a brief tutorial so that readers can get started with using standard machine learning methods for decoding. We provide companion code so that readers can easily use a variety of decoding methods. Next, we compare the performance of many different machine learning methods to decode information from neural spiking activity. We predict movement velocities from macaque motor cortex and sensorimotor cortex, and locations in space from rat hippocampus. In all brain regions, modern methods, in particular neural networks and ensembles, led to the highest accuracy decoding, even for limited amounts of data. Tutorial for getting started with using machine learning for decoding: Code We have made Python code available at https://github.com/KordingLab/Neural_Decoding, which accompanies the tutorial below. This includes code that will correctly format the neural and output data for decoding, a tutorial for hyperparameter optimization, and examples of using many different decoders. We go into more detail on these topics below. General framework for decoding The decoding problem we are considering can be summarized as follows. We have N neurons whose spiking activity is recorded for a period of time, T (Fig. 1a). While we focus here on spiking neurons, the same methods could be used with other forms of neural data, such as the BOLD activity of N voxels, or the power in particular frequency bands of N LFP or EEG signals. We have also recorded outputs that we are trying to predict over that same time period (Fig. 1a). Here, we focus on output variables that are continuous (e.g., velocity, position), rather than discrete (e.g., choice). However, the general framework is very similar for discrete output variables. The first choice we need to make is to decide the temporal resolution, R, for decoding. That is, do we want to make a prediction every 50ms, 100ms, etc? We need to put the input and output into bins of length R (Fig. 1a). It is common (although not necessary) to use the same bin size for the neural data and output data, and we do so here. Thus, we will have approximately T/R total bins of neural activity and outputs. Within each bin, we compute the average activity of all neurons and the average value of the output. Next, we need to choose the time period of neural activity used to predict a given output. In the simplest case, the activity from all neurons in a given time bin would be used to predict the output in that same time bin. However, it is often the case that we want the neural data to precede the output (e.g., in the case of movements) or follow the decoder output (e.g., in the case of sensation). Plus, we often want to use neural data from more than one bin (e.g., using 500 ms of preceding neural data to predict a movement in the current 50 ms bin). In the following, we use the nomenclature that B time bins of neural activity are being used to predict a given output. For example, if we use one bin preceding the output, one concurrent bin, and one following bin, then B=3 . Note that when multiple bins of neural data are used to predict an output (B>1), then overlapping neural data will be used to predict different output times . When multiple bins of neural data are used to predict an output, then we will need to exclude some output bins. For instance, if we are using one bin of neural data preceding the output, then we cannot predict the first output bin, and if we are using one bin of neural data following the output, then we cannot predict the final output bin . Thus, we will be predicting K total output bins, where K is less than the total number of bins (T/R). To summarize, our decoders will be predicting each of these K outputs using B surrounding bins of activity from N neurons. Below, we describe how to format the neural data and output variables for use in different types of decoders. Non-recurrent decoders: For many “non-recurrent” decoders, we are just solving a standard machine learning regression problem. We have N x B features (the firing rates of each neuron in each relevant time bin) that are used to predict each output . If there is a single output that is being predicted, it can be put in a vector, Y, of length K. Note that for many decoders, if there are multiple outputs, each is independently decoded. If multiple outputs are being simultaneously predicted, which can occur with neural network decoders, the outputs can be put in a matrix Y, that has K rows and d columns, where d is the number of outputs being predicted. The input covariate matrix, X, has N x B columns (one for each feature) and K rows (corresponding to each output being predicted). This is now the format of a standard regression problem. Linear regression simply finds a linear combination of these features that predicts the output. More sophisticated forms of regression use nonlinear combinations of features for predictions. In general, this format is beneficial because there are many machine learning regression techniques that can easily be substituted for one another. We provide code for a Wiener filter (linear regression), a Wiener cascade (a linear-nonlinear model), support vector regression, XGBoost (gradient boosted trees), and feedforward neural networks (see Methods). We test the performance of these decoders in  Recurrent neural network decoders: When using recurrent neural networks (RNNs) for decoding, we need to put the inputs in a different format. Recurrent neural networks explicitly model temporal transitions across time . In the non-recurrent decoders, there were N x B features that were equivalently used for prediction, regardless of the time bin they came from. However, with a recurrent decoder, at each time bin, N features (the firing rates of all neurons in that time bin) are used for predicting the hidden state of the system at that time. Along with being a function of the N features, the hidden state at a time bin is also a function of the hidden state at the previous time bin . After transitioning through all B bins, the hidden state in this final bin is used to predict the output. This architecture allows the decoder to take advantage of temporal structure in the data, and allowing it (via its hidden state) to integrate the effect of neural inputs over an extended period of time. For use in this type of decoder, the input can be formatted as a 3-dimensional matrix of size K x N x B . That is, for each row (corresponding to the output that is predicted), there will be N features (2nd matrix dimension) over B bins (3rd matrix dimension) used for prediction. Within this format, different types of RNNs, including those more sophisticated than the standard RNN shown in Fig. 1c, can be easily switched for one another. We provide code for a standard recurrent network, a gated recurrent unit (GRU) network, and a long short-term memory (LSTM) network. In Results, we test the performance of these decoders. Decoders with additional information: While the focus of this tutorial is on decoders that fit into standard machine learning frameworks, we want to briefly mention two other commonly used decoders. The Kalman filter and its variants have frequently been used in the brain computer interface field fo