Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex
Authors: Jesse A Livezey, Kristofer E Bouchard, Edward F Chang
Journal: PLOS Computational Biology
Accepted: 2019-05-10T00:00:00Z


A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network’s confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.Author summary: It has been demonstrated that deep networks can achieve state of the art results on a number of classic machine learning tasks, but it is not currently clear whether deep networks can enjoy the same success in science, where not only accuracy but scientific understanding is desired. For example, the relationship between neural features and features of speech is often examined through the use of single-layer statistical models. However, single-layer models are unlikely to be able to describe the complex representations in higher-order brain areas during speech production. In this study, we show that deep networks achieve state of the art accuracy when classifying speech syllables from the amplitude of cortical surface electrical potentials. Furthermore, deep networks reveal an articulatory speech hierarchy consistent with previously studies which used hand-designed features. We also report a novel positive coupling between the beta and high-gamma bands during speech production in “active” cortical areas. However, using deep networks we show that, compared to lower frequency bands, the high gamma amplitude is by far the most informative signal for classifying speech.