# Deep Representation of EEG Signals Using Spatio-Spectral Feature Images
###### Abstract  
Modern deep neural networks (DNNs) have shown promising results in brain studies involving multi-channel electroencephalogram (EEG) signals. The representations produced by the layers of a DNN trained on EEG signals remain, however, poorly understood. In this paper, we propose an approach to interpret deep representations of EEG signals. Our approach produces spatio-spectral feature images (SSFIS) that encode the EEG input patterns that activate the neurons in each layer of a DNN. We evaluate our approach using the PhyAtt dataset of multi-channel EEG signals for auditory attention. First, we train the same convolutional neural network (CNN) architecture on 25 separate sets of EEG signals from 25 subjects and conduct individual model analysis and inter-subject dependency analysis. Then we generate the SSFI input patterns that activate the layers of each trained CNN. The generated SSFI patterns can identify the main brain regions involved in a given auditory task. Our results show that low-level CNN features focus on larger regions and high-level features focus on smaller regions. In addition, our approach allows us to discern patterns in different frequency bands. Further SSFI saliency analysis reveals common brain regions associated with a specific activity for each subject. Our approach to investigate deep representations using SSFI can be used to enhance our understanding of the brain activity and effectively realize transfer learning. brain-computer interface; EEG; spatio-spectral feature image; deep neural network; convolutional neural network; deep representation +
Footnote †: journal:  
subjects. However, unlike conventional data such as images of physical objects, speech signals, or text, raw EEG signals do not offer an obvious choice of interpretable features. Common options to define EEG features include spectral domain approaches [16]. By including EEG signals from multiple recording sites, richer spatio-spectral features that account for the spatial activity of the brain can also be defined, as in [17]. Given a set of existing features, new sets of derived features can be defined by adding further processing stages. Using this angle, the resulting processing pipeline can be seen as a system that produces multiple representations of the input EEG signals. Such processing pipelines can be hand-crafted or learned by training a model of a predefined architecture. Deep neural networks (DNNs) are trainable processing pipelines and have become a popular machine learning approach in neuroscience and brain studies. Systematic reviews that focus on the application of DNNs to brain studies can be found in [18; 19; 20; 21]. These reviews discuss the main predictive tasks where DNNs are commonly used, examine their internal architecture, and provide a comparison of the performance of different DNN models. An analysis of the reviewed literature reveals that the majority of the existing proposals consider CNN architectures that take as an input one-dimensional data structures, such as temporal EEG segments or spectral domain representations of EEG signals, or two-dimensional data structures, for instance, spectrograms or scalograms. In a limited number of cases, spatio-spectral inputs are considered. For instance, spatio-temporal representations of EEG signals were used as input in [22; 23], who used multi-channel EEG signals as the input of their proposed DNN [24]. As processing pipelines, DNNs internally produce multiple representations of their input, which are known as deep representations. A small fraction of the literature reviewed in [18], where DNN approaches were used in brain studies, considered model inspection, which is needed to interpret the sequence of processing stages in a DNN pipeline. For instance, [23; 25] analyzed the weights of a DNN trained on EEG signals. Occlusion and activation maximization have also been used, for instance, in [26; 27; 28; 29; 30; 31].