# The Riemannian spatial pattern method: mapping and clustering movement imagery using Riemannian geometry

Christelle Larzabal, Vincent Auboiroux, Serpil Karakas, Guillaume Charvet, Alim-Louis Benabid, Stephan Chabardes, Thomas Costecalde, Stéphane Bonnet

<figure><img src="image_1.png" /></figure>

## To cite this version:

Christelle Larzabal, Vincent Auboiroux, Serpil Karakas, Guillaume Charvet, Alim-Louis Benabid, et al. The Riemannian spatial pattern method: mapping and clustering movement imagery using Riemannian geometry. Journal of Neural Engineering, 2021, 18 (5), pp.056014. 10.1088/1741-2552/abf291. hal-04693910

HAL Id: hal-04693910

https://hal.science/hal-04693910v1

Submitted on 11 Sep 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

CrossMark

OPEN ACCESS

RECEIVED

14 September 2020

REVISED

19 March 2021

ACCEPTED FOR PUBLICATION

26 March 2021

PUBLISHED

8 April 2021

Original content from
this work may be used
under the terms of the
Creative Commons
Attribution 4.0 licence.

Any further distribution
of this work must
maintain attribution to
the author(s) and the title
of the work, journal
citation and DOI.

<figure><img src="image_4.png" /></figure>

PAPER

# The Riemannian spatial pattern method: mapping and clustering movement imagery using Riemannian geometry

Christelle Larzabal¹ ID, Vincent Auboiroux¹, Serpil Karakas¹, Guillaume Charvet¹, Alim-Louis Benabid¹, Stephan Chabardes³, Thomas Costecalde¹,⁴ and Stéphane Bonnet²,⁴ ID

¹ University Grenoble Alpes, CEA, LETI, Climatec, F-38000 Grenoble, France

² University Grenoble Alpes, CEA, LETI, DTBS, LS2P, F-38000 Grenoble, France

³ CHU Grenoble Alpes, Grenoble, France

⁴ Shared last authorship.

E-mail: stephane.bonnet@cea.fr

**Keywords:** Riemannian geometry, motor imagery, ECoG, tetraplegic, classification

Supplementary material for this article is available online

## Abstract

**Objective.** Over the last decade, Riemannian geometry has shown promising results for motor imagery classification. However, extracting the underlying spatial features is not as straightforward as for applying common spatial pattern (CSP) filtering prior to classification. In this article, we propose a simple way to extract the spatial patterns obtained from Riemannian classification: the Riemannian spatial pattern (RSP) method, which is based on the backward channel selection procedure. **Approach.** The RSP method was compared to the CSP approach on ECoG data obtained from a quadriplegic patient while performing imagined movements of arm articulations and fingers. **Main results.** Similar results were found between the RSP and CSP methods for mapping each motor imagery task with activations following the classical somatotopic organization. Clustering obtained by pairwise comparisons of imagined motor movements however, revealed higher differentiation for the RSP method compared to the CSP approach. Importantly, the RSP approach could provide a precise comparison of the imagined finger flexions which added supplementary information to the mapping results. **Significance.** Our new RSP method illustrates the interest of the Riemannian framework in the spatial domain and as such offers new avenues for the neuroimaging community. This study is part of an ongoing clinical trial registered with ClinicalTrials.gov, NCT02550522.

## 1. Introduction

The primary motor cortex (M1) located on the pre-
central gyrus is organized into spatially distinct territ-
ories that are specialized for controlling specific body
regions [1]. This somatotopy is well depicted in the
Penfield's motor homunculus representation and can
be used to differentiate the motion of different body
areas. For example, tongue, hand and foot move-
ments can easily be differentiated from each other on
ECOG or EEG signals [2–4]. This somatotopy how-
ever is less clear for fingers. Evidence from fMRI stud-
ies shows that different fingers are controlled by over-
lapping cortical areas distributed throughout the M1
hand region with no strong somatotopic arrangement
[5–8]. This lack of clear segregation contrasts with the

somatotopic arrangement observed in the primary
somatosensory cortex (S1) in the postcentral gyrus
[9–11]. Numerous studies have shown that decoding
individual finger flexions or more complex gestures
obtained from specific finger configurations such as
grasp actions was possible using ECoG data [12–18]
or scalp EEG data [19–21]. These studies showed that
these motor differentiations are mainly supported
by the high frequencies of the time-resolved signal:
from 30 Hz and above, in the so-called gamma band.
Event-related synchronizations (ERSs) are observed
in the gamma band prior to the execution of move-
ment and contrast with the event-related desynchron-
izations (ERDs) found in the mu (8–12 Hz) and
in the beta (13–30 Hz) bands. Gamma ERS might
reflect the involvement of local cortical ensembles.
This spatial selectivity was particularly observed in the high gamma range (~60–130 Hz) [2, 16, 22].

While imagined movements elicit weaker activations than real motor actions, they share similar activation patterns. This was found in spatially resolved recordings for large body parts [23–25] and fingers alike [26–30]. Extracting these spatial features from time-resolved signal however remains a challenging task. The common spatial pattern (CSP) method was proposed to extract the spatial features and the corresponding spatial patterns from the spatial covariance matrices estimated from the time-resolved signals [31]. Using this technique, the spatial patterns associated to ERD generated from right/left hand motor imagery could be identified [32]. These spatial patterns were valuable signal features as attested by the high decoding performances obtained when the corresponding spatial filters were used in subsequent classification. Spatial covariance matrices are symmetric and positive definite (SPD) matrices and as such, they belong to a Riemannian manifold that should not be approximated using the Euclidean framework [33]. Riemannian geometry has been recently introduced in the context of machine learning applied to EEG data [34–38]. Its ease of implementation together with its high resulting performance make it a powerful mathematical tool to perform the classification of mental tasks [39, 40]. In particular, higher classification scores were obtained for Riemannian classification compared to CSP-based classifiers. One main difference between the two approaches however lies in the physiological interpretability of the results. While the CSP method provides a simple way of visualizing the regions activated during motor imagery, extracting this information is not as straightforward for the Riemannian framework. This could be one of the reasons why the Riemannian framework has not widely reached the neuroimaging community yet. Recent strategies based on regression methods have been proposed to tackle this issue. Spatial patterns were obtained by estimating the coefficients that best matched the EEG and EMG signal using a Canonical Partial Least Square method [41]. Results showed that the same spatial patterns could be extracted from the CSP method and the Riemannian framework. Instead of using the EMG signal as a reference, Xu et al proposed to fit a linear regression on the weights obtained from linear classification in the tangent space before the weights were back projected onto the Riemannian space [42, 43]. Compared to the CSP method the resulting patterns were less prone to artifacts and could extract additional neurophysiological activity.

Here, we propose a new method for extracting spatial patterns from the brain time-series data by taking advantage of the Riemannian framework: the Riemannian spatial pattern (RSP) method. Our proposal relies on the backward channel selection procedure which was first proposed as a way of

selecting the most relevant channels [44]. Within this framework, the spatial pattern values correspond to the maximal distance value attributed to each channel when maximizing the distance between two covariance matrices. As such, and compared to the previous studies, our method makes a direct link between Riemannian distances and spatial location. Importantly, these distance values can be directly used for mapping the brain regions that are activated during the imagination of a specific movement as well as for the clustering of these different movements. Indeed, Riemannian distances have been shown to provide an efficient way for the clustering of neural representations [45].

In this study, the RSP approach was compared to the CSP method on imagery tasks realized by the quadriplegic patient who is included in our clinical trial [46]. Imagined movements of arm articulations and fingers which were recorded during this long-term ECoG implantation were analyzed. We expect that the patterns obtained from the RSP method would match the ones obtained from the CSP method and could provide even further details.

# 2. Material and methods

## 2.1. Pattern extraction and classification

Spatial covariance matrices were estimated from the brain time-resolved signals and used as input features for pattern extraction and classification. The number of trials was equalized between each class. For pattern extraction, pairwise comparisons were performed on all the trials. Binary and multi-class classification was performed using a ten-fold cross-validation procedure [47].

### 2.1.1. CSP + SVM

CSP is classically used as a spatial filter to enhance the discrimination of two conditions [32]. Spatial filtering follows the equation: $Z = W^T X$ where $X$ corresponds to the ECoG signal (channel × trial × time). The projection matrix ($W$) obtained from CSP filtering provides information about the spatial patterns ($P$) that arise for a given filter by the relation: $P = W^{-T}$ where the columns of $W$ are the spatial filters. The four highest filters were selected. The log variance of these spatially filtered signals were used as the input features for the SVM classification. Linear SVM classification was performed using the standard SVC algorithm from the Scikit-learn python package [48] with its default regularization constant ($C = 1$).

### 2.1.2. Riemannian approach

Spatial covariance matrices are SPD matrices that lie in the Riemannian manifold. Riemannian distances were therefore used to estimate the centroid of each covariance matrix while preserving this geometric
nature. Once estimated, the mean of these covariance matrices can be used for pattern extraction and classification.

### 2.1.2.1. RSP procedure

The Riemannian distance between the two centroids corresponds to the maximal distance value that differentiates the two class of labels when using all the electrodes (nelec) and before starting the backward iteration process (at step = 0). To associate a Riemannian distance for each electrode, the RSP procedure relies on the backward iteration process initially used for channel selection [44]. At each iteration step, the electrode which had the minimum contribution in the distance maximization between the covariance means is discarded from the channel set (nelec-1). The maximal distance value obtained with these nelec-1 electrodes corresponds to the maximal distance value at step = step + 1. When removed, the maximal Riemannian distance value calculated at step-1 is assigned to the electrode. This backward iteration process continues until all electrodes were removed from the set of channels. For each pairwise comparison, electrodes associated with the highest distance values indicate that they were removed early in the selection process. Electrodes with the smallest distance values were removed at the end of the selection process meaning that they played an important role in the discrimination between the two classes. Similarly to the CSP method and because it is an extension of the Riemannian-based channel selection procedure, the RSP method can also be used as a spatial filter by keeping the most relevant channels before classification in the native space or in the tangent space.

### 2.1.2.2. Classification in the native and in the tangent spaces

The mean covariance matrices of the different classes (centroids) are estimated from the training set and can be used to perform Riemannian classification in the native space or in the tangent space. Both approaches are now well covered in the literature by the pioneering work of Barachant et al [34, 35].

Classification in the native space consists in measuring the Riemannian distance between each covariance matrix from the test set and the centroids estimated from the training set. Following the Minimum Distance to Riemannian Mean (MDM) method, for each test covariance matrix the label of its closest centroid is assigned.

High-performing algorithms such as the support vector machine (SVM) are applied in the Euclidean space and cannot be implemented in the native Riemannian manifold. To do so, it is necessary to map the spatial covariance matrices onto the Euclidean tangent space located at the Riemannian mean. This enables the transformation of the spatial covariance matrices into a vectorized form that can be used

for SVM classification (tsSVM) [36]. Riemannian classification was performed using the pyRiemann python package [49].

### 2.1.3. Pattern extraction and representation

Following the RSP procedure described in section 2.1.2.1, the Riemannian distance value assigned to each electrode was used to plot the RSPs. For the CSP method, the patterns were estimated for each filter from the whole data set used prior to classification. The absolute values of the patterns of each filter were averaged to extract the corresponding spatial patterns.

#### 2.1.3.1. Mapping

To map activations elicited by a specific imagined movement, pairwise comparisons between a motor imagery condition and the idle state were performed (e.g. left elbow/idle). The average topographical representations over the sessions obtained from the RSP and the CSP methods offered a direct view of the electrodes that were activated during the imagination of the movement.

#### 2.1.3.2. Representational similarity analyses and clustering

Pairwise comparisons involving two imagined movements (e.g. left elbow/left wrist) were used to create dissimilarity matrices from the spatial patterns. Following the RSP and CSP methods, each electrode was associated to a given value. By taking the maximal value of the electrodes, one can attribute a single value to each pair of movements which can be used to measure a dissimilarity index for each movement pair. For the RSP procedure, these values correspond to the maximal Riemannian distance measured at step = 0, obtained before the backward iteration process. For the CSP method, this dissimilarity index corresponds to the maximal absolute value across the four filters. Assembled together, the dissimilarity measures obtained from each pairwise comparison create a symmetric dissimilarity matrix [50]. The square form of these dissimilarity matrices was used to estimate the hierarchical clustering of the imagined movements.

One advantage of the RSP method resides in the fact that each electrode is assigned to a distance value which measures the differentiation between two classes. This can be used to plot the dissimilarity matrices for each step of the backward iteration process. The last step is particularly informative as it provides the pairwise distances for the electrodes which had the most dissimilarity.

Representational similarity analyses were also performed on the classification performances. Pairwise classification performances were used to create symmetric dissimilarity matrices that reflect the dissimilarity between each pair of movements. For
<figure><img src="image_5.png" /><figcaption>**Figure 1.** (A) Co-registration of the two WIMAGINE implants on the patient's cortical surface. The 64 electrodes recorded during the sessions are shown in white with their respective labels. (B) Experimental design. The patient was instructed to imagine moving a specific upper-body area every time a red sphere was displayed. Three motor imagery tasks were performed per session. For the ARM sessions, elbow flexion, wrist extension and hand grasping of the left or right arm were performed. For the FINGER sessions, imagined finger flexions of the thumb (F1), index finger (F2) and middle finger (F3) of the left or right hand were executed. The imagined movement to be performed depended on the location of the cue (left, center, right). Each target presentation was separated with an interval ranging from 2 to 5 s.</figcaption></figure>

multi-class classification, this dissimilarity index was reflected in the resulting confusion matrices.

## 2.2. Experimental setup

### 2.2.1. *Ethics statement*

This study is part of an ongoing clinical trial registered with ClinicalTrials.gov, NCT02550522. This clinical protocol was approved by the local Ethical Committee (CPP, n°: 15-CHUG-19) and the French National Agency for Medicines and Health Products Safety (ANSM, RCB n°: 2015-A00650-49). In accordance with the Declaration of Helsinki, the patient provided written informed consent (by proxy) prior to starting the clinical trial.

### 2.2.2. Participant

The quadriplegic, right-handed patient considered in this article was included in a long-term implantation BCI ongoing study that started on the 12 June 2017 [46]. The patient was a 28 year old French man at the time of the surgery who suffered from tetraplegia caused by spinal lesion at the C5–C6 level. Following the American Spinal Injury Association Impairment (ASIA) score, residual volitional movements of

his two elbows and of his left wrist were still possible: elbow motion (score left = 5, score right = 4) and left wrist motion (score = 3). Two WIMAGINE implants [51] both consisting in an epidural wireless 64-electrode recorder were placed on the two hemispheres of the patient's upper limb sensorimotor cortex. Using these implants, the patient's epidural ECoG activity was recorded in real-time with a sampling rate of 586 Hz. Due to data transfer limitations only half of the channels were transmitted in a checkerboard-like pattern: 64 channels (32 by implant) were therefore recorded per session. Co-registration of the implant on the patient's cortical surface was obtained using the FieldTrip toolbox [52] to provide a precise mapping of the 64 channels (figure 1(A)).

### 2.2.3. Cue-based task

The patient was seated in his wheelchair in front of a computer screen. The patient was instructed to imagine moving a specific upper-body part every time a red sphere appeared on the screen. Three different sphere locations corresponding to three different upper-body areas were proposed. Prior to the experimental session, the patient was informed about the
different spheres' locations and the imagined movements to perform respectively. Eight sessions were performed between July 2017 and September 2018 (from 1 month to 1 year and 3 months post-surgery) and are considered in this article. Four sessions were performed in July 2017 and are referred to as ARM sessions as they involved motor imagery of the same arm articulations with two sessions involving the right arm and two sessions involving the left arm. During these ARM sessions, the three motor imagery tasks were: one/elbow flexion, two/wrist extension and three/hand grasping. The other four sessions were performed in September 2018. For these four sessions, referred to as FINGER sessions, the patient was asked to perform finer imagined movements which consisted in flexions of individual fingers. These FINGER sessions involved flexions of one/the thumb (F1), two/the index finger (F2) and three/the middle finger (F3) of the left hand (two sessions) and the right hand (two sessions). The visual cue was displayed on the screen for two seconds for the ARM sessions and for one second for the FINGER sessions. The interval between two cues varied over the trials and could range from 2 to 5 s (figure 1(B)). The duration of each experimental session was about 20 min. The experiment was programmed using the Unity software and C++ custom-made codes.

For the ARM sessions, 150 trials were obtained with 50 trials corresponding to each specific motor imagery task. For the FINGER sessions, 200 trials were recorded with 80 trials for the thumb and for the index finger and 40 trials for the middle finger. One-second epochs during the inter stimulus interval which separated two consecutive motor imagery tasks, were considered as idle trials. The half time value separating the presentation of two cues corresponded to the time point $t = 0.5$ s of the idle trials. This was done to make sure that idle trials were not contaminated by activations originating from the imagined conditions. The number of idle trials was therefore equal to the sum of the number of motor imagery trials performed by the participant during a session (e.g. 200 trials for the FINGER sessions). All the trials were included in the analysis.

### 2.2.4. Functional analyses

#### 2.2.4.1. Correlation analyses

Correlation analyses were performed to identify the electrodes that were the most involved during specific imagined movements. The ECoG signal of the full session was considered. For each movement, the signal could be divided into a motion state when the visual cue was displayed on screen and an idle state during the inter-stimulus interval. For each electrode implant, a complex Morlet wavelet filter bank was used to extract power waveforms. The filter bank split the signal into 40 frequency bands from 1 to 200 Hz with an increase of 5 Hz between each center frequency. A scaling function was used in association

to the filter bank to cover the whole spectrum. The obtained power values were smoothed using a convolution kernel (100 ms-window) and used to calculate the correlation coefficients with the two-state movement values.

#### 2.2.4.2. Time-frequency maps

The channels with the highest and lowest correlation activities in the 1–200 Hz frequency range were selected for every imagined movement. For each trial within a specific imagined movement type, time-frequency analyses were performed between –0.5 s and +2 s post-visual cue onset. The ECoG signal was split every 5 Hz from 5 to 200 Hz with a filter bandwidth of 2 Hz. A fourth-order Butterworth, IIR filter was used for these 40 frequency bands to extract the power values at each time point. From $t = –0.5$ s to $t = 0$ s post visual cue onset, the average power over all the trials was calculated for each frequency band and used as a reference power. The ratio (in dB) between the power values obtained at each time point and the reference power was computed for each trial and each frequency band. The difference between the trial averaged time-frequency maps associated to the electrodes with the highest and lowest correlation values was computed and averaged over the movements. Cluster-based permutation tests were used to assess time-frequency significance. Permutation ($n = 400$) was performed by randomly assigning each trial to the electrode with the highest or lowest correlation value. For each permutation, the trials for the electrode with the highest/lowest correlation value were averaged and their difference computed as before. This procedure was repeated for each movement. The empirical null distribution obtained was then compared to the averaged time-frequency maps. To correct for multiple comparisons, cluster-size statistics were performed. Clusters were considered significant if their size was above the 95% confidence interval (one-tailed, $p < 0.05$) of the cluster size values obtained at the chance level. These chance level cluster size values corresponded to the maximal cluster size found when comparing the randomized time-frequency maps of each permutation to the empirical null distribution.

## 3. Results

The RSP and CSP methods were tested on the ECoG activity of a quadriplegic patient who performed motor imagery of 12 upper-limb movements. Prior to applying these methods, functional analyses were performed to select the appropriate time-frequency range.

### 3.1. Time-frequency range of interest

For each imagined movement, differential time-frequency analyses (epoch: –0.5 to +2 s post-visual cue onset, frequency-range: 5–200 Hz) were performed on the electrode with the highest and lowest
<figure><img src="image_6.png" /></figure>

average correlation score obtained in the 1–200 Hz frequency range over the two sessions (figure 2(A)). The electrodes with the highest average correlation score (in red) were found in the contralateral hemisphere to the imagined movement except for the left middle finger (left F3) for which highest correlation activations were observed at the L03 electrode in the left hemisphere. The location of the electrodes with the lowest average correlation score (in blue) was found in the ipsilateral hemisphere for the left elbow, left wrist, left index finger and right thumb and in the contralateral hemisphere for the right elbow, right wrist, left/right hand, left thumb, left middle finger, right index finger and right middle finger. The differential maps observed for each movement show positive activations in the frequencies above 50 Hz which are particularly visible during imagined movements of the elbow, wrist and hand. For more details, the time-frequency maps obtained for the electrodes with the highest and lowest correlation scores are shown in the supplementary figure S1 (available online at stacks.iop.org/JNE/18/056014/mmedia). Statistical analyses performed on average over the 12 movements showed that significant positive time-frequency values (one-tail, *p* < 0.05, cluster-corrected) were found in the 60–130 frequency range from 0.5 to 2 s post-stimulus onset (figure 3(B)). According to this information we decided to apply classification on the ECoG signal

band pass filtered between 60 and 130 Hz. A 1 s time window was used in relation to the visual cue presentation.

## 3.2. Pairwise motor-idle comparison

Two classes were considered: one class involving motor imagery of a specific upper-body part and one class corresponding to the idle state. The RSP procedure was compared to the CSP method (figure 3).

### 3.2.1. Mapping

The spatial patterns obtained from the Riemannian procedure showed that the electrodes with the lowest distances which best differentiated motor imagery from idle states were all located contralateral to the movement side (figure 3(A)). On average across the 12 movements, a mean distance value of 1.22 (SD = 0.11) was observed for the electrodes in the contralateral hemisphere to the imagined movement which was significantly smaller than the mean distance of 1.39 (SD = 0.07) for the electrodes that were ipsilateral to the imagined movement (U = 623, *p* < 0.001). For the right elbow, and right wrist, the location of the respective L41 and L23 electrodes showed a shift of activation toward the lateral regions along the left central sulcus as observed in the classical motor somatotopy. For the right hand, the localization on the left central sulcus was more widespread on the central sulcus with activations found
<figure><img src="image_7.png" /></figure>

at the L41, L21 and L03 location sites. This gradual
shift however was less clear when considering the
electrodes that were the most involved during ima-
gined movements of the left elbow: R52 and the left
wrist: R30 although left hand activations were found
in the lateral part of the right hemisphere at the R14
electrode. For imagined finger movements the elec-
trodes associated to the lowest distances were all loc-
ated in a closed-up area: electrodes R14 and R23 for
the left fingers and electrodes L03 and L14 for the
right fingers. Interestingly, these electrodes were also

the ones that were involved during imagined move-
ments of the left hand and right hand respectively.
The spatial patterns obtained with the CSP method
revealed similar topographies where the electrodes
which best differentiated motor imagery from idle
states were associated to the highest absolute val-
ues (figure 3(B)). For each movement, the electrode
with the highest absolute value was found in the
contralateral hemisphere to the imagined movement
and the mean absolute value over the electrodes in
the contralateral hemisphere was significantly higher
<figure><img src="image_8.png" /><figcaption>**Figure 4.** Binary classification performances obtained for each motor imagery type against the idle state using the (MDM(-32) and tsSVM(-32)) Riemannian approaches and the CSP + SVM method. The corresponding chance level performance is shown with the grey dots.</figcaption></figure>

than in the ipsilateral hemisphere (mean contralateral = 0.37, SD = 0.15, mean ipsilateral = 0.10, SD = 0.03, *U* = 1525, *p* < 0.001). Strong negative correlations (mean *R* = -0.60, SD = 0.15) between the patterns obtained from the RSP and the CSP methods were observed in the contralateral hemisphere with a correlation coefficient ranging from -0.26 for the Right F3 to -0.8 for the left elbow. Smaller correlation values were found for the finger movements (*R* = -0.53, SD = 0.15) compared to the arm movements (*R* = -0.67, SD = 0.13) but this difference was not significant (*U* = 30, *p* = 0.18).

### 3.2.2. Classification

By identifying key brain regions the CSP and the RSP approach can act as a filter that can be applied before classification. For the CSP method the four highest spatial filters were selected before the SVM classification. For the Riemannian approach, classification was performed in the Riemannian space (MDM) and in the tangent space (tsSVM) with no channel selection or with selecting the best 32 channels: MDM-32 and tsSVM-32. High classification performances were obtained using the Riemannian framework and the CSP-based method when each imagined movement was compared to the idle state (figure 4).

On average over the 12 movements, the obtained scores were: 85.59% (SD = 2.57), 88.46% (SD = 1.93) and 83.43% (SD = 4.23) for the MDM, tsSVM and the CSP + SVM methods respectively. Similarly, high results were observed for the Riemannian framework when performing the analysis on the 32 electrodes that were the most activated with 85.96% (SD = 2.65) and 88.48% (SD = 2.35) of correct classification for the MDM-32 and the tsSVM-32 methods respectively. These methods did not differ significantly from each other but were significantly

higher than the 48.73%, SD = 0.95 chance level performance obtained when shuffling the labels of the data (X²(66, *N* = 12) = 43.05, *p* < 0.001, post-hoc tests).

When each motor imagery was compared to the idle state, the similarity of patterns observed from the RSP and the CSP approaches was reflected in the high classification scores obtained by the two methods. In the next section, we will test how these methods perform for differentiating the 12 imagined movement types.

### 3.3. Representational similarity analyses for motor imagery of the 12 movements

The RSP and CSP procedures were applied for pairwise comparison involving two motor imagery conditions. The two recording sessions were pooled together for each condition. For each pairwise comparison, the highest distance value obtained by the RSP method and the maximal absolute value obtained from the CSP methods were used to compute the symmetric dissimilarity matrices. These dissimilarity matrices were used to estimate the hierarchical clustering between the 12 movements. The lower triangular dissimilarity matrices obtained from these two methods are shown in figure 5(A) with their corresponding clustering. For the RSP method the highest dissimilarity cluster corresponded to the differentiation between imagined arm movements (arm cluster) and imagined finger movements (finger cluster). The second highest cluster was found for differentiating imagined right arm movements (right arm cluster) from imagined left arm movements (left arm cluster). Imagined movements of the elbow, wrist and hand were more easily differentiated from each other in the right arm cluster than in the left arm cluster. Small differentiation was also found
<figure><img src="image_9.png" /></figure>

within the finger cluster between the left fingers and the right fingers. The clustering obtained from the CSP approach however was different from the RSP method ($r(s) = 0.65, p < 0.001$). The cluster with the highest dissimilarity was found between the left arm movements (left arm cluster) and the rest of the movements. The second highest dissimilarity group was observed within this remaining large cluster and differentiated the right arm movements from the

fingers. The third highest dissimilarity cluster was found within the left arm cluster and corresponded to the differentiation between the left elbow and the other two arm movements.

Pairwise dissimilarity was also assessed from the pairwise classifier performances where high scores are associated to high dissimilarities (figure 5(B)). High dissimilarity scores were obtained for the different methods over the 66 comparisons: 95.50%
(SD = 9.03), 95.14% (SD = 9.98), 92.70% (SD = 11.80), 85.65% (SD = 12.46) and 84.21% (SD = 13.00) of correct differentiation were obtained for the tsSVM, tsSVM-32, CSP + SVM, MDM and MDM-32 respectively. The scores obtained with the MDM(-32) approach were significantly lower than the ones obtained from the CSP + SVM and the tsSVM(-32) classifiers ($X^2(325, N = 66) = 78.19, p < 0.001$, post-hoc tests). Interestingly, when comparing the results from the tsSVM(-32) dissimilarity matrices and the RSP dissimilarity matrix, perfect differentiation was observed for Riemannian distances above 2.

The dissimilarity between the imagined movements was finally tested by performing multiclass classification on the 12 imagined movements and extracting the corresponding confusion matrices. The confusion matrices generated from the MDM(-32), tsSVM(-32) and CSP + SVM methods are shown in figure 5(C). The lowest average diagonal performances were obtained by the CSP + SVM approach with a score of 28.43% (SD = 13.60). This was significantly lower than the performances obtained from the tsSVM: 64.81% (SD = 16.31) and tsSVM-32: 55.23% (SD = 19.90) methods ($X^2(55, N = 12) = 22.57, p < 0.001$, post-hoc tests). Post-hoc analyses also revealed that with a score of 33.23% (SD = 18.43) the MDM-32 method was significantly different from the tsSVM score. This was not the case for the MDM with a performance of: 55.23% (SD = 19.90). Importantly, Riemannian classification performance for fingers remained high with an average diagonal performance of 58.06% (SD = 7.09), 45.43% (SD = 7.53), 60.15% (SD = 6.58) and 43.03% (SD = 11.81) for the MDM, MDM-32, tsSVM and tsSVM-32.

Four clusters corresponding to left arm movements, right arm movements, left fingers movements and right fingers movements could be clearly identified from the tsSVM(-32) confusion matrix. This means that these four groups of movements were clearly separated by the tsSVM classifiers but some errors were made for differentiating movements within each of these groups. For the MDM(-32) confusion matrix, two clusters differentiating the right fingers from the left fingers were found. However, some errors were observed for correctly differentiating the right arm movements. For the CSP + SVM confusion matrix, only two main clusters were visible: one composed of the arm movements and the other one composed of the finger movements.

Importantly, representational similarity analyses can take advantage of the RSP procedure by computing the dissimilarity matrices at each step of the backward iteration process. The dissimilarity matrix obtained for the electrodes that were removed at the end of the process are shown in figure 5(D). With this representation, each entry of the symmetric dissimilarity matrix is now associated with a distance

value and an electrode name. No specific information could be extracted for the clusters that were highly dissimilar such as the arm vs fingers clusters and the left arm vs right arm clusters, with the same three electrodes: L70, R10, R01 involved between the different movements. However, specific channels were found when comparing movements within these clusters and especially for the finger flexions. This provides additional information to the mapping dynamic observed for the fingers. In particular, the RSP dissimilarity matrix showed that for the R14 electrode, the highest differentiation was found for the Left F1-Left F2 pair and the Left F1-Left F3 pair. For the L03 electrode, the highest differentiation was found for the Right F1-Right F2 pair and the Right F1-Right F3 pair.

## 4. Discussion

In this article we presented a new method that can easily extract spatial patterns using the Riemannian framework: the RSP method. ECoG data from imagined upper-limb movements performed by a tetraplegic were used to test this method. Although the instructions were clearly about performing mental imagery, we cannot fully assert that the mental task was never associated with some muscle activity. This is possible for the right and left elbows for which the patient had some residual volitional functions but seems unlikely for the flexion of individual fingers. Compared to the CSP method, we observed similar patterns when mapping each motor imagery type: contralateral activations were found for the 12 upper-limb movements, imagined hand movements and finger flexions involved similar lateral regions of the central sulcus, and activations of the right elbow and right wrist showed a medial-to-lateral gradient along this sulcus. The high correlation between the RSP and CSP patterns as well as the coherence of the observed activations with the classical motor somatotopy provide strong evidence in supporting our new RSP method. The relevance of these spatial features was also directly observed from the high performances (~86%) obtained from the Riemannian classifiers and the CSP + SVM method in decoding each motor imagery type with the idle state.

Interestingly, the pairwise comparison of imagined movements revealed a more detailed clustering for the RSP method compared to the CSP approach. While the RSP method clearly identified a hierarchy of clusters: one/arm vs finger, two/left arm vs right arm, three/left fingers vs right fingers, for the CSP method this was less clear with the highest cluster differentiating the left arm from the rest of the movements. Despite this, high performances were still observed for the CSP + SVM method with similar high results (93%) compared the tsSVM method (95%) for pairwise classification. However, the drop in performance could be clearly observed from the
12-class classification. With a correct classification of 28% along the diagonal of the confusion matrix, the score of the CSP + SVM method was significantly lower than the 65% of the tsSVM method. Clustering information could be extracted from these confusion matrices. Four distinct groups were revealed using the tsSVM approaches: left arm, right arm, left fingers, right fingers whereas only two main clusters which differentiated the arm movements from the finger movements could be observed from the CSP + SVM classifier. Interestingly, the dissymmetrical nature of these confusion matrices could be observed from the MDM confusion matrices with a bias in predicting imagined right arm movements when left arm movements were imagined but not the opposite. Clustering from confusion matrices however is limited by the fact that groups which were found to be highly dissimilar by the classifier would not show up, as no error between the two classes would be made. This phenomenon is clearly observed for the arm vs finger group. While this cluster was missed by the tsSVM approach, it could be identified from the errors made by the CSP + SVM approach. As recently demonstrated, our results show that Riemannian distances provide a more precise clustering [45]. Importantly, here, this clustering benefited from our RSP method as a specific channel location was assigned to each entry of the dissimilarity matrix. For highly dissimilar clusters, this channel location was not very specific and could be explained by the fact that many channels were similarly involved in the differentiation of the imagined movements. For more closely-related movements such as the fingers of the same hand, this channel location was more specific and could provide more information about the difference between the two movements. While in this article we just presented the dissimilarity matrix obtained with the electrodes that were removed at the end of the RSP process, this could be done similarly with the electrodes removed at each step of the backward iteration process.

As for the CSP method, the RSP approach can act as a spatial filter that can be easily implemented. Importantly, source separation is a required step to extract the spatial patterns from the CSP method. For the RSP method, spatial patterns are directly estimated in the sensor space for which no such assumption is made. Because the Riemannian distances obtained in the sensor space are similar to the Riemannian distances in the source space, source activations can be then directly inferred. Following the RSP spatial filtering, a direct link between the electrodes that remain and their corresponding Riemannian distance values can be made. This provides an easy way of manipulating the parameters prior classification. In this article, the parameters were kept fixed at 32 electrodes for the Riemannian approach and at four filters for the CSP-based classification. From the multiclass classification

results, we noticed that the reduction to 32 channels was associated with a significant drop in performance for the MDM method and lower scores for the tsSVM method. Having perfect control of these parameters and tracking their effects is therefore crucial.

While a large number of studies were able to classify imagined movements of large body areas in healthy participants [4, 53–56] and in paralyzed patients [53, 57–59], decoding mental imagery of individual finger movements is not widespread in the literature. Recent studies investigated this topic using several classification approaches such as: artificial neural networks [60, 61], linear discriminant analysis [62], SVM [63, 64] and sparse representation-based classifications [65]. These EEG studies showed that above-chance level performance could be obtained from healthy participants. For paralyzed patients, studies have shown that patients were able to command grasp movements in naturalistic settings [66–68]. This was achieved using BCI systems connected to intracortical microelectrode arrays that had direct access to single neuronal activity. In our study the multiclass tsSVM performances were high for the 12 upper-limb movements, including the imagined finger movements with a ~60% correct classification. This is an encouraging result in order to use this method in real-time Brain Computer Interface protocols with paralyzed patients. Indeed, BCI protocols based on motor ECoG activity are investigated in clinical research to help severely paralyzed patients to recover some mobility [46, 58, 69]. Within this context, the ability to execute fine-grained motion control such as finger movements could have a beneficial impact on patients’ daily life. Testing our method on a clinical ECoG dataset was motivated by this purpose. Compared to scalp-based EEG, ECoG signals recorded from a grid of electrodes placed epiderally on the cortical surface offer a better signal-to-noise ratio, a better spatial resolution and are almost free from physiological artefacts. Therefore, no cleaning process was performed before applying the CSP and RSP method. This ensured complete transparency and equality when comparing the two methods. An important limitation of this study however resides in the fact that a small dataset of a single patient was used. The next step is to confirm these results on all the patients who will be included in the BCI clinical protocol. Meanwhile, and because we think that this method could benefit the neuroimaging community at large, we decided to provide the python code associated to this method in the supplementary material. For ease of use, the RSP procedure was implemented as an extra class in the existing channelselection.py file from the pyRiemann python package. This new version of the channelselection.py file comes along with a Jupyter notebook that describes the steps used to extract and display the RSP patterns associated to Riemannian classification. The public EEG dataset:
BCI Competition IV dataset IIa [70] was used for
illustration. The original files, the preprocessed files
and the results associated to this EEG dataset are also
included in the supplementary material.

As recently pointed out, most of the BCI protocols
rely on the so-called abstract somatotopy mapping
[71]. This strategy favors imagined movements that
elicit activations in very distinct brain regions des-
pite the lack of a direct link between the ima-
gined movements and the movements actually per-
formed. This technique seems sufficient for a few
degree-of-freedom controls though it might provide
strong limitations when this number increases. In this
context, real-time Riemannian classification which
relies upon the natural somatotopy could be a valu-
able option. Combined with long-term ECoG mon-
itoring it could offer an ideal framework for BCI
training.

Data availability statement

The data generated and/or analysed during the cur-
rent study are not publicly available for legal/ethical
reasons but are available from the corresponding
author on reasonable request.

Acknowledgments

We are very grateful to all the institutions that con-
tributed to the funding of this work: the CEA, the
French Ministry of Health (Grant No. PHRC-15-
15-0124), the Fonds de Dotation Clinatec and the
Fondation Philanthropique Edmon J Safra. We also
acknowledge all the CHUGA medical staff for com-
mitted patient care. We are also grateful to the
anonymous reviewers for their overall feedback.

Author contributions

C L contributed to the methodology of the article,
analyzed the data and wrote the manuscript; V A col-
lected and processed the patient's MRI data; S K pro-
grammed the experimental software; G C managed
the research activity; A L B contributed to the concep-
tualization of the experiment; S C contributed to the
conceptualization of the experiment and performed
the patient's surgery; T C contributed to the concep-
tualization and methodology of the experiment, col-
lected the data and edited the manuscript; S B con-
tributed to the methodology of the article, analyzed
the data and edited the manuscript.

ORCID iDs

Christelle Larzabal ● https://orcid.org/0000-0003-4651-9265

Stéphane Bonnet ● https://orcid.org/0000-0003-3758-4268

References

[1] Penfield W and Boldrey E 1937 Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation *Brain* **60** 389–443

[2] Crone N E, Miglioretti D L, Gordon B and Lesser R P 1998 Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis II. Event-related synchronization in the gamma band *Brain* **121** 2301–15

[3] Miller K J, Leuthardt E C, Schalk G, Rao R P N, Anderson N R, Moran D W, Miller J W and Ojemann J G 2007 Spectral changes in cortical surface potentials during motor movement *J. Neurosci.* **27** 2424–32

[4] Morash V, Bai O, Furlani S, Lin P and Hallett M 2008 Classifying EEG signals preceding right hand, left hand, tongue, and right foot movements and motor imageries *Clin. Neurophysiol.* **119** 2570–8

[5] Ejaz N, Hamada M and Diedrichsen J 2015 Hand use predicts the structure of representations in sensorimotor cortex *Nat. Neurosci.* **18** 1034–40

[6] Sanes J N, Donoghue J P, Thangaraj V, Edelman R R and Warach S 1995 Shared neural substrates controlling hand movements in human motor cortex *Science* **268** 1775–7

[7] Schieber M H 1999 Somatotopic gradients in the distributed organization of the human primary motor cortex hand area: evidence from small infarcts *Exp. Brain Res.* **128** 139–48

[8] Schieber M H and Hibbard L S 1993 How somatotopic is the motor cortex hand area? *Science* **261** 489–92

[9] Besle J, Sanchez-Panchuelo R-M, Bowtell R, Francis S and Schluppeck D 2013 Single-subject fMRI mapping at 7T of the representation of fingertips in S1: a comparison of event-related and phase-encoding designs *J. Neurophysiol.* **109** 2293–305

[10] O'Neill G C, Sengupta A, Ashgar M, Barratt E L, Besle J, Schluppeck D, Francis S T and Sanchez Panchuelo R M 2020 A probabilistic atlas of finger dominance in the primary somatosensory cortex *NeuroImage* **217** 116880

[11] Schweisfurth M A, Frahm J and Schweizer R 2014 Individual fMRI maps of all phalanges and digit bases of all fingers in human primary somatosensory cortex *Front. Hum. Neurosci.* **8** 658

[12] Bleichner M G, Freudenburg Z V, Jansma J M, Aarnoutse E J, Vansteensel M J and Ramsey N F 2016 Give me a sign: decoding four complex hand gestures based on high-density ECoG *Brain Struct. Funct.* **221** 203–16

[13] Branco M P, Freudenburg Z V, Aarnoutse E J, Bleichner M G, Vansteensel M J and Ramsey N F 2017 Decoding hand gestures from primary somatosensory cortex using high-density ECoG *NeuroImage* **147** 130–42

[14] Chestek C A, Gilja V, Blabe C H, Foster B L, Shenoy K V, Parvizi J and Henderson J M 2013 Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas *J. Neural Eng.* **10** 026002

[15] Kubánek J, Miller K J, Ojemann J G, Wolpaw J R and Schalk G 2009 Decoding flexion of individual fingers using electrocorticographic signals in humans *J. Neural Eng.* **6** 066001

[16] Miller K J, Zanos S, Fetz E E, Den Nijs M and Ojemann J G 2009 Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans *J. Neurosci.* **29** 3132–7

[17] Pistohl T, Schulze-Bonhage A, Aertsen A, Mehring C and Ball T 2012 Decoding natural grasp types from human ECoG *NeuroImage* **59** 248–60

[18] Shenoy P, Miller K J, Ojemann J G and Rao R P N 2007 Finger movement classification for an electrocorticographic BCI 3rd Int. IEEE/EMBS Conf. on Neural Engineering (Kohala Coast, HI, USA) (IEEE) pp 192–5

[19] Agashe H A and Contreras-Vidal J L 2013 Decoding the evolving grasping gesture from electroencephalographic (EEG) activity 35th Annual Int. Conf. IEEE engineering in
*Medicine and Biology Society (EMBC) (Osaka, Japan)*
(IEEE) pp 5590–3

[20] Bera S, Roy R, Sikdar D and Mahadevappa M 2019 An ensemble learning based classification of individual finger movement from EEG (arXiv:190310154)

[21] Liao K, Xiao R, Gonzalez J and Ding L 2014 decoding individual finger movements from one hand using human EEG Signals ed W Zhan PLoS One 9 e85192

[22] Pfurtscheller G, Graimann B, Huggins J E, Levine S P and Schuh L A 2003 Spatiotemporal patterns of beta desynchronization and gamma synchronization in corticographic data during self-paced movement Clin. Neurophysiol. 114 1226–36

[23] Decety J 1996 Do imagined and executed actions share the same neural substrate? *Cogn. Brain Res.* **3** 87–93

[24] Jeannerod M 1994 The representing brain: neural correlates of motor intention and imagery *Behav. Brain Sci.* **17** 187–245

[25] Pfurtscheller G and Neuper C 1997 Motor imagery activates primary sensorimotor area in humans *Neurosci. Lett.* **239** 65–68

[26] Beisteiner R, Höllinger P, Lindinger G, Lang W and Berthoz A 1995 Mental representations of movements. Brain potentials associated with imagination of hand movements *Electroencephalogr. Clin. Neurophysiol. Potentials Sect.* **96** 183–93

[27] Caldara R, Deiber M-P, Andrey C, Michel C M, Thut G and Hauert C-A 2004 Actual and mental motor preparation and execution: a spatiotemporal ERP study *Exp. Brain Res.* **159** 389–99

[28] Ehrsson H H, Geyer S and Naito E 2003 Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations *J. Neurophysiol.* **90** 3304–16

[29] Lang W, Cheyne D, Höllinger P, Gerschlager W and Lindinger G 1996 Electric and magnetic fields of the brain accompanying internal simulation of movement *Cogn. Brain Res.* **3** 125–9

[30] Porro C A, Francescato M P, Cettolo V, Diamond M E, Baraldi P, Zuiani C, Bazzocchi M and Di Prampero P E 1996 Primary motor and sensory cortex activation during motor performance and motor imagery: a functional magnetic resonance imaging study *J. Neurosci.* **16** 7688–98

[31] Koles Z J, Lazar M S and Zhou S Z 1990 Spatial patterns underlying population differences in the background EEG *Brain Topogr.* **2** 275–84

[32] Ramoser H, Muller-Gerking J and Pfurtscheller G 2000 Optimal spatial filtering of single trial EEG during imagined hand movement *IEEE Trans. Rehabil. Eng.* **8** 441–6

[33] Bhatia R 2007 *Positive Definite Matrices* (Princeton, NJ: Princeton University Press)

[34] Barachant A, Bonnet S, Congedo M and Jutten C 2010 Riemannian geometry applied to BCI classification *Int. Conf. on Latent Variable Analysis and Signal Separation* (Springer) pp 629–36

[35] Barachant A, Bonnet S, Congedo M and Jutten C 2012 Multiclass brain–computer interface classification by Riemannian geometry *IEEE Trans. Biomed. Eng.* **59** 920–8

[36] Barachant A, Bonnet S, Congedo M and Jutten C 2013 Classification of covariance matrices using a Riemannian-based kernel for BCI applications *Neurocomputing* **112** 172–8

[37] Kalunga E K, Chevallier S, Barthélemy Q, Djouani K, Monacelli E and Hamam Y 2016 Online SSVEP-based BCI using Riemannian geometry *Neurocomputing* **191** 55–68

[38] Li Y, Wong K M and deBruin H 2009 EEG signal classification based on a Riemannian distance measure *IEEE Toronto Int. Conf. Science and Technology for Humanity* (Toronto: TIC-STH) pp 268–73

[39] Congedo M, Barachant A and Bhatia R 2017 Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review *Brain-Comput. Interfaces* **4** 155–74

[40] Yger F, Berar M and Lotte F 2016 Riemannian approaches in brain-computer interfaces: a review *IEEE Trans. Neural Syst. Rehabil. Eng.* **25** 1753–62

[41] Barachant A, Carmel J, Friel K and Gupta D 2016 Extraction of motor patterns from joint EEG/EMG recording: a Riemannian geometry approach *6th Int. Brain-Computer Interface Meeting (Monterey, USA)* p 2

[42] Xu J, Grosse-Wentrup M and Jayaram V 2020 Tangent space spatial filters for interpretable and efficient Riemannian classification *J. Neural Eng.* **17** 026043

[43] Xu J, Grosse-Wentrup M and Jayaram V 2019 Interpretable Riemannian classification in brain–computer interfacing *Proc. 8th Graz Brain-Computer Interface Conf.* (Austria: Verlag der Technischen Universität Graz, Graz University of Technology)

[44] Barachant A and Bonnet S 2011 Channel selection procedure using Riemannian distance for BCI applications *2011 5th Int. IEEE/EMBS Conf. on Neural Engineering* (IEEE) pp 348–51

[45] Barachant A and King J-R 2017 Riemannian geometry boosts representational similarity analyses of dense neural time series *BioRxiv* https://doi.org/https://doi.org/10.1101/232710

[46] Benabid A L et al 2019 An exoskeleton controlled by an epidural wireless brain–machine interface in a tetraplegic patient: a proof-of-concept demonstration *Lancet Neurol.* **18** 1112

[47] Varoquaux G, Raamana P R, Engemann D A, Hoyos-Idrobo A, Schwartz Y and Thirion B 2017 Assessing and tuning brain decoders: cross-validation, caveats, and guidelines *NeuroImage* **145** 166–79

[48] Pedregosa F et al 2011 Scikit-learn: machine learning in python *J. Mach. Learn. Res.* **12** 2825–30

[49] Barachant A 2015 *PyRiemann: Python package for covariance matrices manipulation and Biosignal classification with application in Brain Computer interface*

[50] Kriegeskorte N 2008 Representational similarity analysis—connecting the branches of systems neuroscience *Front. Syst. Neurosci.* **2** 4

[51] Mestais C S, Charvet G, Sauter-Starace F, Foerster M, Ratel D and Benabid A L 2015 WIMAGINE: wireless 64-Channel ECoG recording implant for long term clinical applications *IEEE Trans. Neural Syst. Rehabil. Eng.* **23** 10–21

[52] Oostenveld R, Fries P, Maris E and Schoffelen J-M 2011 FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data *Comput. Intell. Neurosci.* **2011** 1–9

[53] Gu Y, Dremstrup K and Farina D 2009 Single-trial discrimination of type and speed of wrist movements from EEG recordings *Clin. Neurophysiol.* **120** 1596–600

[54] Pfurtscheller G, Neuper C, Flotzinger D and Pregenzer M 1997 EEG-based discrimination between imagination of right and left hand movement *Electroencephalogr. Clin. Neurophysiol.* **103** 642–51

[55] Pfurtscheller G, Brunner C, Schlögl A and Da Silva F H L 2006 Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks *NeuroImage* **31** 153–9

[56] Zhou J, Yao J, Deng J and Dewald J P A 2009 EEG-based classification for elbow versus shoulder torque intentions involving stroke subjects *Comput. Biol. Med.* **39** 443–52

[57] Kübler A, Nijboer F, Mellinger J, Vaughan T M, Pawelzik H, Schalk G, McFarland D J, Birbaumer N and Wolpaw J R 2005 Patients with ALS can use sensorimotor rhythms to operate a brain-computer interface *Neurology* **64** 1775–7

[58] Leuthardt E C, Schalk G, Wolpaw J R, Ojemann J G and Moran D W 2004 A brain–computer interface using electrocorticographic signals in humans *J. Neural Eng.* **1** 63–71

[59] Pfurtscheller G, Guger C, Müller G, Krausz G and Neuper C 2000 Brain oscillations control hand orthosis in a tetraplegic Neurosci. Lett. **292** 211–4

[60] Sonkin K M, Stankevich L A, Khomenko J G, Nagornova Z V and Shemyakina N V 2015 Development of electroencephalographic pattern classifiers for real and
imaginary thumb and index finger movements of one hand
*Artif. Intell. Med.* **63** 107–17

[61] Stankevich L A, Sonkin K M, Shemyakina N V, Nagornova Z V, Khomenko J G, Perets D S and Koval A V 2016 EEG pattern decoding of rhythmic individual finger imaginary movements of one hand *Hum. Physiol.* **42** 32–42

[62] Hayashi T, Yokoyama H, Nambu I and Wada Y 2017 Prediction of individual finger movements for motor execution and imagery: an EEG study 2017 *IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC)* (IEEE) pp 3020–3

[63] Furman D, Reichart R and Pratt H 2016 Finger flexion imagery: EEG classification through physiologically-inspired feature extraction and hierarchical voting 4th Int. Winter Conf. on Brain-Computer Interface (BCI) (Yongpyong: IEEE)

[64] Paul Y and Jaswal R A 2018 Classification of EEG for upper limb motor imagery: an approach for rehabilitation 2018 Fifth Int. Conf. on Parallel, Distributed and Grid Computing (PDGC) (IEEE) pp 346–50

[65] Miao M, Zeng H, Wang A, Zhao F and Liu F 2017 Index finger motor imagery EEG pattern recognition in BCI applications using dictionary cleaned sparse representation-based classification for healthy people *Rev. Sci. Instrum.* **88** 094305

[66] Ajiboye A B et al 2017 Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration *Lancet* **389** 1821–30

[67] Bouton C E et al 2016 Restoring cortical control of functional movement in a human with quadriplegia *Nature* **533** 247–50

[68] Hochberg L R et al 2012 Reach and grasp by people with tetraplegia using a neurally controlled robotic arm *Nature* **485** 372–5

[69] Wang W et al 2013 An electrocorticographic brain interface in an individual with tetraplegia ed S Hochman *PLoS One* **8** e55344

[70] Brunner C, Leeb R, Müller-Putz G R, Schlögl A and Pfurtscheller G 2008 BCI competition 2008-Graz data set A *Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces)* vol 16 (Graz University of Technology) pp 1–6

[71] Degenhart A D, Hiremath S V, Yang Y, Foldes S, Collinger J L, Boninger M, Tyler-Kabara E C and Wang W 2018 Remapping cortical modulation for electrocorticographic brain–computer interfaces: a somatotopy-based approach in individuals with upper-limb paralysis *J. Neural Eng.* **15** 026021