# A Synchronized Hybrid Brain-Computer Interface System for Simultaneous Detection and Classification of Fusion EEG Signals

Authors: Dalin Yang; Trung-Hau Nguyen; Wan-Young Chung
Journal: Complexity
 (2020)

---

## Abstract

Brain-computer interface (BCI) technology represents a fast-growing field of research and applications for disabled and healthy people, which is a direct communication pathway to translate the neural information into an active command. Owing to the complicated headset structure, low accuracies, extended training periods, and nonstationary noises, BCI still has many challenges that should be dealt with for further facilitation of BCI technology use in daily life. In this study, a simplified synchronized hybrid BCI system is proposed for multiple command control by the electroencephalograph (EEG) signals in the motor cortex. This system can detect the single motor imagery (MI) task, single steady-state visually evoked potential (SSVEP) task, and hybrid MI + SSVEP tasks simultaneously (total ten mental tasks) via 2 EEG channels with high accuracy. The fast independent component analysis algorithm is employed to hybrid signals for obtaining clear EEG signals resulting from denoising. Feature extraction is performed by the wavelet transform, which is extracted by the features in the frequency and time domains. Furthermore, a four-layer convolutional neural network (CNN) is used as a classifier to distinguish different mental tasks. Finally, the hybrid MI + SSVEP system with a simple structure achieves a high accuracy of 95.56%. Additionally, the single MI-based and the SSVEP-based BCI system obtain the classification accuracy of 90.16% and 93.21%, respectively. Experimental results indicate that the synchronized hybrid BCI system could achieve multiple command control with a simple structure. In comparison with the single MI-based and the SSVEP-based BCI system, the hybrid MI + SSVEP BCI system shows a stable performance and higher efficiency. The proposed investigation provides a new method for the multiple command control by a hybrid BCI system. Also, the proposed BCI system offers the possibility of friendly utilization for disabled people because of its reliability, ease of use, and simplified headset structure.


---

## Body

## 1. Introduction

Brain-computer interface (BCI) provides a communication pathway to access and understand the neural activity in which the user’s intent is translated for control of the external device, such as computer, assistive applications, and neural prosthetics [1]. Generally, the noninvasive BCI is divided into four types which include functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), magnetoencephalography (MEG) [2], and electroencephalograph (EEG) [3–5]. With the advantage of the temporal resolution (within millisecond), portable, and low cost, EEG is considered to be the most actively used type in BCI [6]. Unfortunately, the accuracy of classification for EEG signals decreases when the control commands increase [4]. Thus, the hybrid BCI is proposed to overcome the disadvantages mentioned above.Currently, the hybrid BCI is defined as the combination of either more than two BCI types or two brain signals with a single BCI type [7, 8]. In the case of the hybrid BCI with multiple types, different types could provide the brain information in various aspects, in which fNIRS offers the information for hemodynamic metabolism, and EEG delivers the message of the neuron spike. However, the limitation of each type is influenced by each other (e.g., time response, portable, and cost). The hybrid BCI system of multiple brain signals with a single BCI type shows superiority due to its same sample rate and device [9, 10]. In this case, the typical brain activities for a single EEG type include motor imagery (MI) [11, 12], steady-state visually evoked potential (SSVEP) [13, 14], and P300 component of the event-related potential (ERP) [15, 16].Hybrid EEG-based BCI systems have been popularly investigated in previous research studies [17]. There are two methods divided based on the process approach of the EEG signals: sequential mode and simultaneous mode. Most of the studies prefer the sequential mode, implying the step-by-step disposal of tasks [3], for instance, Cao et al. [18] and Ma et al. [19] analysed both MI and SSVEP signals from two independent procedures and combined the results to produce a final command for the hybrid system. For increasing classification accuracy and improve information transmission rates, the simultaneous mode is recommended for the hybrid EEG-based BCI system [20].Generally, P300 is induced by the oddball paradigm, which regards to the endogenous potential of individuals. The amplitude of the P300 wave is weak (2–5µV); it usually needs to record several responses to enhance the magnitude. Furthermore, the high-speed stimulation of the flashes easily leads to fatigue. MI is a mental imagining task. It could evoke the event-related desynchronization/synchronization (ERD/ERS) in the sensorimotor brain area [21, 22]. MI is widely used for device control and rehabilitation. SSVEP has shown reliable performance for discrete control techniques and has a reasonable signal-to-noise ratio produced by the fast-repetitive movement of visual stimuli. Therefore, there are a wide variety of studies [23–25] which employ the MI + SSVEP system to alternate the single systems since the MI task is the endogenous mental task that can avoid fatigue caused by long-term gazing via the SSVEP task. Besides that, the hybrid MI + SSVEP system overcomes the limitations of MI, which have fewer commands. Jin et al. [26] mentioned that MI was less apparent when stimuli tasks were added. Later on, Chio et al. [27] illustrated that employing the threshold method can alleviate interference between two brain activities. Therefore, the hybrid MI + SSVEP system is more appropriate for the system, which requires the demands of multicommands, fast response time, and high accuracy.So far, one of the significant challenges for the BCI system is the complex design structure. It makes it challenging to utilize BCI in daily life. A simplified BCI system is essential for eradicating the problem and maintaining a good performance [28, 29]. As shown in the previous literature [28, 30], C3 and C4 electrodes (i.e., the sensorimotor cortex) can record significant characteristics during the motor imagery task. Also, it was reported [31] that SSVEP signals could be measured from whole-brain areas (e.g., occipital cortex, sensorimotor cortex, and frontal cortex). Therefore, in order to acquire the hybrid signals (i.e., MI and SSVEP) from them as less as the possible channel, C3 and C4 are selected for this study. In addition, the simplified BCI system is more requirable for a clear EEG signal and accurate classifier than a multiple-channel system. In comparison with the traditional band-pass filter, independent component analysis (ICA) shows a robust ability to separate fusion signals into their additive independent or source components. As the recent achievement of the classification method (e.g., machine learning and deep learning) in the BCI field, the traditional classifiers (i.e., linear discrimination analysis and support vector machine) explore the good performance [21, 32]. The researchers try to alternate the traditional classifiers (i.e., linear discrimination analysis and support vector machine) by deep learning methods in BCI fields. Especially, convolutional neural networks (CNNs) show a promising advantage for conducting the classification [33, 34].To verify the feasibility of the simplified hybrid BCI system, this study adopts a simultaneous method of acquiring the fusion of MI and SSVEP signals on the two EEG channels. Totally, ten mental tasks could be recognized with high accuracy from three different EEG types. In this study, the FastICA was used to purify and denoise the interference from the fusion hybrid EEG signals. The features in the time and the feature domain were extracted by the wavelet transform (WT) algorithm. CNN was applied for classifying the different mental signals. To the best of authors’ knowledge, this study is the first of its kind, detecting simultaneous MI and SSVEP hybrid signals from the same channel based on optimized algorithms (i.e., FastICA, WT, and CNN). Compared to simple hybrid simultaneous BCI systems, the proposed system detects three different EEG types in the same CNN model by utilizing a simplified BCI structure (i.e., two channels) for multiple command (i.e., ten comments) control. This investigation verifies the feasibility of the simplified headset structure, which could be applied for the multicommand BCI control system in daily life for the disability.


## 2. Methods

### 2.1. System Parameters

Keeping with regulations and guidelines established by the local ethics committee, a total of 6 healthy adults attended this experiment, including four females and seven males, aged 23 to 38 years. Participants gave written consent before the experiments. EEG signals were recorded from the central part of the head using two Ag/AgCI electrodes supported by a high-density wireless EEG headset (Cognionics Inc., San Diego, USA). The electrode distribution and the channel number are shown in Figure1. The impedance of all electrodes was kept below 5 kΩ, sampled at 500 Hz, and the band-pass was filtered between 1 and 50 Hz. Subjects were seated about 50 cm in front of the stimulation.Figure 1
EEG electrode distribution in accordance with the 10–20 EEG system. The electrodes used for acquisition are marked.


### 2.2. Experiment Paradiagm

A 21 inch light-emitting diode monitor with a 1920 × 1080 resolution and a 60 Hz refresh rate was used as the stimulator. Four different flickering frequencies (i.e., A: 6.6 Hz, B: 7.5 Hz, C: 8.57 Hz, and D: 9.6 Hz) were employed as a stimulus. The experiment involved ten runs for each task (i.e., A, B, C, D, A and MI, B and MI, C and MI, D and MI, idle, and MI task). Subjects were asked to sit in a comfortable chair and were instructed to follow the experimental procedure corresponding to the cues mentioned in Figure2. The fixation period lasts 0–3 s. After 3 s, the subject executes a task based on the cue (i.e., randomly display). Each task includes ten trials. If the screen shows an MI task cue, the subject must imagine he or she is grasping his or her right hand for 5 s. If the interface shows the SSVEP cue, the participants should look at the stimulus flicker window and keep his or her body under a calm state. Then, If MI and SSVEP cues are shown on the screen, the subject must execute both tasks at the same time (i.e., thinking as if he is grasping his right hand and looking at the specific stimulator). After the 5 s execution window, users were given 2 s to relax after each run. Each task totally took 8 s. In the SSVEP tasks, the screen shows four different windows with different frequencies marked. The subjects were asked to look at a specific stimulator for each case (i.e., frequency).Figure 2
Graphical user interface for the hybrid BCI system. (a) The caption EEG tasks shows the 10 mental tasks to be performed, which include idle state, MI task, SSVEP task, Freq. A-SSVEP task, Freq. B-SSVEP task, Freq. C-SSVEP task, Freq. D-SSVEP task, MI + Freq. A-SSVEP task, MI + Freq. B-SSVEP task, MI + Freq. C-SSVEP task, and MI + Freq. D-SSVEP task. (b) Time window for the performance of tasks divided into 2 parts for clarification, 0–3 s (rest state time) and 3–8 s (task-performing window). (c) Signal acquiring from the above performed tasks. (d) Processing, extraction, and classification of signals.After the recording of the signals, the further processing steps are highlighted in Figure3. Each mental task signal was processed, extracted, and classified. The procedure for signal data analysis is listed as follows:(i)
Preprocess 1: used a band-pass filter (5–25 Hz) to acquire the Mu band (8–13 Hz) and SSVEP signals (6.6–9.6 Hz). Mu band was used as it is efficient for detecting the MI tasks [16]. Band-pass filter frequency range covers all the required frequencies; it helps to remove the noise and provide the desired feature.(ii)
Preprocess 2: FastICA was employed to extract pure EEG signals as this method is fast and effective in obtaining the independent source signals from the fusion neural signals.(iii)
Extract feature: the features of time and frequency domains were analysed by the WT since the WT could extract both time and frequency domains simultaneously. In comparison with the FFT, FFT only shows the feature in the frequency domain, which might lead to the loss of the feature in the time domain.(iv)
Classifier: extracted features were the input to train the CNN model. CNN was employed for classification as it can classify multiple features by shared kernel parameters.Figure 3
Schematic architecture of the experiment setup for the hybrid BCI system with MI and SSVEP.


### 2.3. Fast Independent Component Analysis

In this study, we employed the FastICA algorithm to extract components, which have high-speed convergence compared to the traditional ICA. Suppose MI and SSVEP source signals are independent and non-Gaussian. In the preliminary stage, PCA is employed for feature enhancement and decorrelation. The rearrangement of the input matrix along the principal components of the data purifies the MI + SSVEP signals for the second stage. FastICA (i.e., fixed-point algorithm) is a fast optimization iterative algorithm based on batch processing [35]. This paper applies negentropy to measure the properties of non-Gaussian signals. The definition of negentropy is shown in equation (1). Here, YGauss is the Gaussian random variable having a similar variance as Y. HY is the differential entropy for the random variable, as in equation (2).(1)NgY=HYGauss−HY,(2)HY=−∫PyξlgPyξdξ.According to information theory, the random variable with the same variance of the Gaussian distribution has the most extensive differential entropy. WhenY has a Gaussian distribution of NgY=0, the non-Gaussian of Y is stronger. The smaller the differential entropy (i.e., NgY value), the higher NgY can be used as the random variable. The calculation of the Y non-Gaussian using the definition of negative entropy of probability density distribution function needs to know Y. However, in reality, it is challenging to calculate Y. Thus, we use the following approximation formula:(3)NgY=EgY−EgYGauss2,(4)Xt=∑i=1MSit∗ait,(5)St=BSt.EY refers to the formula for average computing. FastICA helps to find a dimension for WTX (y = WTX), which has the largest non-Gaussian signal. Non-Gaussian uses the approximation of the negative entropy to measure the signal. The variance of the WTX constraint is 1 for bleaching data, equal to that of the normal constraint of W to 1. Suppose Sit is an independent source signal, Xt is the mixed signal, and ait are the basis vectors from the column of the mixing matrix. As shown in equation (4), the goal is to estimate both unknowns from Xt, with appropriate assumptions about the statistical properties of the source distributions. The solution is sought in equation (5), where B is the separating matrix.


### 2.4. Wavelet Transformation

In the signal processing region, Fourier transformation (FT), short-time Fourier transform (STFT), and WT are the most popular methods for feature extraction. FT is applied to obtain frequency components for different periods. STFT can get the original signal’s frequency components varied over time by making a time slice of the original signals. In the WT case, the signals can be representations of a square-integrable function by a particular series generated by a mother wavelet. As we know, the STFT window’s length is stationary, which cannot detect the frequency of unsteadily changing EEG signals in the time domain. Since the mother wavelet could make dilation and translation to adjust the represented original signals, we can quickly analyse the raw signals in the time domain and the frequency domain [36].(6)WTα,Γ=1α∫−∞+∞ft∗φt−Γαdt,(7)φx=e−x2cosπ2ln2x.Equation (6) shows how to get the feature WTα,Γ from the original signals ft. The parameters a and Γ show the property of the dilation and the translation, respectively. φx is the function of the wavelet. Here, we use the Morlet wavelet to be as the wavelet, as shown in equation (7).(8)Δt=∫−∞∞t−t02φt2dt∫−∞∞φt2dt,(9)Δω=∫−∞∞ω−ω02φω2dω∫−∞∞φtω2dω,(10)ω0=∫−∞∞ωφω2dω∫−∞∞φtω2dω,(11)t0=∫−∞∞tϕt2dt∫−∞∞ϕt2dt.However, determining the suitable wavelet is a significant point to keep the original feature from the raw EEG signals. Since the properties of the wavelet could be known easily as a product of time rangeΔt and frequency band Δω, which can be calculated using equations (8) and (9), respectively. t0 and ω0 are given in equations (10) and (11). Thus, the WT algorithm can get a high resolution by the time-frequency trade-off.


### 2.5. Convolutional Neural Network

To avoid complicated direct pretreatment and input of the original images, CNN is widely used, including several paired convolution-pooling layers and a fully connected layer as the output. In the general neural network, the number of connected neurons is the same as input data; too much data cannot efficiently train the model. Therefore, the CNN employs local respective field and parameter sharing to reduce dimensions, input images recombined with several 2-dimensional filters in the convolution layer, and it is subsampled to a smaller size in the pooling layer. To explain the procedure, each matrixxi,j, which is achieved by the WT algorithm, is paired with the ith row and jth column. Wm,n is the weight of the mth row and the nth column, and Wb is the essential item of the filter. ƒ is the activation function (i.e., ReLU function). The element of the feature map is shown in the following equation:(12)ai,j=f∑m=0∑n=0wm,nXi+m,j+n+wb.The time-frequency-based feature is used as the input. The network is composed of four layers, each composed of one or more neurons. The CNN topology (Figure4) and interpretation are described as follows:Figure 4
The four-layer convolution neural network classifier structure used in this study.(i)
Layer 0 (L0): based on the result of WT, we obtained a 3-dimensional feature matrix (time, frequency, and amplitude). After collecting all task signals with different labels, the whole of the dataset would feed the CNN to train the model.(ii)
Layer 1 (L1): in the time domain, considering the dynamic feature applied to all data as the input, TS = 4,000. We select 5–15 Hz or 5–25 Hz (harmonic feature), FS = 10/20. Thus, the size of the matrix isFS∗TS. The input layer is Ii,j with 0 ≤ i ≤ FS and 0 ≤ j ≤ TS.(iii)
Layer 2 (L2): the first layer consists of the convolution and pooling layers. The convolution filter size is5∗5 with the same padding, with every step equated to 1. To avoid loss from the input information sources, the max-pooling layer is employed. In this system, the max-pooling method is used with a filter size of 2∗5. Thus, after processing L1, the matrix size will shrink to 4∗700/2∗700=2,800or1,400 neurons.(iv)
Layer 3 (L3): the third hidden layer is the fully connected layer, composed of 280 or 140 neurons.(v)
Layer 4 (L4): the fourth hidden layer is also the fully connected layer, conducted by the ten neurons reflecting the ten commands.


## 2.1. System Parameters

Keeping with regulations and guidelines established by the local ethics committee, a total of 6 healthy adults attended this experiment, including four females and seven males, aged 23 to 38 years. Participants gave written consent before the experiments. EEG signals were recorded from the central part of the head using two Ag/AgCI electrodes supported by a high-density wireless EEG headset (Cognionics Inc., San Diego, USA). The electrode distribution and the channel number are shown in Figure1. The impedance of all electrodes was kept below 5 kΩ, sampled at 500 Hz, and the band-pass was filtered between 1 and 50 Hz. Subjects were seated about 50 cm in front of the stimulation.Figure 1
EEG electrode distribution in accordance with the 10–20 EEG system. The electrodes used for acquisition are marked.


## 2.2. Experiment Paradiagm

A 21 inch light-emitting diode monitor with a 1920 × 1080 resolution and a 60 Hz refresh rate was used as the stimulator. Four different flickering frequencies (i.e., A: 6.6 Hz, B: 7.5 Hz, C: 8.57 Hz, and D: 9.6 Hz) were employed as a stimulus. The experiment involved ten runs for each task (i.e., A, B, C, D, A and MI, B and MI, C and MI, D and MI, idle, and MI task). Subjects were asked to sit in a comfortable chair and were instructed to follow the experimental procedure corresponding to the cues mentioned in Figure2. The fixation period lasts 0–3 s. After 3 s, the subject executes a task based on the cue (i.e., randomly display). Each task includes ten trials. If the screen shows an MI task cue, the subject must imagine he or she is grasping his or her right hand for 5 s. If the interface shows the SSVEP cue, the participants should look at the stimulus flicker window and keep his or her body under a calm state. Then, If MI and SSVEP cues are shown on the screen, the subject must execute both tasks at the same time (i.e., thinking as if he is grasping his right hand and looking at the specific stimulator). After the 5 s execution window, users were given 2 s to relax after each run. Each task totally took 8 s. In the SSVEP tasks, the screen shows four different windows with different frequencies marked. The subjects were asked to look at a specific stimulator for each case (i.e., frequency).Figure 2
Graphical user interface for the hybrid BCI system. (a) The caption EEG tasks shows the 10 mental tasks to be performed, which include idle state, MI task, SSVEP task, Freq. A-SSVEP task, Freq. B-SSVEP task, Freq. C-SSVEP task, Freq. D-SSVEP task, MI + Freq. A-SSVEP task, MI + Freq. B-SSVEP task, MI + Freq. C-SSVEP task, and MI + Freq. D-SSVEP task. (b) Time window for the performance of tasks divided into 2 parts for clarification, 0–3 s (rest state time) and 3–8 s (task-performing window). (c) Signal acquiring from the above performed tasks. (d) Processing, extraction, and classification of signals.After the recording of the signals, the further processing steps are highlighted in Figure3. Each mental task signal was processed, extracted, and classified. The procedure for signal data analysis is listed as follows:(i)
Preprocess 1: used a band-pass filter (5–25 Hz) to acquire the Mu band (8–13 Hz) and SSVEP signals (6.6–9.6 Hz). Mu band was used as it is efficient for detecting the MI tasks [16]. Band-pass filter frequency range covers all the required frequencies; it helps to remove the noise and provide the desired feature.(ii)
Preprocess 2: FastICA was employed to extract pure EEG signals as this method is fast and effective in obtaining the independent source signals from the fusion neural signals.(iii)
Extract feature: the features of time and frequency domains were analysed by the WT since the WT could extract both time and frequency domains simultaneously. In comparison with the FFT, FFT only shows the feature in the frequency domain, which might lead to the loss of the feature in the time domain.(iv)
Classifier: extracted features were the input to train the CNN model. CNN was employed for classification as it can classify multiple features by shared kernel parameters.Figure 3
Schematic architecture of the experiment setup for the hybrid BCI system with MI and SSVEP.


## 2.3. Fast Independent Component Analysis

In this study, we employed the FastICA algorithm to extract components, which have high-speed convergence compared to the traditional ICA. Suppose MI and SSVEP source signals are independent and non-Gaussian. In the preliminary stage, PCA is employed for feature enhancement and decorrelation. The rearrangement of the input matrix along the principal components of the data purifies the MI + SSVEP signals for the second stage. FastICA (i.e., fixed-point algorithm) is a fast optimization iterative algorithm based on batch processing [35]. This paper applies negentropy to measure the properties of non-Gaussian signals. The definition of negentropy is shown in equation (1). Here, YGauss is the Gaussian random variable having a similar variance as Y. HY is the differential entropy for the random variable, as in equation (2).(1)NgY=HYGauss−HY,(2)HY=−∫PyξlgPyξdξ.According to information theory, the random variable with the same variance of the Gaussian distribution has the most extensive differential entropy. WhenY has a Gaussian distribution of NgY=0, the non-Gaussian of Y is stronger. The smaller the differential entropy (i.e., NgY value), the higher NgY can be used as the random variable. The calculation of the Y non-Gaussian using the definition of negative entropy of probability density distribution function needs to know Y. However, in reality, it is challenging to calculate Y. Thus, we use the following approximation formula:(3)NgY=EgY−EgYGauss2,(4)Xt=∑i=1MSit∗ait,(5)St=BSt.EY refers to the formula for average computing. FastICA helps to find a dimension for WTX (y = WTX), which has the largest non-Gaussian signal. Non-Gaussian uses the approximation of the negative entropy to measure the signal. The variance of the WTX constraint is 1 for bleaching data, equal to that of the normal constraint of W to 1. Suppose Sit is an independent source signal, Xt is the mixed signal, and ait are the basis vectors from the column of the mixing matrix. As shown in equation (4), the goal is to estimate both unknowns from Xt, with appropriate assumptions about the statistical properties of the source distributions. The solution is sought in equation (5), where B is the separating matrix.


## 2.4. Wavelet Transformation

In the signal processing region, Fourier transformation (FT), short-time Fourier transform (STFT), and WT are the most popular methods for feature extraction. FT is applied to obtain frequency components for different periods. STFT can get the original signal’s frequency components varied over time by making a time slice of the original signals. In the WT case, the signals can be representations of a square-integrable function by a particular series generated by a mother wavelet. As we know, the STFT window’s length is stationary, which cannot detect the frequency of unsteadily changing EEG signals in the time domain. Since the mother wavelet could make dilation and translation to adjust the represented original signals, we can quickly analyse the raw signals in the time domain and the frequency domain [36].(6)WTα,Γ=1α∫−∞+∞ft∗φt−Γαdt,(7)φx=e−x2cosπ2ln2x.Equation (6) shows how to get the feature WTα,Γ from the original signals ft. The parameters a and Γ show the property of the dilation and the translation, respectively. φx is the function of the wavelet. Here, we use the Morlet wavelet to be as the wavelet, as shown in equation (7).(8)Δt=∫−∞∞t−t02φt2dt∫−∞∞φt2dt,(9)Δω=∫−∞∞ω−ω02φω2dω∫−∞∞φtω2dω,(10)ω0=∫−∞∞ωφω2dω∫−∞∞φtω2dω,(11)t0=∫−∞∞tϕt2dt∫−∞∞ϕt2dt.However, determining the suitable wavelet is a significant point to keep the original feature from the raw EEG signals. Since the properties of the wavelet could be known easily as a product of time rangeΔt and frequency band Δω, which can be calculated using equations (8) and (9), respectively. t0 and ω0 are given in equations (10) and (11). Thus, the WT algorithm can get a high resolution by the time-frequency trade-off.


## 2.5. Convolutional Neural Network

To avoid complicated direct pretreatment and input of the original images, CNN is widely used, including several paired convolution-pooling layers and a fully connected layer as the output. In the general neural network, the number of connected neurons is the same as input data; too much data cannot efficiently train the model. Therefore, the CNN employs local respective field and parameter sharing to reduce dimensions, input images recombined with several 2-dimensional filters in the convolution layer, and it is subsampled to a smaller size in the pooling layer. To explain the procedure, each matrixxi,j, which is achieved by the WT algorithm, is paired with the ith row and jth column. Wm,n is the weight of the mth row and the nth column, and Wb is the essential item of the filter. ƒ is the activation function (i.e., ReLU function). The element of the feature map is shown in the following equation:(12)ai,j=f∑m=0∑n=0wm,nXi+m,j+n+wb.The time-frequency-based feature is used as the input. The network is composed of four layers, each composed of one or more neurons. The CNN topology (Figure4) and interpretation are described as follows:Figure 4
The four-layer convolution neural network classifier structure used in this study.(i)
Layer 0 (L0): based on the result of WT, we obtained a 3-dimensional feature matrix (time, frequency, and amplitude). After collecting all task signals with different labels, the whole of the dataset would feed the CNN to train the model.(ii)
Layer 1 (L1): in the time domain, considering the dynamic feature applied to all data as the input, TS = 4,000. We select 5–15 Hz or 5–25 Hz (harmonic feature), FS = 10/20. Thus, the size of the matrix isFS∗TS. The input layer is Ii,j with 0 ≤ i ≤ FS and 0 ≤ j ≤ TS.(iii)
Layer 2 (L2): the first layer consists of the convolution and pooling layers. The convolution filter size is5∗5 with the same padding, with every step equated to 1. To avoid loss from the input information sources, the max-pooling layer is employed. In this system, the max-pooling method is used with a filter size of 2∗5. Thus, after processing L1, the matrix size will shrink to 4∗700/2∗700=2,800or1,400 neurons.(iv)
Layer 3 (L3): the third hidden layer is the fully connected layer, composed of 280 or 140 neurons.(v)
Layer 4 (L4): the fourth hidden layer is also the fully connected layer, conducted by the ten neurons reflecting the ten commands.


## 3. Results

### 3.1. PC Configuration and Feature Analysis

In this study, the experiments were conducted in MATLAB via an Intel 3.40-GHz core i7 PC with 8 GB RAM. MATLAB was used to process the feature and perform the classification. Figure5 shows the features of the ten mental tasks; the x-axis exhibits the time series, and the y-axis shows the frequency (unit of Hz). When the subject looks for the flicker of 6.6 Hz frequency, the EEG signals generated by the brain correspond to the respective frequency. Therefore, the WT power spectrum shows a high power for the respective frequency (i.e., 7.5 Hz, 8.57 Hz, and 9.6 Hz). When the subject performs the MI task, high power exhibits around 10–12 Hz, as shown in the WT power spectrum. The brain generates double features, while the subject executes hybrid tasks. For instance, the double features of 6.6 Hz + MI show a high-power spectrum, which appears in the frequency band of 6.6 Hz and 10–12 Hz. Therefore, the classifier could recognize various features for the performed tasks.Figure 5
The time and the frequency domain features for 10 different mental tasks: (a) gaze 6.6 Hz, (b) gaze 7.5 Hz, (c) gaze 8.57 Hz, (d) gaze 9.6 Hz, (e) MI task, (f) gaze 6.6 Hz + MI, (g) gaze 7.5 Hz + MI, (h) gaze 8.57 Hz + MI, (i) gaze 9.6 Hz + MI, and (j) idle state. The red circle marks the feature for easy observation.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)


### 3.2. Results for the Comparison of SSVEP Features

Desired to maintain a balance between the amount of the input data and the system’s performance, the experiment used two frequency windows (5–15 Hz and 5–25 Hz) to process data and to provide functional input verification for different CNN models. Moreover, the frequency window of 5–15 Hz could only detect single frequencies (i.e., 6.6 Hz, 7.5 Hz, 8.57 Hz, and. 9.6 Hz), and the 5–25 Hz window could only detect the second harmonic frequencies (i.e., 13.2 Hz, 15 Hz, 17.14 Hz, and 19.12 Hz). The 3rd harmonic frequency was too weak and unclear. Therefore, we did not consider it. Figure6 shows a single frequency feature and the second harmonic frequency feature. Considering that MI could influence the hybrid feature, Figure 7(a) used only the SSVEP signal for comparison. Figure 7(a) represents the accuracy of the six subjects during the SSVEP tasks of the different frequency bands (i.e., 5–15 Hz and 5–25 Hz). The harmonic feature (5–25 Hz) showed better classification accuracy (95.6%) as the harmonic feature includes both the original frequency feature (6.6 Hz, 7.5 Hz, 8.57 Hz, and 9.6 Hz) and the second harmonic frequency feature (13.2 Hz, 15 Hz, 17.14 Hz, and 19.2 Hz).Figure 6
The frequency feature and the second harmonic frequency feature for results of the different SSVEP stimulation: (a) 6.6 Hz flicker, (b) 7.5 Hz flicker, (c) 8.57 Hz flicker, and (d) 9.6 Hz flicker.
(a)(b)(c)(d)Figure 7
(a) The accuracy comparison between the single frequency feature (5–15 Hz) and the harmonic frequency feature (5–25 Hz) of the six subjects. (b) The accuracy comparison between the single frequency feature (5–15 Hz) and the harmonic frequency feature (5–20 Hz) of the six subjects.
(a)(b)


### 3.3. Results for the Comparison of FastICA

Typically, SSVEP signals are measured at the occipital cortex areas. As the recent studies [20, 21] demonstrated, the sensorimotor cortex region can detect the SSVEP signals as well. As mentioned before, the EEG signals are acquired from the motor nerve area. To get a clear feature and reduce interference from two different fusion signals, we applied FastICA to extract MI and SSVEP signals. In this experiment, we compared the signals with and without FastICA as the input to train the CNN model. Figure 7(b) represents classification results by the signals, which employed FastICA and without FastICA. The result indicated that the signal employing FastICA achieved the highest accuracy (95.53%) compared with the one without FastICA (73.41%). Error bar shows that the classification result is stable when the FastICA algorithm processes the dataset.


### 3.4. Results for the Comparison of Different Classifiers

Concerning the simple MI experiment, most studies employed support vector machines (SVMs) as the classier. For the SSVEP case, the least absolute shrinkage selection operator (LASSO) and the canonical correlation analysis (CCA) algorithms are the popular classifiers to be used in BCI areas. In order to verify the advantage of the CNN for the classification, we employ the same dataset which has been processed by FastICA. The results exhibit that the classifier’s performance directly influences the accuracy of the system. Comparing with the CNN, the performance of SVM, CCA, and LASSO could not satisfy the requirement of high accuracy. After using the CNN as the classifier, the accuracy of MI and SSVEP improved by 7.29% and 6.26%, respectively. CNN method shows steadier performance than both traditional methods, as shown in Table1. Meanwhile, the hybrid BCI system has higher accuracy than the single MI and SSVEP signals.Table 1
The accuracy and STD of the same input signals from different classifiers for the six subjects.
S1 (%)S2 (%)S3 (%)S4 (%)S5 (%)S6 (%)Mean (%)STD (%)MISVM82.5585.4781.6681.2384.2681.0282.701.64CNN89.8594.2991.7190.2093.0591.2691.731.55SSVEPCCA84.5288.6287.0089.2584.6987.2186.881.78LASSO85.2488.9588.6289.6689.5485.2587.881.89CNN92.5295.6294.2893.5294.8594.2094.170.98HybridCNN96.2094.8295.0892.9698.1196.0195.531.56


### 3.5. Results for the Single and Hybrid System

In order to exhibit the advantage of the simultaneous hybrid MI + SSVEP system, we divided the dataset to the single feature (i.e., MI/SSVEP) and hybrid feature (i.e., MI + SSVEP). Using the separate signals, the CNN model is trained, and their performances are verified. The results are shown in Figure8. According to the results showing a loss, the hybrid system illustrated a more stable performance than the single system, especially for MI. The accuracy also improved by different degrees. Therefore, the hybrid system was the best choice for the simultaneously controlled multicommand system. The simultaneous hybrid signals could be extended for multiple commands and consequently breaking out the limitation of the number of mental tasks and the cost of time windows. The sequential hybrid system needs at least two time windows to conduct one command, whereas the simultaneous hybrid mode reduces half of the time windows. Moreover, comparing the traditional hybrid system, this system also could conduct synchronized recognition for the single MI mental task, single SSVEP single task, and hybrid MI + SSVEP task.Figure 8
The performance for the MI single system (a, d), SSVEP single system (b, e), and hybrid MI and SSVEP system (c, f).
(a)(b)(c)(d)(e)(f)


## 3.1. PC Configuration and Feature Analysis

In this study, the experiments were conducted in MATLAB via an Intel 3.40-GHz core i7 PC with 8 GB RAM. MATLAB was used to process the feature and perform the classification. Figure5 shows the features of the ten mental tasks; the x-axis exhibits the time series, and the y-axis shows the frequency (unit of Hz). When the subject looks for the flicker of 6.6 Hz frequency, the EEG signals generated by the brain correspond to the respective frequency. Therefore, the WT power spectrum shows a high power for the respective frequency (i.e., 7.5 Hz, 8.57 Hz, and 9.6 Hz). When the subject performs the MI task, high power exhibits around 10–12 Hz, as shown in the WT power spectrum. The brain generates double features, while the subject executes hybrid tasks. For instance, the double features of 6.6 Hz + MI show a high-power spectrum, which appears in the frequency band of 6.6 Hz and 10–12 Hz. Therefore, the classifier could recognize various features for the performed tasks.Figure 5
The time and the frequency domain features for 10 different mental tasks: (a) gaze 6.6 Hz, (b) gaze 7.5 Hz, (c) gaze 8.57 Hz, (d) gaze 9.6 Hz, (e) MI task, (f) gaze 6.6 Hz + MI, (g) gaze 7.5 Hz + MI, (h) gaze 8.57 Hz + MI, (i) gaze 9.6 Hz + MI, and (j) idle state. The red circle marks the feature for easy observation.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)


## 3.2. Results for the Comparison of SSVEP Features

Desired to maintain a balance between the amount of the input data and the system’s performance, the experiment used two frequency windows (5–15 Hz and 5–25 Hz) to process data and to provide functional input verification for different CNN models. Moreover, the frequency window of 5–15 Hz could only detect single frequencies (i.e., 6.6 Hz, 7.5 Hz, 8.57 Hz, and. 9.6 Hz), and the 5–25 Hz window could only detect the second harmonic frequencies (i.e., 13.2 Hz, 15 Hz, 17.14 Hz, and 19.12 Hz). The 3rd harmonic frequency was too weak and unclear. Therefore, we did not consider it. Figure6 shows a single frequency feature and the second harmonic frequency feature. Considering that MI could influence the hybrid feature, Figure 7(a) used only the SSVEP signal for comparison. Figure 7(a) represents the accuracy of the six subjects during the SSVEP tasks of the different frequency bands (i.e., 5–15 Hz and 5–25 Hz). The harmonic feature (5–25 Hz) showed better classification accuracy (95.6%) as the harmonic feature includes both the original frequency feature (6.6 Hz, 7.5 Hz, 8.57 Hz, and 9.6 Hz) and the second harmonic frequency feature (13.2 Hz, 15 Hz, 17.14 Hz, and 19.2 Hz).Figure 6
The frequency feature and the second harmonic frequency feature for results of the different SSVEP stimulation: (a) 6.6 Hz flicker, (b) 7.5 Hz flicker, (c) 8.57 Hz flicker, and (d) 9.6 Hz flicker.
(a)(b)(c)(d)Figure 7
(a) The accuracy comparison between the single frequency feature (5–15 Hz) and the harmonic frequency feature (5–25 Hz) of the six subjects. (b) The accuracy comparison between the single frequency feature (5–15 Hz) and the harmonic frequency feature (5–20 Hz) of the six subjects.
(a)(b)


## 3.3. Results for the Comparison of FastICA

Typically, SSVEP signals are measured at the occipital cortex areas. As the recent studies [20, 21] demonstrated, the sensorimotor cortex region can detect the SSVEP signals as well. As mentioned before, the EEG signals are acquired from the motor nerve area. To get a clear feature and reduce interference from two different fusion signals, we applied FastICA to extract MI and SSVEP signals. In this experiment, we compared the signals with and without FastICA as the input to train the CNN model. Figure 7(b) represents classification results by the signals, which employed FastICA and without FastICA. The result indicated that the signal employing FastICA achieved the highest accuracy (95.53%) compared with the one without FastICA (73.41%). Error bar shows that the classification result is stable when the FastICA algorithm processes the dataset.


## 3.4. Results for the Comparison of Different Classifiers

Concerning the simple MI experiment, most studies employed support vector machines (SVMs) as the classier. For the SSVEP case, the least absolute shrinkage selection operator (LASSO) and the canonical correlation analysis (CCA) algorithms are the popular classifiers to be used in BCI areas. In order to verify the advantage of the CNN for the classification, we employ the same dataset which has been processed by FastICA. The results exhibit that the classifier’s performance directly influences the accuracy of the system. Comparing with the CNN, the performance of SVM, CCA, and LASSO could not satisfy the requirement of high accuracy. After using the CNN as the classifier, the accuracy of MI and SSVEP improved by 7.29% and 6.26%, respectively. CNN method shows steadier performance than both traditional methods, as shown in Table1. Meanwhile, the hybrid BCI system has higher accuracy than the single MI and SSVEP signals.Table 1
The accuracy and STD of the same input signals from different classifiers for the six subjects.
S1 (%)S2 (%)S3 (%)S4 (%)S5 (%)S6 (%)Mean (%)STD (%)MISVM82.5585.4781.6681.2384.2681.0282.701.64CNN89.8594.2991.7190.2093.0591.2691.731.55SSVEPCCA84.5288.6287.0089.2584.6987.2186.881.78LASSO85.2488.9588.6289.6689.5485.2587.881.89CNN92.5295.6294.2893.5294.8594.2094.170.98HybridCNN96.2094.8295.0892.9698.1196.0195.531.56


## 3.5. Results for the Single and Hybrid System

In order to exhibit the advantage of the simultaneous hybrid MI + SSVEP system, we divided the dataset to the single feature (i.e., MI/SSVEP) and hybrid feature (i.e., MI + SSVEP). Using the separate signals, the CNN model is trained, and their performances are verified. The results are shown in Figure8. According to the results showing a loss, the hybrid system illustrated a more stable performance than the single system, especially for MI. The accuracy also improved by different degrees. Therefore, the hybrid system was the best choice for the simultaneously controlled multicommand system. The simultaneous hybrid signals could be extended for multiple commands and consequently breaking out the limitation of the number of mental tasks and the cost of time windows. The sequential hybrid system needs at least two time windows to conduct one command, whereas the simultaneous hybrid mode reduces half of the time windows. Moreover, comparing the traditional hybrid system, this system also could conduct synchronized recognition for the single MI mental task, single SSVEP single task, and hybrid MI + SSVEP task.Figure 8
The performance for the MI single system (a, d), SSVEP single system (b, e), and hybrid MI and SSVEP system (c, f).
(a)(b)(c)(d)(e)(f)


## 4. Discussion

In this study, we offer a synchronized hybrid BCI system via a simple structure to recognize single MI, SSVEP, and hybrid MI + SSVEP modes (i.e., total ten mental tasks) simultaneously with high accuracy. To the best of our knowledge, this is the first research to recognize the ten mental tasks via 2 EEG channels with good accuracy of 95.56%. The critical point for the hybrid BCI system is that the corresponding signal components should be independent and could be evoked simultaneously [16]. For the general studies, as mentioned in Table 2, [18, 19, 23, 29, 37–40] studies employed different classifiers to do the classification, whereas our system applies a single CNN classifier that could recognize all the required tasks. In order to acquire enough signals to train the CNN model, six healthy participants attended our experiment. Each subject’s signal would separate 60% as the training data; the rest of the 40% signals was divided as the test data to demonstrate the trained model. Maximizing the number of commands and minimizing the number of the EEG channel was one of our goals for this study. As demonstrated by Allison et al. [20], 32-channel performance is 2-3% higher than the 3-channels, and considering more channels also could increase the artificial noise and bring inconvenience. Since the SSVEP can be loaded in the whole of the scalp, thus, we selected channels C3 and C4, which acquire the MI and SSVEP signals at the same time. To compensate for the loss which occurs due to the use of a smaller number of channels, we applied the FastICA to obtain more purified EEG signals and used WT to extract the time and frequency features, as more the number of features, easier is to classify different EEG tasks. With reference to Table 2, the range of the commands varied from a minimum number of 2 to a maximum of 8, with an accuracy ranging from 60 to 91%, whereas our study uses ten commands and results in an accuracy of 95.56%.Table 2
Comparison of different studies for hybrid MI + SSVEP-based BCI systems.
StudyModelAccuracy (%)CommandsModalityChannelTime windows (s)Cao et al. [18]290.638Simultaneous152Ma et al. [19]284.3 & 77.44Simultaneous1611Allison et al. [20]281.002Simultaneous58Horki et al. [23]285.004Sequential44Ko et al. [29]285.602Simultaneous28Millán [37]287 and 914Sequential58Long et al. [38]275.405Sequential15N/AAllison et al. [39]260.008Simultaneous318Li et al. [40]250–1004Simultaneous152.77–6.76Our study195.5610Simultaneous25


## 5. Conclusion

This paper introduced a synchronized hybrid EEG system, based on MI and SSVEP signals, which could recognize multiple mental tasks using a simple structure of two EEG channels. FastICA is applied as a filter to purify the fusion EEG signals. Wavelet transform is employed to get the 2-dimensional feature (i.e., time and frequency domains). CNN algorithm is utilized to be the classifier for EEG signals. Results show that the proposed synchronized hybrid system can recognize three different EEG types (i.e., ten control commands) with an accuracy of 95.56%, which is higher accuracy than single MI systems (90.16%) and single SSVEP systems (93.21%). This investigation indicates that the synchronized hybrid BCI system could achieve multiple command control with a simple structure by the MI and SSVEP signals. This finding provides a novel method for the hybrid BCI system and assesses the possibility for a simple hybrid BCI structure for multiple command control.



---

*Source: 4137283-2020-06-15.xml*