# Comparison of Machine Learning and Deep Learning Approaches for Decoding Brain Computer Interface: An fNIRS Study

Jiahao Lu, Hongjie Yan, Chunqi Chang, Nizhuan Wang

► To cite this version:

Jiahao Lu, Hongjie Yan, Chunqi Chang, Nizhuan Wang. Comparison of Machine Learning and Deep Learning Approaches for Decoding Brain Computer Interface: An fNIRS Study. 11th International Conference on Intelligent Information Processing (IIP), Jul 2020, Hangzhou, China. pp.192-201, 10.1007/978-3-030-46931-3_18. hal-03456988

HAL Id: hal-03456988

https://inria.hal.science/hal-03456988

Submitted on 30 Nov 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Comparison of Machine Learning and Deep Learning
Approaches for Decoding Brain Computer Interface: an
fNIRS Study

Jiahao Lu¹, Hongjie Yan², Chunqi Chang¹, *, Nizhuan Wang³, *

School of Biomedical Engineering, Health Science Center, Shenzhen University,
Shenzhen, China

cqchang@szu.edu.cn

Department of Neurology, Affiliated Lianyungang Hospital of Xuzhou Medical
University, Lianyungang, China

Artificial Intelligence & Neuro-Informatics Engineering (ARINE) Laboratory, School
of Computer Engineering, Jiangsu Ocean University, Lianyungang, China

wangnizhuan1120@gmail.com

**Abstract.** Recently, deep learning has gained great attention in decoding the neuro-physiological signal. However, which one (classical machine learning or deep learning) has better performance for decoding the functional near-infrared spectroscopy (fNIRS) signal is still lack of full verification. Thus, in this paper, we systematically compared the performance of many classical machine learning methods and deep learning methods in fNIRS data processing for decoding the mental arithmetic task. The classical machine learning methods such as decision tree, linear discriminant analysis (LDA), support vector machine (SVM), K-Nearest Neighbor (KNN) and ensemble methods with strict feature extraction and screening, were used for performance comparison, while the long short-term memory-fully convolutional network (LSTM-FCN) method as a representative of deep learning methods was applied. Results showed that the classification performance of SVM was the best among the classical machine learning methods, achieving that the average accuracy of the subject-related/unrelated were 91.0% and 83.0%, respectively. Furthermore, the classification accuracy of deep learning was significantly better than that of the involved classical machine learning methods, where the accuracy of deep learning could reach 95.3% with subject-related condition and 97.1% with subject-unrelated condition, respectively. Thus, this paper has totally showed the excellent performance of LSTM-FCN as a representative of deep learning in decoding brain signal from fNIRS dataset, which has outperformed many classical machine learning methods.

**Keywords:** Brain Computer Interface (BCI), Functional Near-infrared Spectroscopy (fNIRS), Brain Signal Decoding, Classical Machine Learning, Deep Learning
# 1. Introduction

The brain computer interface (BCI) is a technology that provides communication for the human or animal brain with the external environment [1]. When the brain performs a functional task, it activates (or suppresses) the functional brain-related regions, directly affecting the regional cerebral blood flow (rCBF) and cerebral blood volume (CBV). Changes occur and eventually manifest as rapid changes (elevation or decline) in blood oxygen levels in the corresponding regions of the brain, which is called neurovascular coupling [2]. Cerebral nerve activity causes the corresponding changes in blood oxygen levels, and the consequent changes in blood oxygen levels can impact the magnetic and optical properties of brain tissue. It is well known that the water, oxygenated hemoglobin (oxy-Hb), and deoxygenated hemoglobin (deoxy-Hb) have different absorption coefficients for near-infrared light of different wavelengths [3,4,5]. According to the neurovascular coupling, when the functional cognitive neural activity tasks are performed, the oxygen demand in the active area under the functional task is increased, with the perfused cerebral blood flow. In functional cognitive tasks, the activated brain region generally showed an increase in the concentration of oxy-Hb and total hemoglobin (t-Hb), a decrease in the concentration of deoxy-Hb. The well-known functional magnetic resonance imaging technique generates the blood-oxygen-level-dependent (BOLD) signals by magnetic changes caused by changes in hemoglobin concentration[6], while fNIRS measures oxy-Hb and deoxy-Hb by the absorption of near-infrared light at wavelengths around 704 nm and 887 nm[3,7,8].

In 2007, Coyle [9], Sitaram [10], Naito [11] has demonstrated the feasibility of controlling the output of fNIRS-BCI. Today, researchers can identify motor execution, motor imagery, metal arithmetic, music imagery with fNIRS-BCI [12]. Although deep learning has become more and more popular in signal processing, deep learning has not attracted enough attention from fNIRS-BCI researchers in recent years, which is possibly due to the limitation of relatively small samples in fNIRS-BCI experiments. For example, in 2015, Johannes Hennrich et al. has compared deep neural network with part of classical machine learning methods and they showed that deep neural network did not yield higher classification rates than the shrinkage LDA [13]. However, whether classical machine learning or deep learning has better performance for decoding the brain signal is still lack of full verification. Thus, in this paper, a public fNIRS-BCI mental arithmetic data was used to aim at completely finding out the which one (classical machine learning or deep learning) could perform better in fNIRS-BCI data processing under the brain decodin'g task. In order to improve the performance of machine learning, a new feature screening method is used to find out positive channels and time period of fNIRS mental arithmetic dataset.

# 2. Experimental Design and Dataset

In this paper, a public fNIRS-BCI dataset was involved in performance validation, which was a mental arithmetic dataset of prefrontal and temporal lobes, with a total of eight subjects, collected and published by the Neuroengineering Laboratory of Graz
University [14,15]. The designed experiment paradigm and data recording were
detailed in the following sub-sections.

**2.1 Experimental paradigm**

During the designed experiment, all eight subjects were firstly asked to keep an eye on the computer screen. The computer screen was black before the task was activated. Before the mental arithmetic task started, the green line appeared on the screen and lasted for two seconds. When the mental arithmetic task started, the mental arithmetic task prompt (e.g. 97-4) appeared on the screen for one second. The subject needed to follow the prompts to calculate the mental arithmetic task. Namely, the subject should calculate the 97 minus 4 task (97, 93, 89, 85...) until the green line in the middle of the screen disappeared. Specifically, the green line appeared for 14 seconds in each trial. The first two seconds prompt that the trial was about to start. After the green prompt line appeared for 2 seconds, the mental arithmetic task calculation formula appeared above the green prompt line for 1 second (e.g., 97-4). Then, the subject was asked to watch the screen and started mental arithmetic until the green line disappeared (the mental arithmetic task lasted for 12 seconds). After the green line disappeared, the subject continued to watch the black screen and to wait for the next trial (as a resting trial without mental arithmetic task).

**2.2 Data recording**

A continuous wave system (ETG-400, Hitachi Medical Co, Japan) was used in the
experimental instrument. Using 16 photo-detectors and 17 light emitters to form a 3×11
grid probe arrangement, 52 channels in total were distributed on the prefrontal and
temporal lobes, and each one was capable of measuring oxy-Hb concentration,
deoxy-Hb concentration and t-Hb concentration. The sampling rate was 10 Hz and the
distance between the source and the probe was 3 cm. The channel closest to the nose
side was arranged along the FP1-FP2 series of the International. The channel 48 in the
electroencephalogram 10-20 system was located at the FP1 position (as shown in Fig.
1).
<figure><img src="image_2.png" /><figcaption>Fig. 1 Schematic illustration of the multi-channel positional layouts for 52 channels [15]</figcaption></figure>

# 3. Classical Machine Learning for fNIRS Signal Decoding

## 3.1 Data Preprocessing

It well known that there is a physiological interference signal in the captured fNIRS data. In order to obtain a brain functional activation signal with high signal-to-noise ratio (SNR), it is necessary to preprocess the original data to reduce physiological noises caused by physiological activities such as heartbeat, respiration, and blood pressure fluctuations[16]. Physiological noise generated by heartbeat, respiration, blood pressure, and Mayer waves are relatively stable and statistically independent interference signal [17]. The most common method for dealing with physiological noise is to use a band-pass filter that uses a digital filter to eliminate the effects of heartbeat, respiration, Mayer wave signals, and baseline drift based on the frequency of physiological noise. In our experiment, the finite impulse response (FIR) band-pass filter is used to pass the 0.05~0.7 Hz signal, aiming at improving the SNR of the signal.

## 3.2 Feature Extraction

The fNIRS-BCI data refers to the time-series signal of the multi-channel scattered light intensity change collected by the fNIRS device. Statistical features directly reflect the statistical characteristics of fNIRS data and reduce the redundant information, which is
widely used in fNIRS data processing [18]. In the past few years, the main feature
extraction method of fNIRS-BCI data in classical machine learning methods is the
average concentration of the time-intercept signal intercepted in seconds [19,20,21,22]
and slope [21,22].

In the feature extraction stage, the average and slope of the 0.5 second window and 1 second window, the average of the full data segment, the linear fitting regression value, the variance, the range, and the skewness were selected as the classification features. Each subject dataset could provide 125 features, including the average and linear regression of the time window, and the average of the entire data segment, linear regression, variance, range, and skewness.

3.3 Feature Screening and Classification

After the stages of data preprocessing and feature extraction, the feature data with the size of 36×156×125 was first obtained. After the feature data was converted into two dimensions, a matrix with size of 36×19500 was obtained. The channel screening and feature screening procedure was performed to remove redundant information, and to determine the main activation channel (brain region) of the mental arithmetic task and the main change time of cerebral blood flow. The classification contribution values of each single-channel with all features data 36×1×125 and the single-feature with all channels data 36×156×1 were evaluated, respectively. The classification contribution values were then selected as follows: the input feature data was classified by five classifiers such as LDA, SVM, KNN, Decision Tree and Ensemble Learning classifier, and then the average value of the top three accuracy was used as a criterion.

Then, based on the channel classification contribution values and the feature
classification contribution values, the optimal combination of the multiple channels and
multiple features should be determined. In this paper, the channels and features
combinations of multi-channel and multi-features were classified and judged by the
exhaustive method with different number of optimal channels and optimal features.
Although the number of channels and the number of features was simply changed in the
feature screening process, the corresponding calculation time could be greatly reduced
in searching optimal combinations. The approximate activation channel (in spatial
domain) and approximate activation period (in temporal domain) of the mental
arithmetic task were obtained by optimal combination through a large number of
calculations. The detailed procedure was shown in Fig. 2.

Finally, after the procedure of feature screening, the chosen features were put into
5 classifiers (i.e., LDA, SVM, KNN, Decision Tree, and Ensemble Learning classifier)
to perform the classification task.
<figure><img src="image_3.png" /><figcaption>Fig. 2 Flowchart of channel and feature screening</figcaption></figure>

4. Deep Learning for fNIRS Signal Decoding

In this paper, a long time-term memory (LSTM) based on a fully convolutional
network (FCN) [23], namely LSTM-FCN was used. FCN replaced the last three fully
connected layers of the classic convolutional neural network (CNN) [24] with three
convolutional layers.

The original fNIRS data of 8 subjects were processed according to the data
interception method, generating the data with dimensions of 36×156×200, where 36
denoted the number of experimental trials, 156 represented the number of channels, and
200 was the length of a trial. Input data is randomly assigned in a 4 to 1 ratio to obtain
the training set and test set, respectively. The followed experimental results were
averaged from the results of 5 cross-validation.

The 1-D convolution has proven to be an effective learning method for time series
classification problems [24]. An FCN with 1-D convolution was commonly used as a
feature extractor. Global average pooling [25] was applied to reduce the number of
parameters in the model before classification. In the proposed model, the FCN was
enhanced by the LSTM module, and then the dropout function was involved to prevent
over-fitting and to accelerate the training process [26]. The FCN module consisted of
three stacked 1-D convolution blocks with filter sizes of 128, 256 and 128, respectively.
Each convolution block was identical to the convolution block in the FCN architecture
proposed by Wang [24]. Each block consisted of a temporal convolutional layer, each
block was processed for bulk normalization [27], and then the activation function was
used. The time series data was input into the LSTM block, and then the dropout function was called. The output of the FCN module and the output of the LSTM were concatenated. The architecture of LSTM-FCN for fNIRS-BCI data was showed in Fig.3.

<figure><img src="image_4.png" /><figcaption>Fig. 3 Architecture of LSTM-FCN for fNIRS-BCI data</figcaption></figure>

# 5. Results and Discussion

## 5.1 Channel and Feature Screening in Classical Machine Learning

In the optimal channels determination procedure, the classical machine learning methods classified 156 channels and 125 features without prior conditions, and performed exhaustive tests on all channels to find the optimal channels. After the data of several subjects was processed, it can be determined that the activated brain area of the mental arithmetic task was more significant in channel 26, 36, 37, 46, 48, which was shown in Fig. 4. Moreover, it was found in the feature screening stage that the average value of the data in the interval [6s, 17s] contributed more to classification performance. Each subject processing could provide a priori condition for the next data processing, which could help determine the activation brain area and activation time period.

<figure><img src="image_5.png" /><figcaption>Fig. 4 The activation channel map of mental arithmetic task</figcaption></figure>

## 5.2 Decoding Performance: Classical Machine Learning V.S. Deep Learning

<caption>**Table 1.** Accuracy of classical machine learning methods and deep learning method</caption>
<table><thead><tr><th></th><th>S01</th><th>S02</th><th>S03</th><th>S04</th><th>S05</th><th>S06</th><th>S07</th><th>S08</th><th>average</th><th>all subjects</th></tr></thead><tbody><tr><td>Decision tree</td><td>69.4 75</td><td></td><td>86.1</td><td>85.4</td><td>64.6</td><td>77.1</td><td>77.1</td><td>75</td><td>76.2</td><td>71</td></tr><tr><td>LDA</td><td>77.8 83.3</td><td></td><td>91.7</td><td>77.1</td><td>66.7</td><td>68.8</td><td>85.4</td><td>93.8</td><td>80.5</td><td>60.1</td></tr><tr><td>Classical machine learning</td><td>SVM <b>91.7</b> 91.7</td><td></td><td><b>97.2</b> <b>93.8</b> <b>77.1</b> <b>83.3</b></td><td>95.8</td><td><b>97.9</b> <b>91.0</b></td><td></td><td></td><td></td><td></td><td><b>83.0</b></td></tr><tr><td></td><td>KNN</td><td>77.8 83.3</td><td>88.9 89.6</td><td>66.7</td><td>77.1</td><td><b>97.9</b></td><td>93.8</td><td>84.3</td><td></td><td>69.8</td></tr><tr><td></td><td>Ensemble</td><td>83.3 <b>97.2</b></td><td>88.9 91.7</td><td>66.7</td><td>81.3</td><td>93.8</td><td>95.8</td><td>87.3</td><td></td><td>75.9</td></tr><tr><td>Deep learning: LSTM-FCN</td><td>0.3 <b>91.7</b> 91.7</td><td></td><td><b>100</b> <b>100</b></td><td>87.5</td><td>93.7</td><td>97.9</td><td><b>100</b> <b>95.3</b></td><td></td><td></td><td>86.4</td></tr><tr><td>with different dropout rate</td><td>0.5 86.1 83.3</td><td></td><td>97.2 <b>100</b></td><td><b>93.7</b> <b>95.8</b> <b>100</b></td><td><b>100</b></td><td>94.5</td><td></td><td></td><td></td><td><b>97.1</b></td></tr><tr><td></td><td>0.6 83.3 86.1</td><td></td><td>97.2 <b>100</b></td><td>89.5</td><td>93.7</td><td><b>100</b> <b>100</b></td><td>93.7</td><td></td><td></td><td>88.5</td></tr><tr><td></td><td>0.8 <b>91.7</b> 88.8</td><td></td><td><b>100</b> <b>100</b></td><td>74.9</td><td>79.2</td><td>95.8</td><td><b>100</b></td><td>91.3</td><td></td><td>90.8</td></tr></tbody></table>

In Table 1, the highest accuracy of SVM in the classical machine learning of the subject-related classifier was 91%, and the accuracy of the deep learning method was 95.3%. Under the subject-unrelated conditions, the accuracy of classical machine learning reached 83%, and the accuracy of deep learning reached 97.1%. For the S03, S04, S07, S08, in the case of different network dropout rate, can achieve 100% accuracy. At this point, deep learning method, i.e., LSTM-FCN, has been significantly better than the classical machine learning methods have, and a 100% accuracy could make BCI system more stable.

In the subject-related classification, accuracy of S05 and S06 was lower than that of other subjects. But in deep learning method, the accuracy of S05 and S06 can reach 90%. Contritely, the deep learning method (LSTM-FCN) still got better decoding performance for the difficult subjects in the classical machine learning. In the subject-unrelated classification, the deep learning method was with significantly higher accuracy than the classical machine learning methods did. The highest accuracy of deep learning was 14.1% higher than that of the classical machine learning methods where the accuracy of deep learning is very close to 100%. Thus, according to Table 1, it was easily concluded that the LSTM-FCN method as a representative of deep learning had better performance for fNIRS-BCI decoding.
## 6. Conclusion

In this paper, we made a full decoding performance comparison between the classical machine learning methods and deep learning method on fNIRS-BCI data. According to the experimental results, it was found that the LSTM-FCN method as a representative of deep learning showed superior performance to all the classical machine learning methods such as decision tree, LDA, SVM, KNN and ensemble classifier. In terms of the training time in the classical machine learning methods, it took more time to do channels screening and features screening task. As the amount of data increased and the new models were developed, the deep learning methods could achieve the classification accuracy and robustness on the small fNIRS-BCI dataset. Thus, the deep learning is a promising tool for decoding fNIRS-BCI data in comparison to the classical machine learning.

## Acknowledgements

This work was supported by National Natural Science Foundation of China (No. 61701318, 61971289), Natural Science Research Project of Jiangsu Higher Education Institutions (No. 18KJB416001), Project of "Six Talent Peaks" of Jiangsu Province (No. SWYY-017), Shenzhen Fundamental Research Project (No. JCYJ20170412111316339), Shenzhen Talent Peacock Plan (No. 827-000083), and Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions.

## References

1. Bauernfeind, G. et al.: Single trial classification of fnirs-based brain-computer interface mental arithmetic data: a comparison between different classifiers. 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2004-2007. IEEE (2014).

2. Pasley B. N., Freeman R. D.: Neurovascular coupling. Scholarpedia, 3(3), 5340 (2008).

3. Scholkmann F., Kleiser S., Metz A.J., et al: A review on continuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neuroimage, 85, 6-27 (2014).

4. Hoshi Y.: Functional near-infrared optical imaging: Utility and limitations in human brain mapping. Psychophysiology, 40(4), 511-520 (2003).

5. Hoshi Y.: Near-infrared spectroscopy for studying higher cognition//Neural Correlates of Thinking. Springer, Berlin, Heidelberg, 83-93 (2009).

6. Kim S.G., Ogawa S.: Biophysical and physiological origins of blood oxygenation level-dependent fMRI signals. Journal of Cerebral Blood Flow & Metabolism, 32(7), 1188-1206 (2012).

7. Leff D.R., Orihuela-Espina F., Elwell C.E., et al.: Assessment of the cerebral cortex during motor task behaviours in adults: a systematic review of functional near infrared spectroscopy
(fNIRS) studies. Neuroimage, 54(4), 2922-2936 (2011).

8. Irani F., Platek S.M., Bunce S., et al.: Functional near infrared spectroscopy (fNIRS): an emerging neuroimaging technology with important applications for the study of brain disorders. The Clinical Neuropsychologist, 21(1), 9-37 (2007).

9. Coyle S.M., Ward T.E., Markham C M. Brain–computer interface using a simplified functional near-infrared spectroscopy system. Journal of Neural Engineering, 4(3), 219 (2007).

10. Sitaram R., Zhang H., Guan C., et al.: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage, 34(4), 1416-1427 (2007).

11. Naito M., Michioka Y., Ozawa K., et al.: A communication means for totally locked-in ALS patients based on changes in cerebral blood volume measured with near-infrared light. IEICE transactions on information and systems, 90(7), 1028-1037 (2007).

12. Naseer N., Hong K. S.: fNIRS-based brain-computer interfaces: a review. Frontiers in human neuroscience, 9, 3 (2015).

13. Hennrich, J., et al.: Investigating deep learning for fNIRS based BCI. 2015 37th Annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2844-2847. IEEE (2015).

14. Günther B., Scherer R., Pfurtscheller G., et al.: Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic. Medical & Biological Engineering & Computing, 49(9), 979-984 (2011).

15. http://bnci-horizon-2020.eu/database/data-sets

16. Kirilina E., Jelzow A., Heine A., et al.: The physiological origin of task-evoked systemic artefacts in functional near infrared spectroscopy. Neuroimage, 61(1), 70-81 (2012).

17. Bauernfeind, G., et al.: Physiological noise removal from fNIRS signals. Biomedical Engineering/Biomedizinische Technik (2013).

18. Cui X., Bray S., Reiss A.L.: Speeded near infrared spectroscopy (NIRS) response detection. PLoS One, 5(11), e15474 (2010)

19. Putze F., Hesslinger S., Tse C.Y., et al.: Hybrid fNIRS-EEG based classification of auditory and visual perception processes. Frontiers in Neuroscience, 8, 373 (2014).

20. Yin X., Xu B., Jiang C., et al: A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching. Journal of Neural Engineering, 12(3), 036004 (2015).

21. Buccino, A. P., Hasan O.K., Ahmet O.: Hybrid EEG-fNIRS asynchronous brain-computer interface for multiple motor tasks. PloS One, 11,1 (2016).

22. Raheel B.M., Hong M.J., Yun-Hee K., et al.: Single-trial lie detection using a combined fNIRS-polygraph system. Frontiers in Psychology, 6, 709 (2015).

23. Karim F., Majumdar S., Darabi H., et al.: LSTM Fully Convolutional Networks for Time Series Classification. IEEE Access, 6,1662-1669 (2017).

24. Wang Z., Yan W., Oates T.: Time series classification from scratch with deep neural networks: A strong baseline. 2017 International joint conference on neural networks (IJCNN), pp. 1578-1585. IEEE (2017).

25. Lin M., Chen Q., Yan S.: Network in network, [online] Available: https://arxiv.org/abs/1312.4400 (2013).

26. Srivastava N., Hinton G., Krizhevsky A., et al.: Dropout: a simple way to prevent neural
networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958 (2014).

27. Ioffe S., Szegedy C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).