—In this study, we investigated the use of multimodal functional neuroimaging in detecting mental stress on the prefrontal cortex (PFC ). We recorded Electroencephalo graphy (EEG) and functional Near Infrared Spectroscopy (fNIRS) simultaneously from 20-subjects performing ment al arithmetic task under control and stress conditions. Stress was induced in this study based on two established stressors – time pressure and negative feedback about peer performance. We explored decision fusion by using support vector machine classifier for each modality, and optimizing the classifiers based on Receiver Operating Characteristic (ROC) curve values. Experiment results revealed significant reduction in alpha rhythm and mean change in concentration of oxygenated hemoglobin at PFC when stress ed, p<0.001 and 0.0001 respectively . The decision fusion improved significantly the detectio n rate of mental stress by +7.76% and +10.57 %, when compared to sole modality of EEG and fNIRS, respectively . Index Terms —Stress, neuroimaging, decision fusion I. INTRODUCTION TRESS is a growing problem in our society and is part of our daily life. We spend most of our time as adult at workplace . Extended periods of high workload and time pressure contribute to increased level of stress [1] [2]. Conventionally, mental stress is evaluated by using subjective questionnaires, cortisol level and other physiological signals (i.e. heart rate, blood pressure and skin conductance) [1]. Such methods known to be affected by cir cadian rhythm humidity and cardiovascular diseases [3, 4] . As stress is originated from the amygdala, it is desirable to study the stress effects on brain activities and measure them with modern neuroimaging modalities. However, each of the modality has its o wn limitations. Medical imaging modalities such as functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) have good spatial resolution but have limitations in term of temporal resolution and susceptibility to movement artefacts [5]. Moreover , subjects need to remain still during fMRI/PET measurements [6]. Electroencephalography (EEG) is a possible alternative neuroimaging modality as it has a temporal resolution in the order of a few milliseconds, which makes it suitable f or measuring dynamic cortical changes during workplace activities [7].Unlike functional magnetic resonance imaging and positron emission tomography, modern EEG hardware is light -weight and portable enough to be used during unconstrained full -body motion. Studies have shown that EEG signals can be used to classify mental stress from resting state [8, 9] . Nevertheless , EEG has traditionally been thought of as possessing poor spatial resolution and being highly prone to motion artifacts [10]. Functional Near -Infrared Spectroscopy (fNIRS) detects the neural activities of the brain by measuring changes in the concentration of oxygenated hemoglobin (O2Hb) and deoxygenated hemoglobin (HHb) in the cortex [11, 12]. The fNIRS is a promising alternative, achieving some middle ground in spatial and temporal resolution as well as mobility between EEG and fMRI techniques [13-16]. Additionally, the measurements have been shown to be consistent with fMRI and EEG [17, 18] . In this paper, we hypothesize that fusion of EEG and fNIRS can improve the detection rate of mental stress from using either modality alone . Compare to traditional neuroimaging, EEG+fNIRS have the advantages of allowing human cortical activities to be measured without major constraints on how subjects are positione d (mobility) , and complement each other in the temporal and spatial resolution . Recently, there has increased interest in the fusion of neuroimaging modalities. Fusion is a method of gathering and analyzing complementary modalities utilizing the ir common and unique features to provide better understanding in the detection and treatment of brain diseases . In the fusion terminology, data from different modalities usually analyzed either in the feature level or at the decision level [19, 20] . In the feature level, fusion techniques incorp orate all features from different modalities into a combined analysis to explore associations across them through variations across subjects. Joint independent component analysis (jICA) and canonical correlation analysis (CCA) are of the most common feature -fusion techniques [19]. In the decision level, each data are analyze d separately to provide a local decisions that are based on individual features. The local decisions then combined using decision fusion technique to improve the overall classification performance [21]. Decision fusion has the advantages of using the most suitable methods for analyzing individual modality, which provide much Decision Fusion of EEG and fNIRS Signals Fares Al -Shargie S 2 flexibility than feature -level fusion. Additionally, decision - level fusion offers scala bility in term of the modalities used in the fusion process which is difficult to achieve in the feature - level fusion [22]. This paper introduced a decision -level fusion to fuse two decisions according to the operating points on their receiver operating characteristic (ROC) curves. Our experimental results demonstrate that, the proposed decision fusion achieves significant improvement over individual EEG/fNIRS classification performance. The paper is organized as follows. In section II, we briefly describe on the participants selection, development of the stress task, and describe the set s of EEG and fNIRS signals used. We explained the methods used to process the data and the features used for the statistical analysis, classification and fusion. In section III, we present the results of individual modality, the statistical analysis and the results of fusion. In section IV, a discussion of the presented results is provided in the light of existing studies. Finally, in section V, we conclude the paper. II. METHODOLOGY A. Participants Participants were t wenty -eight healthy right -handed students from the American University of Sharjah (22±2 years, 13 males; 21±1.5 years, 15 female ). All of them reported normal or corrected -to-normal vision. None of the participants had a history of neurological or psychiatric illness and had no current or prior psy choactive medication use. They were asked to abstain from caffeine, exercise , energy drink and tobacco use for 24 hours before testing. All participants were explained about the study and gave informed consent before the start of the experiment. The experi ment protocol conducted in accordance with the Declaration of Helsinki and approved by local ethics review committee at the American University of Sharjah . B. Emotion -Eliciting Pictures participants. Participants first practice the task for five minutes a nd the average time taken for each individual in answering the questions was recorded. indicators (one for the participant’s performance and one for the averaged peer performance fixed at 90% accuracy) were displayed on the computer monitor to further indu ce stress in experiment participants. Fig. 1. Schematic of the e xperiment al procedure for mental stress study. Two recordings were performed in this experiment; one for control condition and the other one for stress condition. In each recording, there were five blocks. In each block, mental arithmetic was allocated for 30 s followed by 20 s re st. C. Data Acquisition Electrophysiological signals were recorded using 64 Ag/AgCl scalp electrodes arranged according to the standard 10–20 system (ANT waveguard system and ASA Lab 4.9.2 acquisition softwa re, ANT Neuro, the Netherlands), sampled at 500 Hz. The impedances of all EEG e lectrode s were maintained below 20 kΩ throughout the recording. Fig. 2. Probe setting and schematic arrangement points. D. Data processing EEG signals were preprocessed using custom script as well as the EEGLAB toolbox( 9.0.4 ) [25]. The EEG signals were re - referenced using the common average reference. Baseline removal and DC offset was done by subtracting the mean from the data. All EEG signals wer e band -pass filtered using FIR filter with 0.5 Hz and 30 Hz cut -off frequencies. The electrical (A) (B) 3 power line noise was removed on ICs using the CleanLine plug - in of EEGLA B. Eye-blinks and ey e-movements were removed manually by visual inspection as well as using Independent component analysis (ICA) technique available in EEBLAB . The components representing artifacts, such as eye blink, eye movements, and muscular activities were removed and the remaining components were used to reconstruct the clean EEG signals. E. Statistical analysis To reveal the differences in brain responses between control and stress groups in the subjective as well the simultaneous measurement of EEG+fNIRS , we analyzed the differences between them in channel/electrode basis using two -sample t - test. In each electro de/channel, t he differences were considered statistically significant if the p value is less than 0.001, p< 0.001 . F. Classification performance To classify EEG and fNIRS signals, we used support vector machine (SVM) classifier . The SVM is a supervised machine learning technique widely used for classification, regression and density estimation [31]. SVM is selected for its ability to model linear as well as more complex decision boundaries. LIBSVM software was used to build the SVM classifier and employed radial basis function (RBF) kernel to nonlinearly map data onto a higher dimension space [32]. A 10 -fold cross -validation scheme with randomization was applied to each feature vector. In the 10 -fold cross validation, each of the EEG and fNIRS features were split into ten s ubsets. Nine subsets were used to train the SVM classifier, and the remaining one subset was used for estimation of classification accuracy, sensitivity and specificity. This procedure was repeated ten times with each subset having an equal chance of being the testing data. Then we analyzed the classifier performance by using receiver operating characteristic (ROC) curves, which plot the sensitivity versus (1 minus the specificity) with varies threshold . We also evaluated the area under the ROC curve (AUC) as a measure of a classifier’s discriminatory power. G. Decision fusion of EEG and fNIRS signals The decision fusion was achieved by fusing the outputs/ decisions from two classifiers, one for EEG signals and the other for fNIRS signals. Each classifier was calibrated based on the operating points of EEG and fNIRS receiver operating characteristic ( ROC ) curves. The type of classifiers used in this study is support vector machine (SVM) with radial basis functions. Output by individual classifier is denoted as uk, (EEG: k = 1; fNIRS: k = 2), where uk = 0, if the decision of kth classifier is H0 (no-stress) and uk = 1 if the decision of classifier is H1 (stress) . The decision of classifier uk depends on the features vectors xk, k = 1,2. 𝑢𝑘=𝛼𝑘(𝑥𝑘){0,classifier 𝑘 decides 𝐻0 (no−stress ) 1,classifier 𝑘 decides 𝐻1(stress ) (1) where α1 and α2 are sets of threshold value of EEG and fNIRS classifiers respectively. The performance characteristics of classifier k specified by P(uk=1|H j) where P(uk=1|H0) = Pfk is the probability of false positives , and P(uk=1|H1) = Pdk is the probability of true positives. Using these probabilities, the likelihood ratio value of a binary decision variable may be obtained by : 𝑑𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑃(𝑢𝑘|𝐻1) 𝑃(𝑢𝑘|𝐻0)=(𝑃𝑑𝑘 𝑃𝑓𝑘)𝑢𝑘 (1−𝑃𝑑𝑘 1−𝑃𝑓𝑘)1−𝑢𝑘 (2) The decision at the fused level depends only on local decisions and their probability of true positive Pdk and false positive Pfk. Since individual classifiers are based on different modalities, the two decisions assumed to be statistically independent. The fusion likelihood ratio could then be generated using the following form: 𝑓𝑢𝑠𝑖𝑜𝑛 (𝑢1,𝑢2)=∏ (𝑃𝑑𝑘 𝑃𝑓𝑘)𝑢𝑘 ∏ (1−𝑃𝑑𝑘 1−𝑃𝑓𝑘)1−𝑢𝑘 (3)2 𝑘=12 𝑘=1 The optimal fused decision rule uses the fusion likelihood ratio as a classification decision variable and then compares it to threshold β for decision u according to the following : 𝑢=𝐹(𝑢1,𝑢2)=𝐻(𝑓𝑢𝑠𝑖𝑜𝑛 −𝛽) (4) where H is the Heaviside function. By varying the threshold β value (corresponding to different decision rules), N number of operating points were derived for the fused ROC curve. The probability of true positive rate and false positive rate s for decision -fused classifier operating points could then be calculated based on the following equations: 𝑃𝑑𝑓𝑢𝑠𝑖𝑜𝑛 (𝛽)= ∑ 𝑃(=𝑓𝑢𝑠𝑖𝑜𝑛 |𝐻1) (5) 𝑓𝑢𝑠𝑖𝑜𝑛 ≥𝛽 𝑃𝑓𝑓𝑢𝑠𝑖𝑜𝑛 (𝛽)= ∑ 𝑃(=𝑓𝑢𝑠𝑖𝑜𝑛 |𝐻0) (6) 𝑓𝑢𝑠𝑖𝑜𝑛 ≥𝛽 The entire decision -fused ROC curve obtained by deriving a series of Pk,fusion(β) using multiple combinations of operating points on EEG -/fNIRS -based classifier ROC curves. III. RESULTS Experiment result showed that subjects’ performance decreased with stress, accuracy score of answering the tas k correctly was <40%, as expected. The results of subjective ratings of workload measured by NASA -TLX across the control and the stress conditions for all subscales is summarized in Table 1. Additionally, the overall workload of the NASA -TLX is also presen ted with their standard error . The weighted workload (WWL) in the control and stress conditions were 22.2 and 75.4, respectively. Based on the participant’s subjective responses, in control condition own performance ( OP) and in stress condition frustration (FR), temporal demand (TD) and physical demand (PD) had dominant importance and i n control and stress conditions, effort ( EF) had the lowest importance. Overall, there was a significant differences between the subjective score from the control to the stress condition across all the subjects and the NASA -TLX subscales . The t and p values of the subjective evaluations are summarized in Table 1. The result confirms the inducement of moderate stress level in every subject. 4 TABLE 1 COMPARISON OF NASA -TLX SUBSCALES MEANS ± SE AFTER CONTROL AND STRESS CONDITIONS OF THE ARITHMETIC TASK NASA -TLX Control Stress T-test result t-value p-value MD 20±7.6 75±4.7 6.30 <0.0001 PD 25±8.4 77±6.3 6.41 <0.0001 TD 23±6.6 78±4.5 6.31 <0.0001 OP 85±6.3 20±3.1 7.41 <0.0001 EF 30±4.5 68±2.3 5.81 <0.0001 FR 32±3.2 88±1.2 7.62 <0.0001 WWL 22.2±2.5 75.4±3.7 6.21 <0.0001 Abbreviations: MD, mental demand; PD, physical demand; TD, temporal demand; OP, own performance; EF, effort; FR, frustration; WWL, weighted workload; MW, mental workload; T -test analysis; SE, standard error. The result of EEG alpha rhythm revealed significant decrease from control to stress condition, p<0.001. The decrease was found across all the electrodes in all the subjects. The overall statistic al analysis (t and p values) are summarized in Table 2, with F4, F8 and Fp1 showing the highest t -values. Figure 3 shows the normalized alpha rhythm under control and stress condition s as averaged of all electrodes on the PFC represented by the box plots. Fig. 3. Normalized alpha rhythm under control and stress condition. Similarly, the results of fNIRS demonstrated significant increase in the O 2Hb changes associated with a decrease in the HHb during the control arithmetic task relative to the pre -task baseline over the entire PFC (except Ch -18 and Ch -23; in the case of O 2Hb only) . Likewise, under stress condition the O2Hb changes decreased in association with an increase in the HHb in most of the PFC regions relative to pre -task, on average (except at Ch-7 and Ch -12). The overall grand average time - course of O 2Hb (red line) and HHb (blue line) changes at control and stress condition s are shown in Fi g.4 and Fig.5, respectively . As the O 2Hb highly reflect the cortical activities we limited our analysis in this work to their response. As the focus of this study is to investigate on the behavioral responses of the neural activities under neural -control and stress conditions , we conducted the statistical analysis between the two groups . The entire study, revealed significant decreases in the O2Hb changes from control condition Fig.4 to stress condition Fig.5 over some regions of the PFC areas ( excluded at Ch -7, Ch -12, Ch -17, Ch -18 and Ch -23). As can be seen from Fig.4 and Fig.5, stress is a region specific than diverse and showed the left PFC area is the less affected region by stress. Table 2 summaries the statistical analysis of all the subjects and channels represented by t and p values with their standard deviations (SD). The lowest standard error acros s all channels is located at the right ventrolateral PFC, Ch -13, Ch -14 and Ch - 19, indicating that this sub -region is dominant across the subjects to mental stress. Fig. 4. Grand average waveforms of hemoglobin concentration ([Hb]) changes during the arithmetic task at control condition across all the subjects for 23 - channels. The vertical red line marks the beginning of the task and vertical red - dashed line marks the end of the task condition in every channel. Fig. 5. Grand average waveforms of hemoglobin concentration ([Hb]) changes during the arithmetic task at stress condition across all the subjects for 23 - channels. The vertical red line marks the beginning of the task and vertical red - dashed line marks the end of the task condition in every channel. 0.50.60.70.80.9 Control StressNormalized Alpha Rhythm 5 TABLE 2 STATISTICAL ANALYSIS OF O2HB AND ALPHA RHYTHM Channel t-value (SD) p-value (<) Channel t-value (SD) p-value (<) 1 5.3±0.62 0.0002 16 6.1 ±0.64 0.000 1 2 4.3±0.56 0.0005 17 2.8±0.44 0.0573 3 3.1±0.53 0.0012 18 2.1±0.93 0.0665 4 5.6±0.51 0.000 1 19 5.1±0.12 0.0001 5 5.7±0.62 0.0001 20 5.2±0.22 0.000 1 6 6.0±4.3 0 0.0002 21 5.3±0.55 0.0002 7 2.5±0.71 0.0610 22 5.0±0.75 0.0005 8 6.4±0.34 0.0001 23 2.5±0.66 0.0606 9 5.2±0.44 0.000 5 F4 5.7±0.31 0.0001 10 6.5±0.74 0.0001 Fz 4.8±0.22 0.000 2 11 6.5±0.71 0.0001 F3 4.9±0.65 0.000 5 12 2.2±0.73 0.0620 F8 5.6±0.31 0.000 1 13 5.2±0.11 0.0002 F7 4.6±0.61 0.0005 14 4.8±0.14 0.0005 Fp1 5.5±0.73 0.0002 15 6.6±0.71 0.0001 Fp2 5.4±0.41 0.0001 The classification result of individual modality and after fusion is shown by their ROC curve s. Features from t he seven electrodes (FP1, FP2, Fz, F3, F4, F7 and F8) were used for evaluating EEG modality and for fusion with the fNIRS 1 and fNIRS 2. The fNIRS (1,2) channel selection was based on two - lateralize neighboring channels in each subregio n of the PFC areas (within the right and left: DLPFC; VLPFC and FPA) . The first fNIRS 1 lateralize channel s were Ch -2, Ch -4, Ch -3, Ch -23, Ch-13, Ch -22 and Ch -20. The second fNIRS 2 lateralize channels were Ch -2, Ch -1, Ch -7, Ch -18, Ch -19, Ch -15 and Ch - 16. Fig.6 and Fig.7 show the ROC curves of individual EEG, fNIRS 1, fNIRS 2, fusion of EEG+ fNIRS 1 and fusion of EEG+ fNIRS 2. The black line shows the result from fNIRS (1,2), blue line shows the result from EEG and the red line shows the result of fusing EEG+fNIRS (1,2), respectively. The average classification performance of sole EEG, sole fNIRS 1, fusion of EEG+fNIRS 1, sole fNIRS 2 and fusion of EEG+f NIRS 2 in the form of accuracy, sensitivity, specificity and area under the ROC curves are summarized in Table 3. Decision fusion shows improvements in the classificatio n accuracy in the range of +7.76 % to +10.57 % compare to sole EEG and sole fNIRS (1,2) respectively. Similar improvements were found in the classification sensitivity, specificity and in the area under ROC curves in the range of +3.45 % to +11.65 %. Using two sample t -test, fusion of EEG+fNIRS (1,2) significantly improves the overall classification performance in all the metrics with p<0.001 . TABLE 3 CLASSIFICATION PERFO RMANCE OF EEG , FNIRS AND EEG +FNIRS Modality Accuracy (%) Sensitivity (%) Specificity (%) AROC (%) EEG 88.69 87.60 89.70 95.10 fNIRS 1 84.76 82.50 87.00 91.50 fNIRS 2 87.00 84.40 89.70 94.40 fNIRS 1+ fNIRS 2 85.88 83.45 88.35 92.95 EEG+ fNIRS 1 96.42 94.80 97.90 98.80 EEG+ fNIRS 2 96.48 95.40 97.50 98.30 EEG+ fNIRS( 1,2) 96.45 95.10 97.70 98.55 Improvement (EEG) 7.76 7.50 8.00 3.45 Improvement (fNIRS) 10.57 11.65 9.35 5.60 Fig.6. ROC curve of EEG (blue line), fNIRS 1 (black line) and EEG+fNIRS 1 (red line). Fig. 7. ROC curve of EEG (blue line), fNIRS 2 (black line) and EEG+fNIRS 2 (red line). IV. DISCUSSION In the present study, we investigated the performance of fusing simultaneous measurement of EEG and fNIRS signals for discriminating between set of task at control and stress condition s. EEG provides data on electrical cortical activity whereas fNIRS monit or the cerebral blood flow and the oxygen metabolism on the cortex . The two measurement systems complement each other in quality (data sources) as well as in the quantity. Thus , integrating this physiological responses enhance the detection rate of mental stress. The stress stimuli used in this study is based on arithmetic task complexity with time pressure and negative feedback of the subject performance. The task have proven to induce stress on the participants, as pro ven in our previous study in which the salivary cortisol significantly increased from control to stress condition in all the participants [23]. Additionally, we involved the behavioural responses as well as the subjective evaluation collected using NASA -TLX and demonstrated that all 6 participants exp erience stress while solving the arithmetic task under time pressure. The EEG results showed that stress disrupt the cortical activities within the entire PFC region. There was a significant reduce in alpha rhythm from control to stress condition with mean p-value of p<0.001. The reduce in alpha rhythm in this study could be explained as that, the time pressure with negative feedback induced negative emotion which with continuous recording time further increase the level of stress. Similar decrease in alph a rhythm has been found in previous emotional and anxiety studies [33-36]. Similarly, fINRS results demonstrated that stress reduced the cortical activity over the right ventrolateral PFC, DLPFC and right frontopolar areas. There was a significant decrease in the oxygenated hemoglobin concentration in these regions with less sta ndard error on the right ventrolateral PFC. The less error at this particular region indicates that all participants perceived the same level of stress equally and suggest it as the most sensitive region to stress exposure. Using support vector machine (S VM) as an individual classifier, we were able to classify brain activities under stress from that of control state significantly. The performance of SVM classifiers were evaluated based on accuracy, sensitivity, specificity and the area under ROC curve. Us ing features from EEG only gives a mean classification of 88.69% , 87.6%, 89.7 % and 9 5.1% accuracy, sensitivity, specificity and area under ROC respectively. Similarly, the result obtained from the first and second lateralization channels of fNIRS (1,2) show ed on average 85.88%, 83.45%, 8 8.35% and 9 2.95% accuracy, sensitivity, specificity and area under ROC curve respectively . Decision fusion of multiple operating points of EEG+fNIRS (1,2) demonstrated significant improvements in the detection rate of mental s tress with 96.45 %, 95.10 %, 97.70% and 98.55 % accuracy, sensitivity, specificity and area under ROC respectively. The percentage of improvements in the detection rate of mental stress was in the range of 3.45% to 11.65% in all the four metrics namely ; accur acy, sensitivity, specificity and area under the ROC curve. This result support the assumption of that EEG and fNIRS provide complementary information for better stress detection. Up to this date, only few studies have looked into the possibilities of combining hemodynamic responses with their electrophysiological counterparts in a hybrid method aiming at improving the overall sy stems performance [20, 37 -42]. However, t hese studies are based on motor imagery tasks which is relatively simple and stable. Additionally, these studies used large number of EEG and fNIRS channels which take longer time for preparation, discomfort and reduce the portability . For example, Fazli et al. proposed a hybrid sensory motor rhythm combined 24 fNIRS channels and 37 EEG Electrodes . The researcher used a meta -classifier to combine the output probability of the individual classifier -modality. The results showed that the simultaneous measurement of the EEG+fNIRS can significantly improve the classification accuracy of the motor imagery by an average of +5% [20]. Lee et al. [38] used similar number of channels/electrodes and demonstrated that bimodal EEG+fNIRS signals increase the classification performance by 10% for the motor imagery task. Morioka et.al [42] used 49 channels and 64 EEG electrodes and reported that the hybrid system of EEG+f NIRS improved the spatial attention decoding accuracy by 8%. Blokland et al. [43] recently examined the principle of combining these modalities in patients with tetraplegia . The researcher used 2 channels with 8 electrodes and reported an improved accurac y of 1% in the brain s witch control for some subjects. The highest improvements of 11.65% supports the flexibility of our experiment and the robustness of the proposed fusion technique. V. CONCLUSION In this study , we investigated if decision fusion level of EEG and fNIRS signals could improve the detection rate of mental stress on the PFC. The experiment results showed that stress impaired PFC activities, specifically the right ventrolateral PFC area. The proposed fusion technique significantly improved the classification accuracy of mental stress by +10.57%, and +7.76 % compared to sole fNIRS and sole EEG, respectively. Similar improvement s were found in the sensitivity, specificity and area under ROC in stress detection. Our study showed that, albeit with le ss number of electrodes/channels, the improvements of fusion were significant, p<0.001.