— Fusion of Functional N ear i nfrared Spectroscopy (fNIRS) and Electroencephalograph (EEG) is a novel approach. This study aims in improving the detection rate of mental st ress using the complementary nature of fNIRS and EEG neuroimaging modality. Simultaneous measurement s of fNIRS and EEG signals were conducted on 12 subjects while solving arithmetic problems under two different conditions (control and stress). The stress i n this work was based on arithmetic task difficulty, time pressure and negative feedback of individual performance. The study demonstrated significant reduction in the concentration of oxygenated hemoglobin (p=0.0032) and alpha rhythm power (p=0.02 13) on t he PFC under stress condition. Specifically, the right PFC and dorsolateral PFC were highly sensitive to mental stress. Using support vector machine (SVM), the mean detection rate of mental stress was calculated as 91%, 95% and 98% using fNIRS, EEG and fus ion of fNIRS and EEG signals respectively. Keywords—Stress , fNIRS, EEG , Fusion . I. INTRODUCTION Functional N ear-infrared Spectroscopy (fNIRS) is a non - invasive brain imaging technology based on hemodynamic responses to cortical activation . fNIRS measures blood flow through hemoglobin concentrations and tissue oxygenation in the brain [1]. It sends near-infrared lig ht into the head using two wavelengths of 695nm and 850nm . By measuring the light sent at two wavelengths, the change in oxygenated 2O Hb and deoxygenated HHb hemoglobin concentrations can be estimated usi ng modified Beer -Lambert law [2]. fNIRS has several advantages compared to other neuro imaging modalities. Compared to functional magnetic resonance imaging (fMRI) , fNIRS has good temporal resolution , portable, inexpensive and less motion artefacts [3]. Furthermore, fNIRS has a good spatial resolution c ompared to EEG. In recent years, fNIRS has attracted considerable attention as a non-invasive neuroimaging technique for the assessment of hemodynamic alterations in the brain [4]. Various cognitive tasks were performed to understand the relationship between the hemodynamic response of the prefrontal cortex and different ev ents. Mental rotation [5], word generation [6], listening to music [7], and arithmetic task [8] have shown to create a hemodynamic response in the prefrontal cortex (PFC). The change in concent ration of the 2O Hb and HHb hemoglobin has been used in previous studies to classify the brain activation fro m that of rest state. Study in [7] used a piece of music as stimuli to discriminate between the brain activation and baseline resulted in an average classification accuracy of 70%. Another studies [9, 10] used music and arithmetic tasks to investigate the consistency of single -trial classification over multip le sessions resulted in accuracy of 62.7% and 71.2% in classifying music and arithmetic tasks respectively. A comparative study on the classification of three mental tas ks was firstly conducted by[11]. The study reported an averaged classification accuracy of 71% for mental arithmetic, 70% for word generation and 62% for me ntal rotation . However, t he hemodynamic activation is an intrinsically slow response. To overcome this limitation, combining multiple neuroimaging modalities with complementing strength is the main objective. Electroencephalogram (EEG) has an excellent temporal resolution enabling it to measure cognitive changes within millisecond scale. EEG is one of the most common sources of information used to study brain function and condition and can be recorded non -invasively using surface electrodes on the scalp. Combination of fNIRS and EEG has recently introduced. Few studies have used EEG and fNIRS to study the correlation between cortical activation and the hemodynamic re sponse in human subjects [12, 13] . These studies found a positive correlation in occipital cortex between alpha act ivity and concentration changes of HHb hemoglobin . Another studies combined EEG and fNIRS signals to characterize the hemodynamic response to epileptic discharges as measured by EEG [14]. Simultaneous measurement of EEG and fNIRS has been proposed for improving the performance of brain computer interface systems (BCI) [15]. Fazly et al. reported that, the performance of a sensory motor rhythm (SMR) based BCI significantly improved by simultaneous measurement of EEG and fNIRS [16]. In this work we examined the possibility of combining the hemodynamic responses to mental stress with their physiological counterparts in a multimodal fusion technique . We aim to improve the detect ion rate of mental stress using the complementary nature of fNIRS and EEG neuroimaging modalities. II. METHODOLOGY A. Subjects Twelve healthy male right -handed adults participated in the simultaneous EEG and fNIRS measurements. All participants were informed prior to the experiment and gave written consent, in accordance with the declaration of Helsinki and ethical approval granted by local ethical committee. None of these participants had a history of psychiatric, neurological illness or psychotropic drug us e. The participants were asked to minimize their head movements and to keep calm as much as possible during the experiment. B. Experiment setup To measure the hemodynamic response we used an OT -R40 (Hitachi Medical, Japan) Optical Topography system. Brain activities recorded from the PFC cortex using integrated cap of BrainMaster (7 -Electrodes) and fNIRS (2 7 channels) as shown in figure 1. The sampling rate for EEG recording was set to 256 Hz and 10 Hz for fNIRS. The impedance of EEG was minimized using sma ll amount of gel directly to the scalp. Figure 1.The setup of multi -modal fNIRS -EEG experiment for subject performing arithmetic task. The mental stress experiment was designed based on Montreal Imaging Stress Task (MIST) [17]. The task involved three single -digit integers (ranging from 0 to 9) and the operands limited to ‘+’ and ‘ –’ (example 4 -2+2). The experiment was conducted in four phases. First, brief introduction was given to the participants. Second, all participants were trained for five minutes to estimate the average time taken to answer each question. Third (i.e. the control phase), the participants had their simultaneous EEG and fNIRS data record for total of five minutes. During the contr ol phase, all participants were instructed to answer the questions as fast as they can. Fourth, (i.e. stress phase), the averaged time recorded during the training phase was reduced by 10% as time limit set for answering questions in the stress phase to i nduce stress on the participants. Furthermore, the individual negative feedback of answering the questions was displayed on the computer monitor to further induce more stress on the participants. The designed experiment consists of four blocks as shown in figure 2. In each block, mental arithmetic task posed for 40 seconds followed by 30 seconds rest. In this experiment, we developed the control technique using MATLAB to send a marker via parallel port of channels 23 -24 of EEG brainMaster as ‘1’ to mark t he start of the task and ‘0’ for the end of the task for each block. The same marker was sent through serial port to mark the task in fNIRS system as ‘F9’ for starting the task and ‘F7’ to mark the end of the task in each of the experimental block designed . The overall time taken for the entire experiment was less than 1 8 minutes. C. Measurement Locations We investigated the EEG at seven electrode locations namely a: FP1, F3, F7, Fz, FP2, F4 and F8 with one reference electrode A1 attached to the earlobe. Thes e locations were based on the 10 -20 international system of electrod e placement . A total of 2 7 locations over the frontal lobe were initially examined with fNIRS to determine the extent to which 2O Hb concentration in the PFC was infl uenced by the mathematical task. Seven of these locations were subsequently formed the basis of the fusion in this study. Figure 2. Experimental protocol and task designed sequences. D. fNIRS data a nalysis The data collected by f NIRS passed through several pre - processing steps to filter the high frequency components and remove the motion artefacts using the plug -in analysis software Platform for Optical Topography Analysis Tool (developed by Hitachi, Japan; run on MATLAB) . The sig nals were bandpass filtered from 0.0125 to 0.8 Hz using 4th order Butterworth filter. In baseline correction, we defined a period from 5s prior to task condition to the end of the period of rest condition as the analysis experiment. Then we applied linear regression by least mean square method during the 5s period and the last 1s period to determine the linear trend of the baseline unrelated to the arithmetic task [18]. After correcting the baseline by removing the trend, we averaged the baseline corrected dat a to all the analysis blocks [19]. From the analysis blocks , we extracted th e average of the change on oxygenated hemoglobin 2O Hb using the following equation: 22 11()N n nO Hb O HbN (1) Where 2O Hb represent s the segmented oxygenated hemoglobin signal and N is the length of the signal. E. EEG Analysis The raw data from EEG was bandpass filte red between 1 Hz and 30 Hz using 3rd order Butterworth filter. The artefacts were removed using independent components analysis technique (ICA) available in EEGLAB. Then we used wavelet transform to decompose the signal into four frequency bands namely; Delta (0 -4Hz), Theta (4 -8Hz), alpha (8 -16Hz), and Beta (16 -32Hz) in the same manner of our previous study [20]. In this study, only alpha frequency band was considered for features extrac tion. We selected alpha rhythm due to its significant responses to mental stress. From the wavelet coefficients we extracted the average power of alpha frequency band using the following equation : 12 1| ( ) |kN nkP x nN  (2) where x(n) represent the segmented EEG signal and N is the length of the signal. F. Fusion of fNIRS and EEG signals The fusion of both modality was performed based on concatenating the features of both modality into a single feature vector. In this study, seven fNIRS channels wer e fused with seven EEG electrodes. The selection of channels was based on close proximity to EEG electrodes following the 10-20 system. For example, the features of channel 26 were fused with that of FP1, features of channel 24 were fused with that of FP2, features of channel 23 were fused with that of F8, features of channel 27 were fused with that of F, features of channel 1 were fused with that of F4, features of channel 3 were fused with that of Fz and features of channel 5 were fused with that of F3. A ll features were concatenated to form a single feature vector which later go for classification. In short, the fusion here was based on spatial location of Electrodes placement. G. Classification Signal classification was performed on the fNIRS and EEG signals following the processing steps described previously. We aim to classify the activity into one of two classes; ‘stress’ and ‘control’. The classification was done in two phases (individually and after fusion) . First, fNIRS and EEG modality classified se parately. Second, the simultaneous bi -modal (fNIRS -EEG ) was classified after fusing their features and their performance was compared to individual modality. In this work, we used support vector machine (SVM) classifier and calculated the classification ac curacy via leave -one-out cross -validation. In particular, for a number of trials (N), N -1 trials were used for training and the remaining 1 trial used for testing. III. RESULT AND DISCUSSION A. fNIRS With regard to blood flow in the PFC, the level of oxygenated haemoglobin dramatically increased during a control task (compared to baseline) and significantly reduced during a stress task (compared to control condition) in all the subjects. Figure 3 and figure 4 show the topographical map of oxygenated haemoglobin concentration during the control task and stress task respectively for averaged of 12 subjects . From figure 4, the reduction of oxygenated haemoglobin concentration was much localized to the right PFC and dorsolateral PFC. This indicated that, the right PF C and dorsolateral PFC are the most sensitive brain region to the detrimental effects of mental stress. The study concluded that, with increasing time pressure (i.e. stress condition), the blood flow in the prefrontal area and the information processing abilities significantly decreased. Figure 3. Topographical map of oxygenated h emoglobin concentration under control condition for average of 12 subjects. Red colour indicates high concentration levels of oxygenated haemoglobin and blue colour indicated less level of oxygenated haemoglobin concentration . Figure 4. Topographical map of oxygenated hemoglobin concentration under stress condition for average of 12 subjects. Red color indi cates high concentratio n level of oxygenated hemoglobin and blue color indicated less oxygenated hemoglobin concentration . Using two sample t test, the significant differences between the control task and stress task was measured with average p- value of 0.0032. B. EEG EEG results demonstrated significant decrease d in alpha rhythm power under stress task in all the recorded electrodes positions . The decrease d in EEG alpha rhythm power reflect the increase of stress level on the PFC . Figure 5 show s the mean alpha power distribution across all the recorded electrodes under the control and stress task for average of 12 subjects. Using two sample t test, the statistical analysis demonstrated significant reduction in alpha rhythm power under stress task with mean p-value of 0.0213. Figure 5. Normalized alpha rhythm power in all the recorded electrodes position (PFC area) for average of 12 subjects. Blue colour indicated the average alpha power under control task and red colour indicated the average alpha power under stress task. The classification accuracy obtained from fNIRS signals was measured as 9 1% and the classification accuracy obtained from EEG signals was measured as 9 5% as shown in figure 6 . However, the results of the classification of both modalitie s reported significant improvement in the detection rate of mental stress as compared to single modality. Under fusion of both modalities, the study reported 98% classification accuracy indicating that fNIRS and EEG modality complement each other in obtain ing features that highly correlated with mental stress. This improvement is in line with our previous stress studies [21-32]. Figure 6. Boxplot of the classification accuracy obtained using individual and combined modalities. IV. CONCLUSION This study investigated the effectiveness of simultaneous measure of EEG and fNIRS in studying the effects of mental stress to PFC area . The results showed that, the oxygenated haemoglobin concentration and the alpha rhythm power significant reduced under stress condition. Furthermore, the study reported that, co mbination of both modalities improve the performance of mental stress detection rate by +3% . We concluded that, the fusion technique revealed the complementary nature of both modalities in detecting features that highly correlated to mental stress. ACKNOW LEDGMENT This research is funded by the Ministry of Education, Malaysia under the HiCOE grant to CISIR, UTP.