# Exploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
## 5 Results  
### Main Results  
Tables 2 and 3 present the performance of different methods on the social and numeric category classification tasks, respectively. Our proposed Cross-Perception Model consistently outperformed both unimodal and existing multimodal approaches across all evaluation metrics for both classification tasks.  
In terms of unimodal performance, fMRI-based methods generally outperformed EEG-based methods, likely due to the higher spatial resolution of the fMRI data. However, both modalities struggled to achieve high accuracy when used independently, highlighting the challenge of inner speech recognition from a single modality.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Method** & **Acc \(\uparrow\) (\%)** & **F1-Score \(\uparrow\)** & **AUC-ROC \(\uparrow\)** \\ \hline EEG-SVM & 28.5 \(\pm\) 2.1 & 0.27 \(\pm\) 0.02 & 0.32 \(\pm\) 0.01 \\ EEG-RF & 30.2 \(\pm\) 1.8 & 0.29 \(\pm\) 0.02 & 0.34 \(\pm\) 0.01 \\ fMRI-MVPA & 35.8 \(\pm\) 1.5 & 0.35 \(\pm\) 0.01 & 0.38 \(\pm\) 0.01 \\ fMRI-3 DCNN & 38.3 \(\pm\) 1.3 & 0.37 \(\pm\) 0.01 & 0.40 \(\pm\) 0.01 \\ \hline EEG-fMRI-Concat & 40.5 \(\pm\) 1.2 & 0.40 \(\pm\) 0.01 & 0.42 \(\pm\) 0.01 \\ EEG-fMRI-CCA & 42.1 \(\pm\) 1.0 & 0.41 \(\pm\) 0.01 & 0.49 \(\pm\) 0.01 \\ MM-CNN & 44.7 \(\pm\) 0.9 & 0.44 \(\pm\) 0.01 & 0.55 \(\pm\) 0.00 \\ \hline Our Method & **47.2 \(\pm\) 0.7** & **0.47 \(\pm\) 0.01** & **0.56 \(\pm\) 0.00** \\ \hline \hline \end{tabular} \(\uparrow\) indicates that higher values represent better performance for the corresponding metric. Note: Bold values indicate the best performance for each metric.  
\end{table}
Table 2: Results for the social category classification task.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Method** & **Acc \(\uparrow\) (\%)** & **F1-Score \(\uparrow\)** & **AUC-ROC \(\uparrow\)** \\ \hline EEG-SVM & 17.8 \(\pm\) 2.2 & 0.16 \(\pm\) 0.02 & 0.21 \(\pm\) 0.01 \\ EEG-RF & 19.5 \(\pm\) 1.9 & 0.18 \(\pm\) 0.02 & 0.23 \(\pm\) 0.01 \\ fMRI-MVPA & 24.9 \(\pm\) 1.6 & 0.24 \(\pm\) 0.02 & 0.31 \(\pm\) 0.01 \\ fMRI-3 DCNN & 27.6 \(\pm\) 1.4 & 0.26 \(\pm\) 0.01 & 0.33 \(\pm\) 0.01 \\ \hline EEG-fMRI-Concat & 29.8 \(\pm\) 1.3 & 0.29 \(\pm\) 0.01 & 0.39 \(\pm\) 0.01 \\ EEG-fMRI-CCA & 29.3 \(\pm\) 1.1 & 0.30 \(\pm\) 0.01 & 0.41 \(\pm\) 0.01 \\ MM-CNN & 33.9 \(\pm\) 1.0 & 0.33 \(\pm\) 0.01 & 0.44 \(\pm\) 0.00 \\ \hline Our Method & **36.5 \(\pm\) 0.8** & **0.36 \(\pm\) 0.01** & **0.45 \(\pm\) 0.00** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results for the numeric category classification task.  
The combination of EEG and fMRI data led to significant improvements over unimodal approaches. This underscores the complementary nature of the temporal precision of EEG and the spatial resolution of fMRI in capturing inner speech processes. Among existing multimodal methods, MM-CNN showed the best performance, demonstrating the advantage of using deep learning for feature extraction and fusion in multimodal scenarios.  
Our Cross-Perception Model achieved further improvements over existing multimodal methods by incorporating tailored EEG and fMRI processing techniques. For the social category task, we observed a 2.5% increase in accuracy and a 0.03 increase in F1-score compared to the best-performing baseline (MM-CNN). Similar improvements were seen for the numeric category task.  
The superior performance of our Cross-Perception Model can be attributed to several factors. The use of Singular Spectrum Analysis (SSA) for EEG decomposition effectively addresses the non-stationary nature of EEG signals during inner speech. The transformation of EEG signals into Markov Transition Field (MTF) images allows for the capture of complex temporal dynamics. The specialized 3D convolutional network for fMRI feature extraction is tailored to capture both local and global spatial patterns relevant to inner speech. Finally, the adaptive fusion mechanism dynamically weighs the contributions of different modalities based on their relevance to the recognition task.