# Exploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
###### Abstract  
Multimodal brain signal analysis has shown great potential in decoding complex cognitive processes, particularly in the challenging task of inner speech recognition. This paper introduces an innovative I Inner Speech Recognition via Cross-Perception (ISRCP) approach that significantly enhances accuracy by fusing electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data. Our approach comprises three core components: (1) multigranularity encoders that separately process EEG time series, EEG Markov Transition Fields, and fMRI spatial data; (2) a cross-perception expert structure that learns both modality-specific and shared representations; and (3) an attention-based adaptive fusion strategy that dynamically adjusts the contributions of different modalities based on task relevance. Extensive experiments on the Bimodal Dataset on Inner Speech demonstrate that our model outperforms existing methods across accuracy and F1 score. inner speech recognition; cross-perception; EEG; fMRI; deep learning; computational perception +
Footnote †: journal: Medical  
poral characteristics and lower temporal resolution compared to natural images, requiring specialized processing techniques [11; 12; 13; 14]. Therefore, many researchers have found that there are complementary and enhanced representations between multimodal data, and they have used multimodal data for pattern recognition of biological signals and successfully built models with stronger performance. For example, [PERSON] et al. [15] combined EEG and fMRI data to improve the classification of emotional states, achieving higher accuracy than unimodal approaches. [PERSON] et al. [16] integrated EEG and fNIRS (functional near-infrared spectroscopy) for motor imagery classification, demonstrating superior performance over single-modality methods. In the context of speech processing, [PERSON] et al. [17] fused EEG and audio features for robust speech recognition in noisy environments. However, existing fusion methods may not fully leverage the complementary information provided by EEG and fMRI in the specific context of inner speech recognition. This is because the temporal dynamics of EEG and the spatial patterns of fMRI require careful alignment and integration to capture the full complexity of inner speech processes [18; 19; 20]. The recent release of the Bimodal dataset on Inner Speech [21] provides a unique opportunity to explore multimodal approaches to inner speech recognition. This dataset combines EEG and fMRI recordings from participants performing an inner speech task, offering a rich resource for developing and evaluating multimodal decoding methods. In this paper, we propose a novel cross-perception model for multimodal inner speech recognition that leverages both EEG and fMRI data. Our proposed approach is shown in Figure 1.  
Figure 1: Inner speech using EEG and FMRI data. By leveraging the strengths of both EEG and fMRI, our cross-perception model aims to achieve more accurate and robust inner speech recognition compared to unimodal approaches. We evaluate our model on the Bimodal dataset on Inner Speech, demonstrating its effectiveness in decoding inner speech across different semantic categories. The main contributions of this work are as follows:  
* We propose a novel cross-perception model that effectively integrates EEG and fMRI data for inner speech recognition. * We introduce a multigranularity encoding scheme that captures both temporal and spatial aspects of brain activity during inner speech. * We develop an adaptive fusion mechanism that dynamically weights the contributions of different modalities based on their relevance to the recognition task. * We provide extensive experimental results and analyses, demonstrating the superiority of our multimodal approach over unimodal baselines. The rest of the paper is organized as follows: Section 2 reviews related work in inner speech recognition, including unimodal and multimodal approaches, and it discusses the limitations of existing methods.