Title: VISTA: Visual-Semantic Disentanglement and Dynamic Spatial-Temporal Asynchrony for Brain Decoding
Primary area: applications to neuroscience & cognitive science
Keywords: Brain decoding, object recognition, neural signal, human computer interaction

Abstract: Electroencephalogram (EEG) offers a portable, low-cost, and millisecond-scale window into neural dynamics, making it an attractive alternative to functional magnetic resonance imaging (fMRI) for real-world brain visual decoding. Yet fine-grained visual representations and high-level semantic concepts emerge in distinct temporal intervals and spatial topology connections, creating asynchronous patterns that hinder their joint visual-semantic disentanglement. We present VISTA, an EEG-centric neural decoding framework that disentangles visual-semantic modalities along asynchronous spatial-temporal dimensions. Temporally, VISTA divides EEG into non-overlapping time patches and employs an attention mechanism to assign soft weights to each slice, enhancing EEG to capture the heterogeneous temporal distributions of visual and semantic activations. Spatially, it learns modality-specific brain topology connections and derives spatial representation via low-rank decomposition and normalized Laplacian spectral decomposition. The resulting visual and semantic embeddings are each aligned with CLIP’s image and text spaces to leverage rich pretrained knowledge. On the large-scale and widely used EEG-visual dataset THINGS-EEG, VISTA outperforms prior EEG methods in zero-shot object recognition. Moreover, on the magnetoencephalogram (MEG) dataset THINGS-MEG, it demonstrates cross-modal generality beyond EEG, achieving comparable gains. Our results underscore the value of asynchronous, disentangled feature extraction and cross-modal alignment for robust neural decoding. Code and pretrained models will be available.

TL;DR: An EEG-centric neural decoding framework that disentangles visual-semantic modalities along asynchronous spatial-temporal dimensions.