CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic
  Audiovisual Narrative Processing
Authors: name: Jianxiong Gao, name: Yichang Liu, name: Baofeng Yang, name: Jianfeng Feng, name: Yanwei Fu

In this paper, we introduce CineBrain, the first large-scale dataset
featuring simultaneous EEG and fMRI recordings during dynamic audiovisual
stimulation. Recognizing the complementary strengths of EEG's high temporal
resolution and fMRI's deep-brain spatial coverage, CineBrain provides
approximately six hours of narrative-driven content from the popular television
series The Big Bang Theory for each of six participants. Building upon this
unique dataset, we propose CineSync, an innovative multimodal decoding
framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural
Latent Decoder. Our approach effectively fuses EEG and fMRI signals,
significantly improving the reconstruction quality of complex audiovisual
stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a
comprehensive evaluation protocol that assesses reconstructions across semantic
and perceptual dimensions. Experimental results demonstrate that CineSync
achieves state-of-the-art video reconstruction performance and highlight our
initial success in combining fMRI and EEG for reconstructing both video and
audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.