fMRI-3D: A Comprehensive Dataset for Enhancing fMRI-based 3D
  Reconstruction
Authors: name: Jianxiong Gao; name: Yuqian Fu; name: Yun Wang; name: Xuelin Qian; name: Jianfeng Feng; name: Yanwei Fu
Published: 2024-09-17 16:13:59

Abstract: Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI)
data, introduced as Recon3DMind in our conference work, is of significant
interest to both cognitive neuroscience and computer vision. To advance this
task, we present the fMRI-3D dataset, which includes data from 15 participants
and showcases a total of 4768 3D objects. The dataset comprises two components:
fMRI-Shape, previously introduced and accessible at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse,
proposed in this paper and available at
https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse
includes data from 5 subjects, 4 of whom are also part of the Core set in
fMRI-Shape, with each subject viewing 3142 3D objects across 117 categories,
all accompanied by text captions. This significantly enhances the diversity and
potential applications of the dataset. Additionally, we propose MinD-3D, a
novel framework designed to decode 3D visual information from fMRI signals. The
framework first extracts and aggregates features from fMRI data using a
neuro-fusion encoder, then employs a feature-bridge diffusion model to generate
visual features, and finally reconstructs the 3D object using a generative
transformer decoder. We establish new benchmarks by designing metrics at both
semantic and structural levels to evaluate model performance. Furthermore, we
assess our model's effectiveness in an Out-of-Distribution setting and analyze
the attribution of the extracted features and the visual ROIs in fMRI signals.
Our experiments demonstrate that MinD-3D not only reconstructs 3D objects with
high semantic and spatial accuracy but also deepens our understanding of how
human brain processes 3D visual information. Project page at:
https://jianxgao.github.io/MinD-3D.