MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
Authors: name: Paul S. Scotti; name: Mihir Tripathy; name: Cesar Kadir Torrico Villanueva; name: Reese Kneeland; name: Tong Chen; name: Ashutosh Narang; name: Charan Santhirasegaran; name: Jonathan Xu; name: Thomas Naselaris; name: Kenneth A. Norman; name: Tanishq Mathew Abraham
Published: 2024-03-17 13:15:22

Abstract: Reconstructions of visual perception from brain activity have improved
tremendously, but the practical utility of such methods has been limited. This
is because such models are trained independently per subject where each subject
requires dozens of hours of expensive fMRI training data to attain high-quality
results. The present work showcases high-quality reconstructions using only 1
hour of fMRI training data. We pretrain our model across 7 subjects and then
fine-tune on minimal data from a new subject. Our novel functional alignment
procedure linearly maps all brain data to a shared-subject latent space,
followed by a shared non-linear mapping to CLIP image space. We then map from
CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP
latents as inputs instead of text. This approach improves out-of-subject
generalization with limited training data and also attains state-of-the-art
image retrieval and reconstruction metrics compared to single-subject
approaches. MindEye2 demonstrates how accurate reconstructions of perception
are possible from a single visit to the MRI facility. All code is available on
GitHub.