SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two-way identification refers to percent correct across comparisons gauging if the original image embedding is more similar to its paired brain embedding or a randomly selected brain embedding. Comparison was performed for AlexNet [37] (second and fifth layers), InceptionV3 [38] (last pooling layer), and CLIP (final layer of ViT-L/14). We use the same settings as Ozcelik and VanRullen [4] for our metrics. For more details refer to Appendix A.6.
3.3 Ablations
In this subsection we try to explain where MindEye performance improvements come from through ablations. To study the effects of architectural changes and training strategies we train only the retrieval pipeline (no diffusion prior) for 120 epochs with batch size 300. All models in this section are trained on Subject 1. Table entries with * correspond to the final version of MindEye's settings.
Architectural Improvements: To study the effect of model depth and parameter count we train multiple MLPs of various sizes (Table 2). Among models that map to the last hidden layer of CLIP ViT-L/14, we observe a clear trend of increased performance with added residual blocks. For 2 blocks, the effect of skip connections is not too significant but at 4 blocks the model does significantly worse without them, indicating that skip connections are important for training deeper models.
Method	Low-Level	High-Level	Retrieval
PixCorr ↑	SSIM ↑	Alex(2) ↑	Alex(5) ↑	Incep ↑	CLIP ↑	Eff ↓	SwAV ↓	Image ↑	Brain ↑
Lin et al. [11]	-	-	-	-	78.2%	-	-	-	11.0%	49.0%
Takagi... [3]	-	-	83.0%	83.0%	76.0%	77.0%	-	-	-	-
Gu et al. [28]	.150	.325	-	-	-	-	.862	.465	-	-
Ozcelik... [4]	.254	.356	94.2%	96.2%	87.2%	91.5%	.775	.423	21.1%	30.3%
MindEye	.309	.323	94.7%	97.8%	93.8%	94.1%	.645	.367	93.6%	90.1%
MindEye (Low-Level)	.360	.479	78.1%	74.8%	58.7%	59.2%	1.00	.663	-	-
MindEye (High-Level)	.194	.308	91.7%	97.4%	93.6%	94.2%	.645	.369	93.6%	90.1%
MindEye (LAION)	.130	.308	84.0%	92.6%	86.9%	86.1%	.778	.477	-	-
Ozcelik... (Low-, S1)	.358	.437	97.7%	97.6%	77.0%	71.1%	.906	.581	-	-
MindEye (Low-, S1)	.456	.493	87.1%	84.1%	61.6%	62.4%	.992	.638	-	-
Table 1: Quantitative comparison of MindEye retrieval and reconstruction performance against other models. Top and middle sections average across the same 4 participants (see Appendix A.8 for individual subject models), except Lin et al. [11] which only analyzed Subject 1. Middle section reflects outputs from only the high- or low-level pipeline, and metrics when evaluating images retrieved from LAION-5B. Bottom section compares our low-level reconstructions to the low-level reconstructions from Ozcelik and VanRullen [4] which only reported metrics for Subject 1. Image retrieval refers to the percent of the time the correct image was retrieved out of 300 candidates, given the associated brain sample (chance  $= 0.3\%$ ); vice-versa for brain retrieval. PixCorr=pixelwise correlation between ground truth and reconstructions; SSIM=structural similarity index metric [34]; EfficientNet-B1 ("Eff") [35] and SwAV-ResNet50 ("SwAV") [36] refer to average correlation distance; all other metrics refer to two-way identification (chance  $= 50\%$ ). Missing values are from papers not reporting all metrics or metrics being non-applicable. We followed the same image preprocessing as Ozcelik and VanRullen [4]. Previous state-of-the-art Ozcelik and VanRullen [4] results are directly comparable to MindEye as the same test set and Versatile Diffusion model were used. Bold indicates best performance within sections.
Method	Param Count	Image Retrieval	Brain Retrieval
No ResBlocks	873M	0.880	0.820
2 ResBlocks + No Skip	907M	0.881	0.822
2 ResBlocks	907M	0.886	0.837
4 ResBlocks + No Skip	940M	0.836	0.767
4 ResBlocks*	940M	0.896	0.822
4 ResBlocks + Only CLS	135M	0.611	0.576
Table 2: Effects of varying the architecture of the MLP backbone on retrieval accuracy.
We also show a comparison with a 4-resblock model that maps to the final layer of CLIP (only the CLS classification token). This model has  $7 \times$  fewer parameters and does much worse than all other models. This indicates two things: (1) MindEye strongly benefits from a large parameter count MLP backbone and does not overfit even in the sample constrained settings of the NSD dataset, and (2) the fMRI voxels contain fine-grained information about images, allowing us to effectively predict all 257 CLIP image embeddings instead of just the CLS token.
Training Strategies (Losses and Data Augmentations): We observe that with InfoNCE, MindEye only does well on brain retrieval (Table 3). A similar trend was observed in Lin et al. [11]. We attribute this to InfoNCE being a one-sided loss that only optimizes for one retrieval objective. Simply replacing InfoNCE with CLIP loss significantly improves image retrieval. MixCo augmentation helps both unidirectional and bidirectional losses.
Method	Image Retrieval	Brain Retrieval
InfoNCE	0.237	0.784
CLIP Loss	0.837	0.791
InfoNCE + MixCo	0.303	0.856
CLIP Loss + MixCo (BiMixCo)	0.884	0.841
SoftCLIP Loss	0.837	0.816
BiMixCo + SoftCLIP (MindEye)*	0.896	0.822
Table 3: Effects of different losses and MixCo augmentation on MLP retrieval performance.
We also show the effect of training with our SoftCLIP loss. SoftCLIP improves over hard CLIP loss for brain retrieval but performs worse than BiMixCo. Our training regime combining SoftCLIP with BiMixCo gives the best image retrieval performance.
Reconstruction Strategies: To demonstrate the need for a separate diffusion prior, we train a version of MindEye where both contrastive and MSE losses are applied to the outputs of the MLP backbone. We observe that this model does poorly in terms of retrieval metrics, indicating a tradeoff between retrieval and reconstruction objectives where it is difficult to learn a single embedding space that satisfies both objectives. Inspired by recent works in self-supervised learning [39-42], we decouple these losses using a separate MLP projector, where MSE loss is applied to the outputs of the MLP backbone and contrastive loss is applied to the outputs of the projector. This model does slightly worse in terms of reconstruction but is much better at retrieval. Finally, we train a model with a diffusion prior but no MLP projector. Contrastive loss is computed for the MLP backbone and MSE loss is computed for the diffusion prior. This model is comparable to high-level MindEye in terms of reconstruction but does worse in retrieval, giving further evidence of a tradeoff. Example reconstructions for these models are in Appendix Figure 8.
Method	Low-Level	High-Level	Retrieval
PixCorr	SSIM	Alex(2)	Alex(5)	Incep	CLIP	Image	Brain
Only MLP Backbone	0.119	0.346	73.8%	84.1%	81.5%	82.6%	0.133	0.631
Backbone + Projector	0.154	0.296	73.2%	85.2%	75.2%	77.3%	0.888	0.849
Backbone + Prior	0.206	0.303	92.1%	97.2%	94.8%	95.1%	0.934	0.901
MindEye (only BiMixCo)	0.195	0.290	91.1%	96.6%	93.7%	94.4%	0.974	0.942
MindEye (0.33 BiMixCo)*	0.198	0.302	91.6%	96.8%	94.6%	95.0%	0.972	0.960
Table 4: Effects of diffusion prior and MLP projector on reconstruction and retrieval metrics.
In the 2000s, researchers demonstrated that visual information, such as spatial position [43], orientation [44, 45], and coarse image category [46, 47] could be decoded from fMRI signals using linear classifiers. With the introduction of generative adversarial networks [48], more sophisticated decoding became feasible as researchers mapped brain activity to the latent space of these models to reconstruct handwritten digits [49], human faces [50, 51], and natural scenes [52, 5, 53]. More recently, with the release of multimodal contrastive models like CLIP [7], diffusion models [54, 55] like Stable Diffusion [9], and new large-scale fMRI datasets [26], fMRI-to-image reconstructions have reached an unprecedented level of quality [4, 3, 28].
Lin et al. [11] reconstructed NSD images by mapping voxels to CLIP space (see also Wang et al. [56]) and fed outputs through a fine-tuned Lafite [32] GAN (MindEye reconstructions using Lafite in Appendix A.7). Differences from MindEye include using a convolutional model, no projector to separate contrastive loss from MSE loss, InfoNCE instead of CLIP loss, fine-tuning of a pretrained GAN, no diffusion prior, and mapping to both CLIP image and text space. Ozcelik and VanRullen [4] used a low- and high-level pipeline with Versatile Diffusion [6]. Differences include mapping to CLIP space via ridge regression, no contrastive learning or diffusion prior, and mapping to a VDVAE [57] for low-level reconstructions. Gu et al. [28] used a low- and high-level pipeline and extended on Ozcelik et al. [5] by reconstructing with IC-GAN [58]; they did not flatten voxels and mapped to SwAV [36] features with surface-based convolutional networks. Takagi and Nishimoto [3] used ridge regression to map to Stable Diffusion latents and CLIP text latents, using different voxel selections for different components. Mind-Vis [59] did not use the Natural Scenes Dataset but tackled the fMRI-to-image problem from the unique angle of first pre-training a masked brain model on a separate large-scale dataset. This enabled the authors to use a more informative latent as the input to their image reconstruction model. Mind-Video [60] subsequently extended on the Mind-Vis approach by reconstructing video rather than images. Overall, MindEye is unique in its use of reconstruction and retrieval submodules, a deep MLP backbone with 940 million parameters (parameter size comparison in Table 10), a diffusion prior for more accurate translation across brain and image modalities, and state-of-the-art reconstruction and retrieval results.
We present MindEye, a novel mental decoding approach that achieves state-of-the-art reconstructions of natural scenes presented to humans in the MRI machine. These reconstructions retain semantic and perceptual similarity to the original images due to the use of a combined high-level and low-level pipeline. The novel use of specialized submodules for contrastive-based retrieval and diffusion-based reconstruction allows MindEye to learn mappings for both tasks in parallel. MindEye brain latents contain fine-grained image-specific signal as demonstrated by the ability to select ground truth images out of a set of nearly 1,000 possible images (see Figure 3). We leveraged pretrained CLIP [7], a model trained with billions of image and text data samples, as a teacher to guide MindEye where we have a relative scarcity of brain data (less than 30,000 training samples per participant). Our diffusion prior submodule is trained from scratch and allows for accurate translation of brain embeddings into pretrained CLIP space such that any model that accepts CLIP image embeddings can be provided with CLIP fMRI embeddings without fine-tuning. This flexibility suggests that MindEye reconstructions will continue to improve as newer, more powerful image generation models are released.
Privacy Concerns & Societal Benefits: The ability to accurately reconstruct perception from brain activity prompts questions about broader societal impacts. For instance, it should be possible to generalize current reconstruction models from perception to mental imagery without training a new model [61-64]. However, current models are not capable of across-subject decoding and each NSD participant spent up to 40 hours in the MRI machine to procure sufficient training data (see Appendix 9 for results using a subset of the total training data). Furthermore, non-invasive neuroimaging methods in general require compliance because participants can easily resist decoding by moving their head or thinking about unrelated information [65]. MindEye is also limited to natural scenes such as those in MS-COCO; for other image distributions additional data collection and specialized generative models would be required. While high-quality image reconstruction via non-invasive neuroimaging is not currently practical for real-world applications, technology is constantly improving and it is important that brain data be carefully protected and companies collecting such data be transparent with their use.
Image reconstruction from brain activity can enable various potential societal benefits. Reconstructions are expected to be systematically distorted due to mental state, neurological conditions, etc. This could potentially enable novel clinical diagnosis and assessment approaches. For example, patients suffering from major depressive disorder might produce reconstructions where emotionally negative aspects of images are more salient [66]. MindEye results also suggest potential for improved locked-in (pseudocoma) patient communication via fine-grained visual communication beyond simple classification [67], as well as brain-computer interface performance if adapted to real-time fMRI analysis [68] or non-fMRI neuroimaging modalities.
Future Directions: Future directions we wish to explore include mapping individual subjects to a shared embedding space to enable training models that are generalizable across people (e.g., [69, 70]), and exploring model interpretability approaches like GradCAM [71] or Network Dissection [72] to identify the fMRI voxels that most strongly respond to the presence of certain image features.
6 Open Research:  $100\%$  Transparent Volunteer-Driven Science
MindEye was openly developed through volunteer contributions in the MedARC Discord server. Source code was always accessible via a public GitHub repository throughout the lifespan of the project. Research discussions were held via public Discord channels, and weekly video conference calls were recorded and shared publicly. We continue to extend a global invitation to contribute to MedARC Neuroimaging & AI Lab projects to cultivate an internationally diversified, volunteer-driven research team composed of members from varied backgrounds possessing a wide array of expertise. We contend that fully transparent open-research initiatives such as this and others like EleutherAI, LAION, OpenBioML, and ML Collective could redefine the traditional framework of scientific research, democratizing entry into machine learning and medical research through the harnessing of crowd-sourced collective intelligence and community collaboration.
7 Author Contributions
For detailed author contributions see Appendix A.1.
Thanks to the MedARC community, including Jeremy Howard, Tommaso Furlanello, Mihir Tripathy, and Cesar Torrico for useful discussion and reviewing the manuscript. Thank you to Furkan Ozcelik, author of Brain-Diffuser, for sharing his code and expert knowledge with our group. We thank LAION for being the initial community where this project developed, and thank Romain Beaumont and Zion English for useful discussion during that time. We thank Stability AI for sharing their high-performance computing workplace and giving us the computational resources necessary to develop MindEye. Thank you to Richard Vencu for help navigating the Stability HPC. Collection of the Natural Scenes Dataset was supported by NSF IIS-1822683 and NSF IIS-1822929.
[1] Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fMRI. Neuroimage, 56(2):400-410, 2011. Publisher: Elsevier.
[2] David Linden. Introduction. In fMRI Neurofeedback, pages 161-169. Elsevier, 2021. ISBN 978-0-12-822421-2. doi: 10.1016/B978-0-12-822421-2.00008-9. URL https://linkinghub.elsevier.com/retrieve/pii/B9780128224212000089.
[3] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. preprint, Neuroscience, November 2022. URL http://biorxiv.org/lookup/doi/10.1101/2022.11.18.517004.
[4] Furkan Ozcelik and Rufin VanRullen. Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion, March 2023. URL http://arxiv.org/abs/2303.05334.arXiv:2303.05334 [cs, q-bio].
[5] Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs, February 2022. URL http://arxiv.org/abs/2202.12692.arXiv:2202.12692 [cs, eess, q-bio].
[6] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile Diffusion: Text, Images and Variations All in One Diffusion Model, March 2023. URL http://arxiv.org/abs/2211.08332. arXiv:2211.08332 [cs].
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv.org/abs/2103.00020. arXiv:2103.00020 [cs].
[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022. URL http://arxiv.org/abs/2204.06125.arXiv:2204.06125 [cs].
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxiv.org/abs/2112.10752.arXiv:2112.10752 [cs].
[10] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, January 2022. URL http://arxiv.org/abs/2108.01073.arXiv:2108.01073 [cs].
[11] Sikun Lin, Thomas Sprague, and Ambuj K. Singh. Mind Reader: Reconstructing complex images from brain activities, September 2022. URL http://arxiv.org/abs/2210.01769.arXiv:2210.01769 [cs, eess, q-bio].
[12] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech from non-invasive brain recordings, August 2022. URL http://arxiv.org/abs/2208.12266.arXiv:2208.12266 [cs, eess, q-bio].
[13] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-368, May 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06031-6. URL https://www.nature.com/articles/s41586-023-06031-6. Number: 7960 Publisher: Nature Publishing Group.
A.1 Author Contributions
PSS devised the project, led the team, developed the models, drafted the manuscript, and otherwise contributed to all parts of MindEye development. AB drafted the manuscript, developed the models, and contributed to all parts of MindEye development including creating the low-level pipeline, conception of BiMixCo and soft CLIP loss, and modification of the DALL-E 2 diffusion prior. JG developed the models, tracked/compared model variants, and significantly contributed to the MindEye codebase. SS conceived of and implemented LAION-5B retrieval using the CLIP Retrieval client and conducted various exploratory experiments. AN implemented the Lafite pipeline for MindEye reconstructions. EC conducted various initial explorations into using a diffusion prior for aligning voxels to CLIP space. AJD created the initial webdatasets used to train MindEye and created various model architectures to compare different mapping approaches. NV conducted various exploratory experiments mapping voxels to StyleGAN-XL [73] latent space. EY shared code to automatically identify identical images for qualitative comparisons and added code to ensure LAION-5B retrieval did not retrieve ground truth images. DW conducted various exploratory experiments and helped with project discussions. KAN oversaw the project and contributed valuable feedback. TMA oversaw the project, conducted initial explorations using VQGAN [74], and helped keep the project on-track through MedARC and Stability AI communication.
A.2 Additional Dataset Information
The Natural Scenes Dataset (NSD) [26] is a public 7-Tesla fMRI dataset containing the brain responses of several human participants each spending up to 40 hours in the MRI machine passively viewing images. These square-cropped images of natural scenes were sourced from the MS-COCO dataset [27]. Each of 9,000-10,000 unique images was presented for three seconds at a time, shown three times across 30-40 scanning sessions, totaling 22,000-30,000 trials of fMRI responses per participant. fMRI responses correspond to session-wise z-scored single-trial betas output from GLMSingle [75]. Following the procedure used in other reconstruction studies that used NSD [5, 28, 3], we train individual-subject models for the four participants who completed all scanning sessions (participants 1, 2, 5, and 7) and used a test set corresponding to the shared 1,000 images presented to every participant. This yields a dataset consisting of 24,980 training samples and 2,770 test samples—we average across the three same-image repetitions for the test set (leaving 982 test samples) but not the training set, similar to Takagi and Nishimoto [3]. We use preprocessed flattened fMRI voxels in 1.8-mm native volume space corresponding to the "nsdgeneral" brain region, defined by the NSD authors as the subset of voxels in posterior cortex most responsive to the visual stimuli presented (between 13,000 to 16,000 voxels per participant). MindEye was developed using a training and validation set of Subject 1's data, with the test set (and other subjects' data) untouched until final training of models.
A.3 MindEye Architecture
PyTorch code for the MLP backbone and projector is depicted in Algorithm 1. Specifics on how we modified the open-source implementation of the DALL-E 2 diffusion prior are discussed in A.3.1.
A.3.1 Modifications from DALL-E 2 Diffusion Prior
The inputs for the diffusion prior are 257 backbone embeddings, 1 timestep embedding, and 257 noised CLIP image embeddings, and the output is 257 denoised CLIP image embeddings. Unlike the DALL-E 2 prior, we do not use learnable queries and instead directly predict denoised CLIP embeddings from the noised embeddings. This significantly saves on memory and allows us to train the backbone and prior end-to-end on a single GPU. We observe that adding absolute positional embeddings to the noised CLIP embeddings improves performance in the absence of learnable queries. We also observe that our prior can work with just 100 timesteps instead of 1000 as used in DALL-E 2. This makes our prior much faster at inference time. We conducted experiments with both causal and bidirectional attention and did not observe any significant difference in reconstruction performance. For simplicity we use bidirectional attention in our final model.
Algorithm 1 PyTorch code for MindEye MLP backbone and MLP projector
class BrainMLP(nnModule): def__init__(self,out_dim  $= 257*768$  ,in_dim  $= 15724$  ,clip_size  $= 768$ $\mathrm{h} = 4096)$  super(）.__init_（） #in_dim corresponds to the subject-specific #number of voxels in the "nsdgeneral" brain region. self.lin0  $=$  nn Sequential( nn.Linear(in_dim,h,bias  $=$  False), nn.layersNorm(h), nn.GELU(inplace  $\equiv$  True), nnDDDropout(0.5)) self.mlp  $=$  nnModuleList([ nnSequential( nn.Linear(h,h,bias  $=$  False), nn.layersNorm(h), nn.GELU(inplace  $\equiv$  True), nnDDDropout(0.15) )for_in range(4)]self.lin1  $=$  nn.Linear(h,out_dim,bias  $\equiv$  True) self.proj  $=$  nn Sequential( nngLernNorm(clip_size), nn.GELU(inplace  $\equiv$  True), nn.Linear(clip_size,2048,bias  $=$  False), nngLernorm(2048), nn.GELU(inplace  $\equiv$  True), nn.Linear(clip_size,2048,bias  $=$  False), nngLernorm(2048), nn.GELU(inplace  $\equiv$  True), nn.Linear(2048,clip_size,bias  $\equiv$  True)) self.clip_size  $=$  clip_size def forward(self,x): x  $=$  self.lin0(x) residual  $= x$  for res_block in range(len(self.mlp)): x  $=$  self.mlp[res_block](x) x  $+ =$  residual residual  $= x$  diffusion_prior_input  $=$  self.lin1(x).reshape(len(x),-1,selfclip_size) disjointed Clip_fmri  $=$  self.proj(diffusion_prior_input) return diffusion_prior_input,disjointed Clip_fmri
A.3.2 Low-Level Pipeline: Mapping to Stable Diffusion Variational Autoencoder