Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Authors: Paul Steven Scotti
Atmadeep Banerjee
Jimmie Goode
Stepan Shabalin
Alex Nguyen
Cohen Ethan
Aidan James Dempster
Nathalie Verlinde
Elad Yundler
David Weisberg
Kenneth Norman
Tanishq Mathew Abraham
Venue: NeurIPS 2023 spotlight (2023)
Keywords: fMRI
computational neuroscience
mind reading
diffusion models
TL;DR: We present MindEye, a novel state-of-the-art fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity using contrastive learning and diffusion models.

Abstract: We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.

Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Paul S. Scotti Atmadeep Banerjee" Jimmie Goode Stepan Shabalin', Alex Nguyen', Ethan Cohen², Aidan J. Dempster', Nathalie Verlinde', Elad Yundler', David Weisberg!?
Kenneth A. Norman +.,, and Tanishq Mathew Abraham +.2.77
Princeton Neuroscience Institute Medical AI Research Center (MedARC)
Ecole Normale Supérieure, PSL University
University of Toronto
Hebrew University of Jerusalem
EleutherAAl
Stabillity AI
Project Page: https://medarc. wwt.aarc...mminnn ai/mindeye/
Abstract
We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity any high dimensional multimodal latent space, like CL.IP image space, enabling image re- construction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-ar performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates, indicating that its brain embeddings retain fine-grained mmge-speccfiic information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that Mind- Eye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub
Introduction
The problem of decoding environmental inputs and cognitive states from brain activity is funda- mental to the field of neuroscience, where improved computational approaches allow for further understanding of brain mechanisms [ A neuroimaging methodology that has significant success in this domain is functional magnetic resonance imaging (fMRI,, where neural activity is measured by detecting changes in blood oxygenation. fMRI decoding is already being used in
"Equal contributions.
1 Core contribution. Joint senior authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

image
Reconsmuation
image
Reconstration
image
Reconetructio
image
Reconstructio
Seen image
Reconstructin
Figure I: Example images reconstructed from human brain activity corresponding passive viewing of natural scenes. Reconstructions depict oulputs from Versatile Diffusion [6] given CLIP IMRI embeddings generated by MindEye for Subject 1. See Figure 4 and Appendix A.4 for more samples.
real-time clinical domains [2] and has potential for novel mind reading applications in computr interfaces. Previous works mapped fMRI activity to the embeddings of image generation models via relatively simple mappings, usually ridge regression 5] Here we propose MindEye, a novel approach that involves mapping via large-scale multilayer perceptrons (MLPs), contrastive learning, and diffusion models to achieve state-of-the-ar image reconstruction. See Figure for select samples of reconstructions
MindEye learns to map flattened spatial patterns of fMRI activity across voxels (3-dimensional cubes of cortical tissue) the image embedding latent space of pretrained CLIP 7 model. MindEye has an MLP backbone and specialized submodules for retrieval and reconstruction. The retrieval submodule is contrastively trained and produces "disjointed CLIP fMRI" embeddings that have high cosine similarity with the corresponding image embeddings but differ in magnitude. To reconstruct images, we train diffusion prior [8] from scratch to take in the outputs from the MLP backbone and produce aligned embeddings suitable as inputs to any pretrained image generation model that accepts CLIP image embeddings. In order ensure that reconstructions also match the original images' low-level features (e.g., color, texture, spatial position), we train separate encoder that directly maps voxels to the embedding space of Stable Diffusion's [9] variational autoencoder (VAE), obtaining blurry image reconstructions that lack high-level semantic content but perform state-of-the-arr on low- level image metrics. Combining the high-level "semantic" pipeline with the low-level "perceptual" pipeline in an img2img [0] setting allows MindEye to output state-of-the-aa reconstructions across both low- and high-level image metries.
In addition to image reconstruction metrics, our disjointed CLIP fMRI embeddings attain state-of- the-art performance image retrieval and brain retrieval metrics. Image retrieval refers to finding the original seen image out of a pool of other images given a brain sample, while brain retrieval refers to finding the brain sample given an image. MindEye finds exact (top-1) matches in the pool of NSD test samples with >90% accuracy for both image and brain retrieval, outperforming previous state-of-the-art [ 4 which showed <50% retrieval accuracies. These results demonstrate that MindEye brain embeddings possess fine-grained exemplar-level signal.
Our main findings are: (1) Specialized submodules for retrieval (using contrastive learning) and reconstruction (using a diffusion prior) enable a single model to achieve state-of sfate-of-ther of-the-art results across both tasks even though the objectives exhibit tradeoff. (2) Mapping to deep MLP with parameter count orders of magnitude higher than previous methods does not produce overfitting and instead directly benefits model performance. (3) A novel bidirectional version of mixup contrastive data augmentation further improves model performance this low-sample setting. (4) State-of-the-at reconstructions for low-level image metrics can be obtained by independently mapping to Stable Diffusion's s VAE latent space. (5) IMRI-to-image retrieval can find the exact original image even among highly similar candidates, suggesting that fine-grained image-specific information is contained in brain embeddings, thus allowing retrieval to be scaled up to large-scale databases like LAION-5B output images without generative models.
IImages containing each subject's 982 test set reconstructions and retrievals available GitHub

---

High-level Isemantica pizeline
images
Digjoknsed CLIF F-RMI natriwaal Algned CUPP-MRI recoestructio)
IMRI vokets
Difusion prior
CLIP-MRI
Reconstruction
Nx 15000
Versatie
MLP backbone
Diflusion
N x 257 766
CLIP VIT-L11
Varafions
projector 768
CL CLIP-Image
N x 257 x 788
Contrastive loss
Hetrioval from LAXONI-E
img2ing
MSE loss
Low-level (perosaluall pioelne
16000
Stable
MLP
Stable Diffusion
Diffusion
84
encoddr
64
MSE loss
Upsampling
decoder
64
N 4 64
reconstructon
Low-level
Autoencoder lalent
Figure 2: MindEye overall schematic. A high-level "semantic" pipeline maps voxels CLIP embeddings for image reconstruction (outputs from diffusion prior feed through generative models like Versatile Diffusion) or retricval tasks (such as K K-ncarest neighbor querying of brain embeddings to the CLIP embeddings of LAION-5B images). A Jow-level "perceptual" pipeline maps voxels to the variational autoencoder used by Stable Diffusion obtain blurry reconstructions. which used the initialization for subsequent diffusion- image generation. The contrastive loss for the low-level pipeline is omitted for simplicity; Appendix A.3.2 for details.
2 MindEye
MindEye consists of two pipelines (see Figure 2) a high-level (semantic) pipeline where fMRI voxels mapped to the CLIP ViT-L/14 image space and low-level (perceptual) pipeline where the voxels are mapped to the image embedding space of a VAE. Both pipelines follow a common structure: residual MLP backbone followed by two task-specific submodules. For the high-level pipeline the submodules are an MLP projector and a diffusion prior. For the low-level pipeline the submodules are an MLP projector and CNN decoder that performs 4x upsampling. For both pipelines we observe that training the projector submodule with a contrastive loss and the second submodule with mean squared error (MSE) loss gives the best performance.
2.1 High-Level (Semantic) Pipeline
The high-level pipeline is the core of MindEye it maps voxels to CLIP image space to be fed through pretrained image generation models. We refer to as a "high-level" pipeline because CLIP embeddings are inherently more semantic than perceptual, given that CLIP image encoders were trained to maximize similarity with text captions (low-level features like color and object location not typically preserved in these captions). MindEye be used without the low-level pipeline, which simply aids to better preserve low-level image features during reconstruction.
The MI.P backbone for our high-level pipeline maps flattened voxels to an intermediate space of size 257 x 768, corresponding to the last hidden layer of CLIP ViT/L-14 14 (see Appendix [ for Py Torch model code). The backbone consists of linear layer followed by 4 residual blocks and final linear projector. The embeddings from the backbone are fed to an MLP projector and a diffusion prior in parallel. The whole model is trained end-to-ent with the prior getting MSE loss and the projector getting a bidirectional CLIP loss. The projector outputs can be used for retrieval tasks and the diffusion prior outputs can be used by generative models to reconstruct images.
Contrastive Learning: Contrastive learning is an effective method for learning representations across modalities by maximizing cosine similarity for positive pairs while minimizing similarity for negative pairs. Previous work has shown the potential benefits of using contrastive learning
alongside neural data 12 13
CLIP 7 is example of multimodal contrastive model that maps

---

images and text captions to a shared embedding space. MindEye is trained to introduce fMRI as an additional modality to the embedding space of a pretrained CLIP model, keeping the CLIP image space frozen done with locked-image text tuning (LiT) [14] We use the CLIP loss 7 as our contrastive objective. This loss is bidirectional and helps improve both image and brain retrieval.
Recent work [15H18] has explored novel data augmentation techniques that offer several benefits like improving performance, increasing robustness, and reducing training data requirements. Mixup [[S] is one such technique that trains models on synthetic data created through convex combinations of two datapoint-label pairs [9]. Kim et al. 20 introduce MixCo, an extension of mixup that uses the InfoNCE loss, and show that MixCo improves classification performance in a supervise setting. Based on the same principle, we modify the bidirectional CLIP loss use MixCo. While Kim et al. (00 observed that MixCo gives largest performance benefit for smaller models, we observe that it also helps large models low data regimes.
To combine MixCo with CLIP loss, we mix voxels using a factor À sampled from the Beta distribution with a 3 0.15.
Umixi, A xj + Ac) Tke pf [Imixi
Pt = [(x)); 4 = CLIP (yR) (I)
Here, Iy and y/ represent the i-th fMRI sample and image respectively. k [J is an arbitrary mixing index for the i-th datapoint and f represents the combined MLP and projector. p", and t are L2-normalized. The CLIP loss with MixCo is defined as:
exp
exp
LBiMixCo
Σ
d log
+ +11 A1).log
Σ m=1
p.
Σ exp
exp
N
exp
Piits
exp
[piti
Aj
log
Σ (1- log
N Em=1
exp
Pinit
Em=1
N
exp
Pin
We term this bidirectional loss as BiMixCo. Here r is temperature hyperparameter, and N is the batch size.
Recent works 22 have shown that stopping mixup augmentation after a certain number of epochs leads to better classification performance. As per these findings, stop using mixup and switch from hard contrastive loss to a soft contrastive loss one- one-third the way through training. This improves reconstructions without harming gives the highest retrieval performance but slightly hurts reconstructions (Table 4) likely due to how the reconstruction task needs absolute targets for the mixed inputs (which generate by mixing the original targets in the same ratio as the mixup inputs), causing a slight shift in the distribution of target embeddings. Our final schedule combines BiMixCo and soft contrastive loss strike the best balance between retrieval and reconstruction performance in a single model.
retrieval performance (see Table 4.. BiMixCo
Our soft contrastive loss is inspired by knowledge distillation [23], where the authors argue that the softmax probability distribution produced by a powerful teacher model acts as a better teaching signal for a student than hard labels. To generate the soft labels take the dot product of CLIP image embeddings in a batch with themselves. The loss (with bidirectional component omitted for brevity) is calculated between CLIP-CLIP and Brain-CLIP matrices
N
exp
4:tyy
exp
Pi-ti
LSSNCLIP
log
N -==1
ΣN 1==1
exp
exp
Diffusion Prior: Using a diffusion model to align the outputs contrastive learning model was inspired by DALL-E 2 [8), where "diffusion prior" was used to map CLIP text embeddings CLIP

---

image space before using an unCLIP decoder to reconstruct images. Here we train our own diffusion prior from scratch to map CLIP fMRI embeddings CLIP image space, which are then fed into pretrained Versatile Diffusion model to generate image reconstructions. We modified an open-source implementation of the DALL,-F 2 diffusion prior available on GitHub (see Appendix A..3IT. 1) We used the same prior loss as Ramesh et al. [8]. Our total end-to-end loss is defined as:
= CBMiiCo|SSnCLLPP + a Lprior
We use a = 0.3 and switch from BiMixCo to SoftCLIP after one-third of the train cycle. All our models are trained parameter count, MindEye (including both high- and low-level pipelines) can be trained on a single A100 in less than 18 hours. This efficiency is due to the bulk of the parameters stemming from MLPs, which are faster to compute than transformers or CNNs.
single A100 GPU for 240 epochs with batch size of 32. Despite a high
The diffusion prior is critical for reconstruction because contrastive learning only incentivizes the CLIP fMRT embeddings to match the vector direction of the associated CL.IP image embeddings. This generates disjointed embeddings as observed by Ramesh et al. [8]. Theoretically, multimodal contrastive learning will always produce disjointed embeddings because of the "modality gap" phenomenon whereby encoding modalities into shared space restricts the effective embedding space to narrow cone in geometric space 244). We use pre-trained models that expect CLIP image embeddings as input, thus motivating our training of diffusion prior align disjointed embeddings.
To rectify this issue, the diffusion prior learns a distribution of CLIP image embeddings conditioned on CLIP fMRI embeddings. UMAP 25 plots of disjointed CLIP fMRI embeddings next to aligned CLIP fMRI embeddings in Appendix AA. show how the diffusion prior addresses the disjointed embedding spaces problem. We observe that the prior's role cannot be fulfilled by simply adding MSE loss to the MIP projector in Table This is because there is tradeoff between reconstruction and retrieval objectives and model cannot effectively learn a single embedding space that does well on both.
2.2 Low-Level (Perceptual) Pipeline
The low-level pipeline maps voxels the embedding space of Stable Diffusion's 's VAE. The output of this pipeline can be fed into the VAE decoder to produce blurry image reconstructions that lack high-level semantic content but exhibit state-of-the-art low-level image metries. We use img2img [[0] to improve our final image reconstructions in terms of low-level metrics, with minimal impairment to high-level metrics, such that we start the diffusion process from the noised encodings of our blurry reconstructions rather than pure noise.
The MLP backbone for our low-level pipeline follows the same architecure except that the final outputs are of size (16, 16, 64). These are upsampled to (64, 64, 4) by a CNN upsampler. An MLP projector projects the backbone outputs an auxiliary contrastive loss is applied. For more information on the low-level pipeline see Ap- pendix A.222 See Appendix Figure for example blurry reconstructions and Appendix Table 5 to see the effect of changing img2img strength on subsequent reconstruction metrics.
high-level pipeline,
512 dimensional space where
3 Results
For all experiments, we used the Natural Scenes Dataset (NSD) (26). a public IMRI dataset containing the brain responses of human participants passively viewing natural scenes from MS-COCO [27].
By utilizing MS-COCO, this dataset provides measured brain responses to rich naturalistic stimuli, allowing us to study how well low- and high-level image features are reconstructed by MindEye. We
used the same standardized train/test splits as other NSD reconstruction papers B 4 28 training subject-specific models for each of participants. We averaged across three same-image repetitions for the test set (leaving 982 test samples) but not the training set (24,980 training samples), similar to
Takagi and Nishimoto [3] For more information on NSD and data preprocessing see Appendix A.2 for single-trial triil and reduced dataset results see Appendix A.9 and Appendix A.10
5

---

MindEyo the exact Top-1 Image peir from the lest sel of 962 images with 93 2% accuracy (random chance = 0.1%%
sample:
Seen image
Top 1
Top
Tap
C C
Find the corresponding image that was presenled
MRI voxels
x 15000
Top
Tap
MindEye
x 257 768
Soen image
Too
Top
Top
Select image with highest CLIP cosine similarity
Retrieve nearest neighbor in CLIP space in LAION-SB database
Seen image
Retrioval
Seen imago
Retrieval
Retrieval
Sean Image
Retrieval
Figure 3: MindEye image retrieval. Given pool of candidate images, nearest neighbor search in CLIP space enables searching for the original image based on brain activity. Top section depicts how, given 982 test NSD images (many containing very similar looking images, c.g., over dozen zebras), MindEye top-1 performance is 93.2% for Subject 1. The ability to distinguish among confusable candidates suggests brain embeddings retain fine-grained, image-specific information. Bottom section depicts scaling up the LAION-5B dataset (sec Appendix A.4 for examples). Even with billions of images, MindEye finds images similar the original.
3.1 Image/Brain Retrieval
Image retrieval evaluations reveal the level of fine-grained image-specific information contained in the predicted brain embeddings. For example, if the model is given dozen pictures of zebras and the brain sample corresponding viewing one of those zebras, can the model correctly find the corresponding zebra? If the model can correctly deduce that the brain sample corresponds to an image of a zebra but cannot deduce the specific image amongst various candidates, this would suggest that category-level information but not exemplar-specific information is preserved in the CLIP fMRI embedding. MindEye not only succeeds this zebra example but also demonstrates 93.2% overall accuracy for Subject in finding the exact original image within the 982 test images (see Figure
Although we use the full test dataset for retrievals Figure 3 we followed the same procedure as Lin et al. [ for calculating the retrieval metrics reported in Table I Brain retrieval performance was calculated according to the following procedure: for each test image, the image converted to CLIP image embedding and we compute the cosine similarity to both its respective ground truth disjointed CLIP fMRI embedding as well as 299 other randomly selected disjointed CLIP fMRI embeddings in the test set. For each test sample, success is determined if the cosine similarity is greatest between the ground truth CLIP embedding and respective fMRI embedding (aka top-1 retrieval performance, chance=1/300) We average retrieval performance across all test samples and repeat the entire process 30 times to account for the variability in random sampling of batches. For image retrieval, the same procedure is used except image and brain samples are flipped such that the goal to find the corresponding paired CLIP image embedding out of 300 possible CLIP embeddings in the batch. Lin et al. [II] refer to image retrieval "forward retrieval" and brain retrieval as "backward retrieval" in their paper.
We can scale up image retrieval using pool of billions of image candidates. In Figure 3 we show results querying the LAION-5B dataset (29] using our CLIP fMRI embeddings. The final layer CLIP ViT-L/114 embeddings for all 5 billion images are available at knn.kaioon.a and can be queried for K-nearest neighbor lookup via the CLIP Retrieval client (0). For each test sample, we first retrieve 16 candidate images using this method (using variant of MindEye that maps voxels to the final

---

MindEye
Ozcelik et al. (2022))
Takagi (8022)
MindEye (ours)
Ozcelk (2023))
Gu (2023)
MRI
Figure 4: Side-by-side comparison of reconstructions from IMRI-to imaee NSD papers. The same test set was used across papers. All reconstructions come from Subject
layer of CLIP, see Appendix A.7 The best image is then selected based on having the highest CLIP embedding cosine similarity to the CLIP fMRI embedding. This image retrieval approach is especially well-suited for tasks involving fine-grained classification, and can be used as an alternative to image reconstruction without a generative model (evaluations in Table I
3.2 IMRI-to-Image Reconstruction
The diffusion prior outputs from MindEye are aligned CLIP fMRI embeddings that can be used with any pretrained image generation model that accepts latents from CLIP image space. We evaluate the outputs of MindEye reconstructions across several models including Versatile Diffusion (6), Stable Diffusion (Image Variations) [3]] and Lafite [32 II. Here report results from Versatile Diffusion since it yielded the best results, and we report results from the other models in Appendix AA.7 We qualitatively compare reconstructions side-by-side with outputs from other IMRI-to-image reconstruction models in Figure and quantitatively compare against other models in Table demonstrating state-of-the-art MindEye reconstructions.
For each subject, for each test brain sample, we output 16 CLIP image embeddings from MindEye and feed these embeddings through the image variations pipeline of Versatile Diffusion. This produces 16 image reconstructions per brain sample. For our reconstructions we use 20 denoising timesteps with UniPCMultistep noise scheduling [33] and start the denoising process from the noised output of our low-level pipeline (img2img). We then select the best of 16 reconstructions by computing last hidden layer CLIP embeddings and picking the image with the highest cosine similarity to the disjointed CLIP fMRI embedding. This automatic second-order selection was inspired by DALL-E 2 [8), which used similar process of selecting the best of generated samples.
Two-way identification refers to percent correct across comparisons gauging if the original image embedding is more similar to its paired brain embedding or randomly selected brain embedding. Comparison was performed for AlexNet (37] (second and fifth layers), Inception V3 (38] (last pooling layer), and CLIP (final layer of ViT-L/14). We use the same settings as Ozcelik and VanRullen 4 for our metrics. For more details refer to Appendix A.6
3.3 Ablations
In this subsection try to explain where MindEye performance improvements come from through ablations. To study the effects of architectural changes and training strategies we train only the retrieval pipeline (no diffusion prior) for 120 epochs with batch size 300. All models in this section are trained on Subject 1. Table entries with correspond to the final version of MindEye's 's settings.
Architectural Improvements: To study the effect of model depth and parameter count we train multiple MLPs of various sizes (Table 2). Among models that map to the last hidden layer of CLIP VIT-L/14, we observe a clear trend of increased performance with added residual blocks. For 2 blocks, the effect of skip connections not too significant but at blocks the model does significantly worse without them, indicating that skip connections are important for training deeper models.
7

---

Method
Low-Level
High-Level
Retrieval
PixCorr 1 SSIM Alex(2) Alex(5)
Ineep
CLIP Eff SWAV
Image
Brain t
Lin et al.
11
78.2%
11.0%
49.0%
Takagi...
3
83.0%
83.0%
76.0%
77.0%
Gu al.
28
.150
.25
.622
.465
Ozcelik...
.254
356
94.2%
96.2%
87 87.2%
91.5%
.775
.23
21.1%
30.3%
MindEye
309
.233
94.7%
97. 8%
93 93.8% 94. 1%
645
.367
93.6%
90.1%
MindEye (Low-Level)
.300
.479
78.1%
74.8%
58.7% 59. 59.2%
1.00
.663
MindEye (High-Level)
.194
:388
91.7%
97. 97.4%
93. 93.6%
94 2%
.655
.369
98 68.6% 90.1%
MindEye (LAION)
.130
.38
84.0%
92.6%
86.9%
86. 1%
.778
.477
Ozcelik... (Low- S1)
.358
.377
97.7%
97. 6%
77.0% 71.1%
.906
.581
MindEye (Low-, S1)
.566
.493
87.1%
84.1%
61.6%
62.4% .992
.638
Quantitative comparison MindEye retricval and reconstruction performance against other models. Top and middle sections average across the same 4 participants (see Appendix A.8 for individual subject models), except Lin et al. [[T] which only analyzed Subject 1. Middle section reflects outputs from only the high- or low-level pipeline, and metries when evaluating images retrieved from LAION-SB. Bottom section compares low-level reconstructions the low-level reconstructions from Ozcelik and VanRullen [ which only reported metries for Subject 1. Image retrieval refers to the percent the time the correct image was retrieved out of 300 candidates, given the associated brain sample (chance=0.3l 3%); vice-versa for brain retrieval. PixCor=pixelwise correlation between ground truth and reconstructions; SSIM=structural similarity index metrie 34 EfficientNet- B1 ("EIf") [35] and Sw.AV-ResNe550 ("SwAV") (36] refer to average correlation distance; all other metries refer to two-way identification (chance = 50%). Missing values are from papers not reporting all metrics or metrics heing non-applicable. We followed the same image preprocessing as Ozcelik and VanRullen 4 Previous state-of-the-art Ozcelik and VanRullen 4 results are directly comparable to MindEye the same test set and Versatile Diffusion model were used. Bold indicates best performance within sections.
Table 1:
Method
Param Count
Image Retrieval
Brain Retrieval
No ResBlocks
873M
0.880
0.820
2 ResBlocks + No Skip
907M
0.881
0.822
2 ResBlocks
907M
0.886
0.837
4 ResBlocks No Skip
940M
0.836
0.767
4 ResBlocks*
9400M
0.896
0.822
4 ResBlocks Only CLS
135M
0.611
0.576
Table 2: Efffects of varying the architecture of the MLP backbone on retrieval accuracy.
We also show a comparison with a 4-resblock model that maps the final layer of CLIP (only the CLS classification token). This model has 7x fewer parameters and does much worse than all other models. This indicates two things: (I) MindEye strongly benefits from a large parameter count MLP backhone and does not overfit even in the sample constrained settings of the NSD dataset, and (2) the fMRI voxels contain fine-grained information about images, allowing us to effectively predict all 257 CLIP image embeddings instead of just the CLS token.
Training Strategies (Losses and Data Augmentations): We observe that with InfoNCE, MindEye only does well on brain retrieval (Table 3). A similar trend was observed in Lin et al. M. We attribute this to InfoNCE being one-sided Toss that only optimizes for one retrieval objective. Simply replacing InfoNCE with CLIP loss significantly improves image retrieval. MixCo augmentation helps both unidirectional and bidirectional losses.
Method
Image Retrieval
Brain Retrieval
InfoNCE
0.237
0.784
CLIP Loss
0.837
0.791
InfoNCE MixCo
0.303
0,856
CLIP Loss + MixCo (BiMixCo)
0,884
0,841
SoftCLIP Loss
0.837
0.816
BiMixCo + SoftCLIP (MindEye)*
0.896
0.822
Table 3: Effects of different losses and MixCo augmentation MLP retrieval performance.

---

We also show the effect of training with our SoftCLIP loss. SoftCLIP improves over hard CLIP loss for brain retrieval but performs worse than BiMixCo. Our training regime combining SoftCLIP with BiMixCo gives the best image retrieval performance.
Reconstruction Strategies: To demonstrate the need for a separate diffusion prior, we train version of MindEye where both contrastive and MSE losses are applied to the ouputs of the MLP backbone. We observe that this model does poorly in terms of retrieval metries, indicating a tradeoff between retrieval and reconstruction objectives where it is difficult to learn a single embedding space that satisfies both objectives. Inspired by recent works in self-suuer learning [39 42,
we decouple these losses using separate MLP projector, where MSE loss is applied to the outputs of the MLP backbone and contrastive loss is applied to the outputs of the projector. This model does slightly worse in terms of reconstruction but is much better at retrieval. Finally, we train a model with diffusion prior but no MI.P projector. Contrastive loss is computed for the MI.P backbone and MSE loss is computed for the diffusion prior. This model is comparable to high-level MindEye in terms of reconstruction but does worse in retrieval, giving further evidence of a tradeoff. Example reconstructions for these models are in Appendix Figure
Method
LoweLevel
High-Level
Retrieval
PixCorr
SSIM
A2ex(2)
Alex(5)
Incep
CLIP
Image
Brain
Only MLP Backbone
0.119
0.346
73.8%
84.1%
81.5%
82.6%
0.133
0.631
Backbone Projector
0.154
0.296
73.2%
85.2%
75.2%
77.3%
0.888
0.849
Backbone Prior
0.206
0.303
92.1%
97.2%
94.8%
95.1%
0.934
0.901
MindEye (only BiMixCo)
0.195
0.290
91.1%
96.6%
93.7%
94.4%
0.974
0.942
MindEye (0.33 BiMixCo)*
0.198
0.302
91.6%
96.8%
94.6%
95.0%
0.972
0.960
Table 4: Fffects of diffusion prior and MI.P projector on reconstruction and retrieval metries.
4 Related Work
In the 2000s, researchers demonstrated that visual information, such as spatial position
43 ori-
entation 44 45 and coarse image category [46 47 could be decoded from fMRI signals using
linear classifiers. With the introduction of generative adversarial networks [48], more sophisticated
decoding became feasible researchers mapped brain activity to the latent space of these models
to reconstruct handwritten digits [4), human faces [50] [1] and natural scenes [2 5 53 More
recently, with the release of multimodal contrastive models like CLIP (7), diffusion models (54 55
like Stable Diffusion [9] and new large-scale fMRI datasets 26 fMRI-to-image reconstructions
have reached an unprecedented level of quality 28
Lin et al. [[]] reconstructed NSD images by mapping voxels to CLIP space (see also Wang et al. (561) and fed outputs through a fine-tuned Lafite [3] GAN (MindEye reconstructions using Lafite in Appendix A.7p. Differences from MindEye include using a convolutional model, no projector to separate contrastive loss from MSE loss InfoNCE instead of CLIP loss, fine-tuning of pretrained
GAN, no diffusson prior, and mapping to both CLIP image and text space. Ozcelik and VanRullen 4 used low- and high-level pipeline with Versatile Diffusion (6]. Differences include mapping to CLIP space via ridge regression, no contrastive learning or diffusion prior, and mapping to a
VDVAE (57 for lowel reconstructions. Gu et al. (28 used low- and high-level pipeline and
extended on Ozcelik et al. (5) by reconstructing with IC-GAN (58); they did not flatten voxels and
mapped to SwAV (36] features with surface-based convolutional networks. Takagi and Nishimoto
3 used ridge regression to map to Stable Diffusion latents and CLIP text latents, using different voxel selections for different components. Mind-Vis [59] did not use the Natural Scenes Dataset but tackled the IMRI-to-iman model on separate large-scat dataset. This enabled the authors to use a more informative latent as the input to their image reconstruction model. Mind-Ve Vis approach by reconstructing video rather than images. Overall, MindEye is unique in its use of reconstruction and retrieval submodules, a deep MLP backbone with 940 million parameters (parameter size comparison Table 10). a diffusion prior for more accurate translation across brain and image modalities, and state-of-ther -hthe-rrtt reconstuction and retrieval results.
problem from the unique angle of first pre- pre-training masked brain
-ideo [60] subsequently extended on the

---

5 Conclusion
We present MindEEee novel mental decoding approach that achieves state-of-the-an reconstructions of natural scenes presented to humans in the MRI machine. These reconstructions retain semantic and perceptual similarity to the original images due the use of combined high-level and low-level pipeline. The novel use of specialized submodules for contrastive-based retrieval and diffusion-basee reconstruction allows MindEye to learn mappings for both tasks in parallel. MindlEe brain latents contain fine-grained image-specific signal as demonstrated by the ability to select ground truth images out of a set of nearly 1,000 possible images (see Figure 3 We leveraged pretrained CLIP 7. a model trained with billions of image and text data samples, as a teacher to guide MindEye where we have a relative scarcity of brain data (less than 30,000 training samples per participant). Our diffusion prior submodule is trained from scratch and allows for accurate translation of brain embeddings into pretrained CLIP space such that any model that accepts CLIP image embeddings can be provided with CLIP fMRI embeddings without fine-tuning. This flexibility suggests that MindEye reconstructions will continue to improve as newer, more powerful image generation models are released.
The ability to accurately reconstruct perception from brain activity prompts questions about broader societal impacts. For instance, it should be possible to generalize current reconstruction models from perception to mental imagery without training new model [61. However, current models are not capable of across-subject decoding and each NSD participant spent up to 40 hours in the MRI machine to procure sufficient training data (see Appendix g for results using a subset of the total training data). Furthermore, non-invasive neuroimaging methods in general require compliance because participants can easily resist decoding by moving their head or thinking about unrelated information 665) MindEye is also limited to natural scenes such as those in MS-COCO; for other image distributions additional data collection and specialized generative models would be required. While high-quality image reconstruction via non-invasive neuroimaging is not currently practical for real-world applications, technology is constantly improving and it is important that brain data be carefully protected and companies collecting such data transparent with their use.
Privacy Concerns & Societal Benefits:
Image reconstruction from brain activity can enable various potential socictal benefits. Reconstruc- tions are expected to be systematically distorted due to mental state, neurological conditions, etc. This could potentially enable novel clinical diagnosis and assessment approaches. For example, patients suffering from major depressive disorder might produce reconstructions where emotionally negative aspects of images are more salient [66]. MindEye results also suggest potential for improved locked-in (pseudocoma) patient communication via fine-grained visual communication beyond simple classification [67]. as well as brain-computer interface performance if adapted to real-time fMRI analysis (68) or non-fMRI neuroimaging modalities.
Future directions we wish to explore include mapping individual subjects to a shared embedding space to enable training models that are generalizeable across people (c.g..
Future Directions:
69
70), ), and exploring model intepretability approaches like GradCAM 71] or Network Dissection to identify the fMRI voxels that most strongly respond to the presence of certain image features.
6
Open Research: 100% Transparent Volunteer-Drive Science
MindEye was openly developed through volunteer contributions in the MedARC Discord server. Source code was always accessible via public GitHub repository throughout the lifespan of the project. Research discussions were held via public Discord channels, and weekly video conference calls were recorded and shared publicly. We continue extend global invitation to contribute to Neuroimain & AT Lab] projects to cultivate an internationally diversified, volunteer-driven research team composed of members from varied backgrounds possessing wide array of expertise. We contend that fully transparent open-research initiatives such as this and others like EleutherA] LAION OpL and ML Collective could redefine the traditional framework of scientitic research, democratizing entry into machine learning and medical research through the harnessing of crowd-soureed collective intelligence and community collaboration.
7 Author Contributions
For detailed author contributions see Appendix A.I
10

---

8 Acknowledgements
Thanks to the MedARC community, including Jeremy Howard, Tommaso Furlanello, Mihir Tripathy, and Cesar Torrico for useful discussion and reviewing the manuscript. Thank you to Furkan Ozcelik, author of Brain-Diffuser, for sharing his code and expert knowledge with our group. We thank LAION for being the initial community where this project developed, and thank Romain Beaumont and Zion English for useful discussion during that time. We thank Stability Al for sharing their computing workplace and giving us the computational resources necessary to develop MindEye. Thank you to Richard Vencu for help navigating the Stability HPC. Collection of the Natural Scenes Dataset was supported by NSF IIS- 1822683 and NSF I1S-1822929.
References
Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fMRL. Neuroimage, 56(2):400-414 410, 2011. Publisher: Elsevier.
[2] David Linden. Introduction. In MMI Neurofeedback, pages 161-169. Elsevier, 2021. ISBN 978-0-12 12- 822421-2. doi: 10.1016/B978-01 12 82.422-2..00088--
URL (tLLitttps:://iiiiingguue elsevier..... com/
[3] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. preprint, Neuroscience, November 2022. URL http:://bioxxivoorg/ooooo
doi 110./2022.222.2.. 18. 517004
Furkan Ozcelik and Rufin VanRullen. generative latent diffusion, March 2023. URL httpp://aaaaa...rg//a//////////0 05334 0rXiv:2303.05334
Natural recosttuutnnnnnnnnnntciicioo from IMRI signals using
[cs, q-bio].
[5] Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of Perceived Images from [MRI Patterns and Semantic Brain Exploration using Instance-Condittoned GANs, February 2022. URL http://ttt::ii org/ 2202.. 12692 arXiv:2202.1 12699 [es, cess, q-bio].
[6] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile Diffusion: Text, Images and Variations All in One Diffusion Model, March 2023. URL http://arxiv www.xxii......oooooooooooo org/ 2211 08332 arXiv:2211.08 1.08332 [es].
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://t: org/abs/2100 00020 urXiv:2103.0000 [es].
[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Tex-iiiiaaaieennnniiiiiioaaaa Image Generation with CLIP Latents, April 2022. URL http://arxiv.. wttp:///wwwr//////22..0... 06125 arXiv:2204.061 [es].
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://pp//arr..ooo///////22222. wttp://wrr.gaab........m arXiv:21 10752 [es].
[10] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, January 2022. URL http: //arxi/ org/ aba/ 2108. .00033 ar #rXiv:2108.0100 [cs].
[11] Sikun Lin, Thomas Sprague, and Ambuj K. Singh. Mind Reader: Reconstructing complex images from brain activities, September 2022. URL /tttp///arxiv org/ awtpaa////222 01769 1rX0V:2210.01769 [cs. eess, q-bio].
[12] Alexandre Défosse7, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Rémi King. Decoding speech from non-invasive brain recordings, August 2022. URL htttp://aariv.orr//aa/22200 12266 arXiv:2208.1 12266 (cs, cess, q-bio].
[13] Steffen Schneider, Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360 368, May 2023. ISSN 1476-4687 doi: 10.1038//41588 023-06031- URL https: / /wwp://wwwww.o nature wwwatueeeeoeeeeeeeeeeee wtmm//rtiilee//4166666616.11 023- 0663000-------0000------ Number: 7960 Publisher: Nature Publishing Group.

---

[14] Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner. Daniel Keysers, Alexander Kolesnikov. and Lucas Beyer. LIT: Shot Transfer with Locked-image text Tuning, June 2022. URL h awrxii..rg//ba//11111 07991 MEXiv:2111.075 [es].
[15] Hongyi Zhang. Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, ArXiv, abs/ 110.09912, 2017.
[16] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022- 10 2019.
[17] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan. and Quoe V. Le. Autoaugment: Learning augmentation policies from data, ArXiv, abs/1805.0995) 2018.
[18] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv, abs/1912.02781, 2019.
[19] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Naumovich Vapnik. Vicinal risk minimization. In NIPS, 2000.
[20] Sungnyun Kim, Gihun Lee, Sangmin Bae, and Seyoung Yun. Mixco: Mix-up contrastive learning for visual representation. ArXiv, abs/2010.06300, 2020.
[21] Zixuan Liu and Ziqiao Wang. Over-training with mixup may hurt generalization. ArXiv, abs/2303.01475, 2023.
[22] Hao Yu, Huanyu Wang, and Jianxin Wu. Mixup without hesitation. ArXiv, abs/2101.042 04342, 2021.
[23] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
[24] Weixin Liang. Yuhui Zhang. Yongchan Kwon, Serena Yeung. and James Zou. Mind the Gap: Understanding the Modality Gap Multi-modal Contrastive Representation Learning, October 2022. URL, hhtpp
//arxiv. wtwp/rgg///a//////eeeee//e/////e .20053 arXiv:2203.02055 [es].
[25] Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, September 2020. URL http://arxiv. DXiv:1802.00322 [cs, stat].
1802. 03426
[26] Emily J. Allen, Ghislain St-Y Yihan Wu. Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle. Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. massive 7T IMRI dataset to bridge cognitive neuroscience and artificial in- telligence. Nature Neuroscience, 25(1):116 1266 January 2022. ISSN 1097-6256, 1546-1726 doi: 10. 10.10/415933-0211 -21-6092-x.
httpt::t/////tt nature. com/
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, Lecture Notes in Computer Science, pages 740-755. Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1, doi: 10.1007/978-33 3-39999----1----- 100---3311---02--11111 10602-1_48.
[28] Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuneu. Decoding natural image stimuli from fMRI data with surface-hased convolutional network, March 2023. URL httt::///axiiii org/ aa/aabba2221 2212 02409 arXiv:2212.02400 [cs. b-bi
[29] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION- 5B: An open large-scale dataset for training next generation image-text models, October 2022. URL http://arxiv. org/ awaaaaab//211 08402 arXiv:2210.08400. [cs].
[30] Romain Beaumont, Clip Retrieval: Easily compute clip embeddings and build a clip retrieval system with them, 2022. URL httpt:ttt///hhuubbbbbb com/ wot.//ooo10///lopooooooooo retrievval publication mise; publisher: GitHub; journal: Gitllub repository.
[31] Justin Pinkney. Lambda Diffusers, 2022.
https://gi /ht.bbb.ttttuttt.//eg www LamaaabamaaL publicationType: mise; publisher: GitHub: journal: GitHub repository.
URL
lambda- Samba--diffusera
12

---

[32] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. LAFITE: Towards Language-Free Training for Text-to-lmage Generation, March 2022. URL http:/tttt//iiiiv otg//bs///// 13792 arXiv:2111.13799 [cs].
[33] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen La. UniPC: A Unified Correcto Framework for Fast Sampling of Diffusion Models, February 2023. URL htttp://arsiv..rrg abs/
7302.04867 arXiv:2302.04866 [es].
[34] Zhou Wang. A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-61 612, April 2004, ISSN 1941-0042, doi: 10.1 10..1199912.200381999890
Conference Name: IEFF Transactions on Image Processing.
[35] Mingxing Tan and Quoe V. Le. EfficientiNe:: Rethinking Model Scaling for Convolutional Neural Networks, September 2020. URL http://arxiy httt:://arrx..orr//a////96. 11946 arXiv:1905. 11946 [cs, stat].
[36] Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised Learning of Visual Features by Contrasting Cluster Assignments, January 2021. URL http://arxiy www./og///aa/eeee.ooo.oo aaaaabaa///0 2006/2006..0000 09882 arXiv:2006.0988. [cs].
[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolu- tional Neural Networks. In Advances Neural Information Processing Systems, volume 25. Curran As- sociates, Inc., 2012. URL
/ite///////aapee/2///aaaaaaa neurips httprr/pp.ccappppp.e.ee////////p
[38] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, June 2016, doi: 10. 10.1109/CVP..2016.330
ISSN: 1063-6919.
[39] Jean-Bastien Grill, Florian Strub, Florent Altch'e e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot. Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach supe-sup learning. ArXiv, abs/2006.07733 07733, 2020.
[40] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[41] Ting Chen, Simon Kornblith, Mohammad Norouzi. and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
[42] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality hetween contrastive and non-contrastive self-supervised learning. ArXiv, 0bss/200.005574 2022.
[43] Bertrand Thirion, Edouard Duchesmay, Edward Hubbard, Jessica Dubois, Jean- Baptiste Poline, Denis Lebi- han, and Stanislas Dehaene. Inverse retinotopy: Inferring the visual content of images from brain activation patterns. Neurolmage, 33(4):1104-11 December 2006. ISSN 105381 19. doi: 10. 2006.06.062. URI. http::tttts:://iniinnnnee httpsuuu.lllleeeeeeeeeeeeeeeeeee www.weeeeeeeeeeeeeeeeeeeee http://ww./aa////aaa.a/a/aaa/aaa htt.a//eeeeaa../1000011111110100
[44] John- Mohn-Dylan&un Haynes and Geraint Recs. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature Newroscience, 8(5)686 691, May 2005. ISSN 1097-6256, 1546 1726. doi: 10. 10.1038/nn1443 URL http: nature...aure.....o com www.aaatiillsssee 144444
[45] Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5):679-685. May 2005. ISSN 1097-6256, 1546-1726. doi: 10. 1048//4414444 URL http://vse .nauure. com watmm//rriccees/n11ga/a.
[46] David D Cox and Robert I. Savoy. Functional magnetic resonance imaging (fMRT) "brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neurolmage, 19(2):261-270. June 2003. ISSN 10538119. doi:
(1A //linkinghub. werpeaubb.llevieee elsevier. ala.rrccc..ccceeerreeee//////s/s
URL litpp
[47] James V. Haxby, M. Ida Gobbini, Maura L. Furey, Alumit Ishai, Jennifer L. Schouten, and Pietro Pietrini. Distributed and Overlapping Representations of Faces and Objects Ventral Temporal Cortex. Science, doi: 10. 266/scicncee106677.6
293(55399 24300 September 2001. ISSN 0036-8075, 1095-9203.
https://ww// http://wwiiiieeeee.a//////////// ort//oi/////ooooooooooo 100 wweee11/eeciiiii hwtp//////eccceeeeeeeeeeeeeeeeee 1063736)
[48] Ian J. Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014.
org/
abs/ abs/1406.r 2661 14662140 [cs, stat].
13

---

[49] Sanne Schoenmakers, Markus Barth, Tom Heskes, and Marcel van Gerven. Linear reconstruction of perceived images from human brain activity. Neurolmage, 83:951-911, December 2013, ISSN 105381 doi: 10. new.omigeeee
0105381191300993
[50] Rufin VanRullen and Leila Reddy. Reconstructing faces from IMRI patterns using deep generative neural networks. Communications Biology, 2(1): December 2019. ISSN 2399-3642. doi: 10.1038/ +42003-019-0438-y.
httphttt:///sat nature. com/ http://rrrcceeeseaaeee
019 019-0438-119-----1 0438-y
[51] Thirza Dado, Yagmur Güşlütürk, Luca Ambrogioni, Gabriëlle Ras, Sander Bosch, Marcel van Gerven, and Umut Güçlü. Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space. Scientific Reports. 12(1):141, December 2022. ISSN 2045-2323 2322. doi: 10.1038/ 10998-021-03988- URL fttppt:ttt///// wawure. wttparrww.....taaaeaeaaeeeeaeaee /ttaa//arriilleeee11888------- 021 031-03938-
[52] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLOS Computational Biology, e501)xe January 2019. ISSN 1553-7358 doi: 10.1371/jourma..ctt 1006633. URL httt/htts///dd http:///xxpplll...ooooooooooooos org/ 10. (371///ournaa pcb.. 1006633
[53] K. Secliger, U. Güglüü L.. Ambrogioni, Güglüttükk, and M.A.J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage, 181:775-785, November 2018. ISSN 10538119 19. doi: 10. /01////.cuuoommagg........ 2tw.://aaa......7..... 8.077007043
elsevier
[54] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising Diffusion Probabilistic Models, December 2020.
URI /ttx/vxxvvvv///////ee///aaaalla/ httppppp///axxv..orr/aaaa/////// 2006. 11239
ar arXiv:2006.1 11239 [es, stat].
[55] Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models, October 2020. http://////axx wtgg/ba/////..o..////// 09011 xxX10:2006.09011 [es, stat].
[56] Aria Y. Wang, Kendrick Kay. Thomas Naselaris. Michael J. Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex, Septem- ber 2022. URL https: /www. biorxiv. www. ht////nnttetttt/ee 11010/022.2222.1. 09. 27..27.5088 508760v1 0022.09.27.50870 Section; New Results.
Pages:
[57] Rewon Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them Images, March 2021. http:/tpp///ariivoorr/a/aa//////
10650 arXiv:2011.1065) [cs].
[58] Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero- Soriano. arXiv:2109.05070 [cs].
GAN, November 2021. URL httpp://axxv..rr// wrg//bs//////////o htte//219..0000 0
[59] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Sceing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding, March 2023. URL
attp://aaxiss www.:rgg/aaaooooo 2211 06956 arXiv:2211 .,6956 [cs].
[60] Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity, May 2023. URI httt::::::a:iiii org/ abs// ab/p://bb///3...// 2305. 11675 arXiv:2305. 11675 [es].
[61] Mark Stokes, Russell Thompson, Rhodri Cusack, and John Duncan. Top-Down Activation of Shape- Specific Population Codes in Visual Cortex during Mental Imagery. Journal of Neuroscience, 29(5); 1565-1572, Februury 2009. ISSN 0270-6474, 1529-2401. doi: 10. SESSIIEEUROSLLA667---00
URL
https://www. inngur:// wttg//onteett/9///1100e Articles.
Publisher: Society for Neuroscience Section:
[62] Rainer Gocbel, Rick van Hoof, Salil Bhat, Michael Lührs, and Mario Senden. Reading Imagined Letter Shapes from the Mind's Eye Using Real-time 7 Tesla fMRL. In 2022 10th International Winter Conference on ISSN: 2572-7672.
Interface (BCI), pages 1-3, February 2022. doi: 10.1 /11////C13372.2222.777557
[63] Thomas Naselaris, Cheryl A. Olman, Dustin F. Stanshury, Kamil Ugurbil, and Jack I.. Gallant. A voxel- wise encoding model for early visual areas decodes mental images of remembered scenes. Neurolmage, 105:215-228.
January 2015. ISSN 1053-8119. doi: n0..01///..uurrmmage... 2014..10.00 10.018. URL httpss 101 m016iiieurro 2009. 11.04, http:////eneeaaacccca/a////////a
//ииw. www.sccieneedrect.co com/ wwwescceeeeecceeeeeeeee
[64] Leila Reddy, Naotsugu Tsuchiya, and Thomas Serre. Reading the mind's eye: Decoding category information during mental imagery. Neurolmage, 50(2)8188 825, April 2010. ISSN 1053-8119. doi: 10.
$1055811909012701
14

---

[65] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth. Semantic reconstruction of con- tinuous language from non-invasive brain recordings. Nature Neuroscience, pages 1-9, May 2023. ISSN 1546-1726. doi: 10. 10.1038/41193--03-0001 304-99. URL httpp:://wwww..tttt http://www.aattttae////
541593-023 91304-9 Publisher: Nature Publishing Group.
[66] Anne C Mennen, Kenneth A Norman, and Nicholas B Turk-Browne Attentional bias in depression: understanding mechanisms to improve training and treatment. Current Opinion in Psychology, 29: 266-273, October 2019. ISSN 2352-250X. doi: 10. 10.101666..copp. cotp///cpsyc..219...77 sciencedirect.
URL htttp::/wwe
[67] Martin M. Monti, Audrey Vanhaudenhuyse, Martin R. Coleman, Melanic Boly, John D. Pickard, Luaba Tshibanda, Adrian M. Owen, and Steven Laureys. Willful Modulation of Brain Activity in Disorders of Consciousness. New England Journal of Medicine, 362(7):57--589, February 2010. ISSN 0028- 4793. doi: 10. 001106//E//Mc0000537000
http:/ttt/////oo org/ 10 1056/ 11156/NNE.M000055377 Publisher:
Massachusetts Medical Society _eprint:
[68] Grant Wallace, Stephen Polcyn, Paula P. Brooks, Anne C. Mennen, Kc Zhao, Paul Scotti, Scbastian Michelmann, Kai Li, Nicholas Turk-Browne, Jonathan Cohen, and Kenneth A. Norman. RT- Cloud: A cloud-based software framework to simplify and standardize real-time IMRI. Neurolmage, 19295, August 2022. ISSN 10538119. doi: 10. 101666 202.. 19295. URL https
//linkinghub.co www.hhgehh.u.eeeeeeeeeeeeee.eee. elsevier....... wwt.///eeeeeeeee//////ee 011111000101111044111
[69]
Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun, Uni Hasson, James Haxby, and Peter Ramadge. A Reduced-Dimensiin IMRI Shared Response Model. In Advances in Neural Information Processing
Systems, volume 28, Curran Associates, Inc., 2015, URL htttt:://ppppeee www..ii............ httppcccppppeeetieeeeeee files/ paper/2015/hase wetaaahhhbe67aeeeee 03386700638862226440226563333333 ebd5a- Awtaaaatbttttatttc html
[70] James V. Haxby, J. Swaroop Guntupalli, Andrew C. Connolly, Yaroslav O. Halchenko, Bryan R. Conroy, M. Ida Gobbini, Michael Hanke, and Peter J. Ramadge. A Common, High- Bigh--imenssonal Model of the Representational Space in Human Ventral Temporal Cortex. Newron, 72(2):404-416, October 201 1. ISSN 08966273. doi: 101 101.//..eerro..00000 1.08.026. URL https: htttps://Linkinghuub. elsevier.
[71] Ramprasaath R. Sclvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-basse based Localization. International Journal of Computer Vision, 128(2)333-359, February 2020. ISSN 0920-5691, 1573-1405. doi: 10. 120007/11226--11--12222-- URL [cs].
wttpaa/////........e............ 02391 arXiv:1610.02391
[72] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantify- ing Interpretability of Deep Visual Representations, April 2017. URL http::/ttt:.ii..//////aa org/ aba/ 1704 05796 arXiv:1704.0577 [es].
[73] Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN XL: Scaling StyleGAN to Large Diverse Datasets, May 2022. http:///://axx 2220 00273 arXiv:2202.00273 [es].
[74] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized Image Modeling with Improved VQGAN, June 2022. [cs].
URL BRR/http:///axxi
org/ abs/ 22117.411106077
arXiv:21
[75] Jacob S Prince, Ian Charest. Jan W Kurzawski. John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy single-trial IMRI response estimates using GLMsingle. eLife, 11:977599, November 2022. ISSN 2050-084X. doi: 10.7554/cLife.77559 URL htttp:://doi.org// 7554/
eLife. 77599 Publisher: eLife Sciences Publications, Ltd.
[76] Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3:47-57, 2017.
[77] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132- 1340, 2017.
[78] Kevin Turner. Decoding latents to rgb without upscaling. https: httss///cssuusssugggnnneeeeeee decoding-Latents-- weaes-lltetttttt to- r-b-
www..hggggc.......o co/t/ ing/3244//////222///// 23204/2 2022. Accessed: 2023-05-23.
[79] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features.
ArXiv, abs/2200.01571, 2022.
[80] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg:
regularization for
self-supervised learning. ArXiv, Abs/2105.049900 2021.
15

---

A Appendix
A.1 Author Contributions
PSS devised the project, led the team, developed the models, drafted the manuscript, and otherwise contributed to all parts of MindEye development. AB drafted the manuscript, developed the models, and contributed to all parts of MindEye development including creating the low-level pipeline, conception of BiMixCo and soft CLIP loss, and modification of the DALL-E 2 diffusion prior. JG developed the models, tracked/compared model variants, and significantly contributed to the MindEye codebase. SS conceived of and implemented LAION-5B retrieval using the CLIP Retrieval client and conducted various exploratory experiments. AN implemented the Lafite pipeline for MindEye reconstructions. EC conducted various initial explorations into using a diffusion prior for aligning voxels CLIP space. AJD created the initial webdatasets used train MindEye and created various model architectures to compare different mapping approaches. NV conducted various exploratory experiments mapping voxels to StyleGAN-XL, [33 latent space. EY shared code automatically identify identical images for qualitative comparisons and added code to ensure LAION-5B retrieval did not retrieve ground truth images. DW conducted various exploratory experiments and helped with project discussions. KAN oversaw the project and contributed valuable feedback. TMA oversaw the project, conducted initial explorations using VQGAN (74. and helped keep the project on-track through MedARC and Stability Al communication.
A.2 Additional Dataset Information
The Natural Scenes Dataset (NSD) 26) is a public 7-Tesla fMRI dataset containing the brain responses of several human participants each spending up to 40 hours in the MRI machine passively viewing images. These square-cropped images of natural scenes were sourced from the MS-COCO dataset [27. Each of 9,000-10, 10000-1,000 unique images was presented for three seconds at time, shown three times across 30-40 scanning sessions, totaling 22,000-30,000 trials of fMRI responses per participant fMRI responses correspond to session-wise z-scored single-trial betas output from GL GLMSingle [75].
Following the procedure used in other reconstruction studies that used NSD 55] individual-subject models for the four participants who completed all scanning sessions (participants 1, 2, 5, and 7) and used a test set corresponding to the shared 1,000 images presented to every participant. This yields dataset consisting of 24,980 training samples and 2,770 test samples- average across the three same-image repetitions for the test set (leaving 982 test samples) but not the training set, similar to Takagi and Nishimoto 33. We use preprocessed flattened fMRI voxels in 1.8-mm native volume space corresponding to the "nsdgeneral" brain region, defined by the NSD) authors the subset of voxels in posterior cortex most responsive to the visual stimuli presented (between 13,000 to 16,000 voxels per participant). MindEye was developed using a training and validation set of Subject I's data, with the test set (and other subjects' data) untouched until final training of models.
28 31,
we train
A.3 MindEye Architecture
PyTorch code for the MLP backbone and projector is depicted in Algorithm Specifics on how we modified the open-source implementation of the DALL-E 2 diffusion prior are discussed in A.3.
A.3.1 Modifications from DALL-E 2 Diffusion Prior
The inputs for the diffusion prior are 257 backbone embeddings. 1 timestep embedding, and 257 noised CLIP image embeddings, and the output is 257 denoised CLIP image embeddings. Unlike the DALL-E 2 prior, we do not use learnable queries and instead directly predict denoised CLIP embeddings from the noised embeddings. This significantly saves on memory and allows us to train the backbone and prior end-to- aad-to-end on a single GPU. We observe that adding absolute positional embeddings to the noised CLIP embeddings improves performance in the absence of learnable queries. We also observe that our prior can work with just 100 timesteps instead of 1000 as used in DALL-E 2. This makes our prior much faster at inference time. We conducted experiments with both causal and bidirectional attention and did not observe any significant difference reconstruction performance, For simplicity use bidirectional attention in our final model.
16

---

Algorithm PyTorch code for MindEye MLP backbone and MLP projector
clasa BrainHLP (
def
out__din=287+783, in_din-16724, clip_aize-768, h-4096):
super ()...iitt.......................
in_dim corregponds a number of voxels in the "asdgeneral" brain region. self. 1ind an.Sequentiall an Linear h, biss-False),
subject-epecife
un Lm..Lyyroor(b))- nn GELU GELU((tglaca==ruu)
mu..Dropout(0.5)
self .mpp =m No.NoduleListt((
nn tn.Sequential
nn. (h, biee-False),
nn.Layyexorn(a))
na na. (Epl
Dropout (0.
range (4)1)
self.linf na.Linear out_din, bias-True)
relf .roj (n..Sequettaal
nn [..LayyyrNomm(llp.aa44))))
nn (n.cc.LU((n)]acc-TTr)))) wm.Liierr(clipppie. na mm.LayyrNNom
2048, bias-False),
an. mm.ccELU(nnpaccceTrrr
an Liieear aa Lt.Lyyyroor (2048)
2048, bias-False),
an. an.GELU (1plaa
an Linear (2048, clip_size, bias-True))
self clip.ciip.siee clip_size
def fsrlf x): 1xn0(x)
residual =
for res_block in
(self.ml))) self. s.sot..ipprrss.booct
+- residual residual -x
siffusion_prior_inne salf.lini (x) reshape self clip_aize)
clajointed,clippfnrr self. prop:
return diffusion.prior_inngg dasjointed_clipptaas
A.3.2 Low-Level Pipeline: Mapping to Stable Diffusion Variational Autoencoder
To map to Stable Diffusion's 's VAE latent space we use low-level pipeline with the same architecture as the high level pipeline. We use separate residual MLP backbone with 4 residual blocks that maps flattened voxels to a 16 x 16 x 64 dimensional latent space. The reconstruction submodule in the low-level pipeline a CNN upsampler that upsamples these latents by 4x to create embeddings of size (64, 64, 4). The CNN upsampler uses similar architecture to Stable Diffusion's 's VAE decoder, which does an upsampling. To create the targets for the upsampler we upsample NSD images to 512 x 512 through bilinear interpolation and encode them with the SD VAE encoder. The resulting (64, 64, 4) embeddings form the targets for the high-level pipeline.
Recent works in low-level vision (super-resolution, denoising, deblurring, etc.) have observed that mean absolute error performs better than mean squared enror for pixel-level metrics like PSNR and SSIM [76] [77 due to better convergence properties. It has been shown that the 4-channel SD latent space effectively compresses images, and latents can converted to RGB images with a linear mapping from latent space to pixel space (78). We observe that the problem of mapping to SD embedding space follows the same properties as low-level vision tasks, such that mean absolute error performs better than mean squared error. We also experiment with using "full reconstruction" loss where we reconstruct complete images using the SD VAE decoder and apply the loss in pixel space. This performs worse than only applying the loss in latent space and also requires significantly more GPU memory.
The contrastive submodule in the low-level pipeline acts as auxiliary loss to improve the perfor- mance of the reconstruction submodule. uses an MLP projector that maps the (16, 16, 64) backbone outputs to (16, 16, 512). Since we do not care about retrieval performance for the low-level pipeline, we simply use SoftCLIP loss without BiMixCo. maximize low-level performance we distill the knowledge of VICRegI [79] ConvNext-XXI. instead of CLIP VIT. VICRegI. with = 0.75 is specialized for low-level tasks and achieves otate-of-the-art linear segmentation results, unlike CL.P which has been trained with high-level text guidance.
17

---

Seen image
Recetottttion
Seen image
Seen nays
Recensmrko
Seen imoge
Pecorenition
Bes image
Reordhndon
Figure 5: Additional MindEye reconstructions for Subject 1, randomly selected.
Figure Additional MindEye retrievals from LAION-5B for Subject 1, randomly selected.
A.4 More Reconstructions/ / Retrievals
Images containing all 982 reconstructions and retrievals for each subject on GitHub, Figure depicts subset of randomly selected reconstruction examples from Subject (first try random selection of 30 samples). Figure likewise depicts randomly selected examples from LAION-5B retrieval. Figure depicts randomly selected example reconstructions from the low-level pipeline. Figure depicts randomly selected reconstructions for the models described in 3.3
A.5
UMAP Comparison
As depicted in Figure 9 the CLIP image and CLIP fMRI embedding spaces are disjointed before being fed through the diffusion prior. While the MLP projector does improve alignment compared to the outputs of the MI.P backbone, the diffusion prior does much better job at aligning the two spaces as shown by decreased euclidean distance between data points following UMAP dimensionality reduction.
18

---

Seen
Recoratuatin
Sean
Pecerstrctin
Seen
Pecoratuctoo
Seet
Pecsrahutton
Figure Example MindEye reconstructions Subject output from low-level pipeline.
Seen image
Only MLP
Backbone Projector
Backbone
MindEye (only BiMaCCo)
MindEye (0.33 BiMixCoj
Figure 8: Example reconstructions for ablation models from Table g
19

---

Img21mg Strength
LeveL
High-Level
PixCorr
SSIM
Alex(2) 1 Alcx(5) f Incep f CLIP 1 Eff 4 SwAV
1.0 (Only low-level)
.466
.493
87.1%
84.1%
61.6% 62.4%
.992
.638
0.7
439
.466
92.7%
95.1%
90.0%
87.5%
.803
.514
0.5
.29
.389
96.3%
98.4%
94.7%
92.3% .674
.405
0.3
410
.38
97.5%
98. 98.8%
94.7%
94.5%
.638
.362
0,15*
.300
.337
97.4%
98.7%
94.5%
94.6%
.630
.358
0.0 (Only high-level)
.209
.318
92.8%
98.0%
94.5% 94.8% .635
.361
Table 5: Evaluations from Subject varying img2ing strength from 0 (no img2img) (only low-level pipeline). The final MindEye uses an img2img strength 0.15.
CLIP Image MLP backbone Avo. suclidean distance 5.815
CLIP Image x MLP projector Avg. ouclidean distance 0.414
CLIP Image Diffusion prior cuclidean diszance 0.180
UMAP
UMAP1
UMAP 1
UMAP
Figure 9: UMAP plots depict CLIP image latents (blue), MindEye MLP backbone latents (orange). MindEye MLP projector latents (green). and MindEye diffusion prior latents (red). UMAPs were estimated from 1,000 random samples from Subject 1. CLIP image latents correspond to the last hidden layer of ViT-I/14, Euclidean distance between the given MindEye embedding space and CLIP image space is lowest for the diffusion prior, suggesting that the diffusion prior helps to align the two embedding spaces.
A.6 Reconstruction Evaluations: Additional Information
Two-way identification was performed in the same manner Ozcelik and VanRullen 4 model, we computed the Pearson correlation between embeddings for the ground truth image and the reconstructed image, well as the correlation between the ground truth image and different reconstruction elsewhere in the test set. If the correlation for the former was higher than the latter, this was marked correct. For each test sample, performance was averaged across all possible pairwise comparisons using the other 981 reconstructions to ensure no bias from random sample selection. This yielded 982 averaged percent correct outputs, which averaged across to obtain the metrics reported in Table
For each
were not reported in the original paper; we Retrieval evaluations for Ozcelik and VanRullen 4
calculated image/brain retrieval ourselves with the help of the
GitHub repository
A.7 Reconstructions from Stable Diffusson (Image Variations) and Lafite
We also attempted reconstructions using Stable Diffusion (Image Variations) [31 and Lafite [32 rather than Versatile Diffusion. Reconstructions from these models for Subject I are depicted in Figure T0 with metries reported in T
For Stable Diffusion (Image Variations) we use the same approach as MindEye + Versatile Diffusion except we map from voxels to the 768 final layer outputs of ViT-L/14 (same architecture as "4 ResBlocks + Only CLS" in Table 2). For the diffusion prior we fine-tune an oopen-sourced implementation of the DALL-E prior that was trained to generate CLIP image embeddings from CLIP text embeddings using 250M image-caption pairs from LAION-Aesthetics We note that using this pretrained prior works much better than training from scratch, suggesting that a similar large-scale pretrained prior for Versatile Diffusion might further improve fMRI reconstructions. We
20

---

Seen Reconstruction image
Reconsttuction
Sean image
Recxnstruction
image
Reconstraction
imge
Reconstrucion
Stable Difuzionn jimage Varations)
Latte (CLIP-VITI CPPPIIT S0)
Figure 10: Corresponding reconstructions Figure when swapping Versatile Diffusion with Stable Diffusion (Image Variations) or Lafite.
this MindEye model variant both for reconstructing via Stable Diffusion (Image Variations) and for retrieving the top- nearest neighbors in CLIP space for LAION-5B image retrieval. This is because the CLIP Retrieval client [30] only has precomputed CLIP embeddings for the final layer of CLIP, not the last hidden layer as used by Versatile Diffusion.
For Lafite tried to replicate the same approach as Lin al. [ but with inputs from the MindEye MLP backbone. Lafite is a conditional image generation pipeline that uses the CLIP-aligned voxel embeddings "condition vectors". In particular, Lafite leverages a StyleGAN that uses the CLIP embeddings as "style" vectors to generate images. Lafite's 's discriminator is trained to distinguish generated images from ground truth images and also to semantically align the CLIP embedding of the generated image with the condition vector using contrastive learning. Here we train two mapping models fmi and fice that map voxels to the final layer of CLIP ViT-B/32, where fms is contrastively aligned with CLIP image embeddings and fm is contrastively aligned with CLIP text embeddings. We used the same contrastive learning schedule MindEye with BiMixCo for the first one-third of the training cycle and SoftCLIP for the rest. Note that Lafite doesn't 't require training a prior so we only train the MLP backbone. Once the mapping models fmi and fmo are trained, we follow Lin et al. M| to fine-tune pretrained language-free Lafite model provided by (32], Finally, we use low-level "perceptual" pipeline by aligning layer-2 ResNet features of the generated image with those of the ground truth image using contrastive learning. The ResNet was trained using sele-suue
VICReg loss [80].
Method
LeveLevel
High-Level
PixCorr 1 SSIM f Alex(2) 1 Alex(5) Incep 1 CLIP 1 Eff J. SwAV +
Versatile Diffusion (S1)
.300
.337
97.4%
98.7%
94.5% 94.6%
.630
.358
SD Image Variations (S1)
.376
.300
95.7%
96.4%
92.5%
92.5%
.734
.446
Lafite (S1)
.411
.304
92.5%
98.1%
93.7%
87.0% .701
.436
Table 6: Evaluations for Subject across three pretrained final image generation models (Lafite was fine-tuned the same manner as Lin (T]). Both Versatile Diffusion and Stable Diffusion (image variations) used an img2img strength of .15 with the low-level reconstructions output from MindEye (Lafite is a GAN and not compatible with the same img2img process).
21

---

Reconntruction Seen
Reconstruction
Seen
Resonstructon
Seen
Reconstracton
Reconstruction
Bubjectt
Suseer 3
Figure 11:
Corresponding reconstructions to Figure for the other NSD participants.
A.8
Subject-Speeific Results
Here we depict reconstructions across the other 3 NSD participants in Figure subject evaluations metrics in Table 7
with individual
Method
Low-Level
High-Level
Retrieval
PixCorr f SSIM 1 Alex(2) Alex(5) Incep 1 CLIP t Eff SwAV
Image
Brain f
MindEye (Subj
.390
.337
97.4%
98.7%
94.5%
94.6%
.630
.358
97.2%
94.7%
MindEye (Subj 2)
.318
.327
95.8%
98.1%
93.2%
93.7%
.656
.368
97.1%
93.9%
MindEye (Subj 3)
.265
.311
93.2%
97.8%
94.9%
94.9%
.628
.353
90.7%
85.7%
MindEye (Subj 4)
.261
.316
92.3%
96.6%
92.4% 93.0% .666
.387
89.4%
85.9%
Table 7:
MindEye retrieval and reconstruction performance individual participants. averaged across participants for the values shown Table
These scores were
A.9 Single- Trial Results
In the main paper we report results from the test dataset following the standard approach of averaging voxels across the three same-image repetitions. Reconstruction evaluations using only one brain sample for each image is shown Table with example reconstructions in Figure 12
22

---

Image
Reconstrucion
Image
Reconsmaction
Image
Reconstruction
mage
Reconstruction
Sean imago
Resoonstrucion
Figure 12: Corresponding reconstructions Figure D Jsiig brain activity from only the first sample of every image. This contrast Figure | which reconstructed from brain activity averaged across three same-image repetitions.
Method
Low-Level
High-Level
Retrieval
PixCor
SSIM Alex(2) f Alex(5) f Incep 1 CLIP Eff 1 SwAV Image
Brain
MindFye
.309
.323
94.7%
97.8%
93.8% 94.1%
.645
.367
93.6%
MindFye (single-trial)
255
.308
91 31.6%
95.9%
91.3%
91.6%
.691
.398
80.3%
77.6%
MindEye (single-, S1)
.329
.323
94.8%
97.3%
92.8%
92.7%
.680
.387
89.0%
86.5%
MindEyc (single-, $2)
.267
.311
93.1%
96.9%
91.5%
91. 11.2%
.687
.398
88.5%
86.1%
MindEye (single- $3)
.217
.297
90.2%
96.3%
93.2%
94.0%
.671
.381
75.3%
71.8%
MindEye (single- $4)
-209
.302
88.3%
93.1%
87.6%
88. 7% .727
.427
68.5%
66.1%
Table 8: MindEye retricval and reconstruction performance for single-trial trial brain activations, chosen randomly out of three possible samples per unique image. Other than using single-trial trial brain activity, the same settings were used as in Table
23

---

A.10
Performance with varying dataset size
Method
LoveLevel
High-Level
Retrieval
PixCorr SSIM f Alex(2) 1 Alex(5) Incep 1 CLIP f Eff SwAV 4 Image f Brain f
Data (High-Leevel)
.209
.318
92.8%
98.0%
94.5% 94.8% .635
.361
97.2%
94.7%
Half Data (High-Level)
.149
.276
87.7%
94.3%
87.1%
90.1%
.738
.421
77.5%
60.8%
2-Sessions (High-Level) .119
.281
81.0%
88.2%
79.2%
84.1%
.824
.472
17.9%
12.0%
Table 9: Quantitative comparison of MindFye performance with varying dataset sizes Subject with the high-level pipeline. Half Data corresponds to MindEye trained with half of the training samples randomly removed. 2-Sessions corresponds to MindEye trained with random selection of 500 training image samples (or 1,500 training IMRI samples given 3 repetitions per image), equivalent to the number of samples collected across scan sessions. Notably, image and brain retrieval metrics maintained state-of -f-thh--rtttttt performance when training the model with half of the training samples removed, and reconstruction performance remained competitive with previous models even with reduced training data. This suggests that our MindEye approach flexible to being trained with smaller datasets.
A.11 Model size comparison with other methods
Method
Parameter Count
Lin et al.
1.17M deep models + StyleGAN
Low Level
37M linear regression model
Takagi et al.
High Level
450M linear regression model
Low Level
1.45B linear regression model
Ozcelik et al.
High Level
257 separate 12M linear regression models
Low Level
206M residual MLP CNN decoder model
MindEye
High Level
996M residual MLP + diffusion prior model
Table 10: Comparison of MindEye parameter count with other competing methods, Other methods primarily rely on linear regression or relatively small deep models.
24


Reviews:
[{"review_id": "I00S7sO3iu", "reviewer": "Reviewer_DnYs", "summary": "The paper improves the fMRI reconstruction method using contrastive learning strategy and diffusion prior model. The concept is relative simple but the details of the proposed method, which is the key to make difference, is well-implemented. First, the BiMixCo implements a contrastive loss between the fMRI voxel representation and the CLIP image representation with the utilization of mixup augmentation of the fMRI vectors. Second, the Diffusion prior is implemented on the fMRI representation after the MLP backbone (which is the input to the MLP projector of which output is the input the contrastive loss) for the generating the representation for reconstructing the image. Moreover, the paper implements additional low-level pipeline based on VAE and Stable Diffusion so that an initialization of reconstruction can be attained. The overall reconstruction and retrieval performance significantly outperforms the baselines with large margin, particularly for the retrieval performance.  ", "questions": "Please see my comments in Weakness. ", "limitations": "- The exact process for the evaluation for the retrieval task is not very clearly described. \n- The performance gap between the reconstruction and retrieval is not clearly explained. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- While the problem is not new and the proposed methods are clever combinations of existing techniques rather than brand new ones, the overall fMRI-based reconstruction and retrieval performances are impressive. \n- Both retrieval and reconstruction results are presented with thorough ablation studies to justify the modeling choice. \n- The visualization results are compelling and convincing.", "weaknesses": "- The performance gaps between MindEye and other baselines are vast while those for reconstructions are not as much. It seems like the reconstruction results for the baselines are copied from the original papers, while the retrieval results are reproduces..Is that correct? What is the main reason for such huge difference? The representations used for reconstruction and retrieval should not be that different, so such a wide gap is little mysterious. In case the retrieval results for the baselines are reproduces, why not also reproduce the results for the reconstructions? \n- The effect of BiMixCo is not very convincing since as shown in Table 4, it seems to be helpful only for the retrieval task while not as much for the reconstruction task. Would there be some additional justification? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The performance gap between the reconstruction and retrieval is not clearly explained. The performance gaps between MindEye and other baselines are vast while those for reconstructions are not as much. It seems like the reconstruction results for the baselines are copied from the original papers, while the retrieval results are reproduces..Is that correct? What is the main reason for such huge difference? The representations used for reconstruction and retrieval should not be that different, so such a wide gap is little mysterious. In case the retrieval results for the baselines are reproduces, why not also reproduce the results for the reconstructions?\n\nA1: The related papers all shared the goal of reconstructing images from brain activity, not retrieving images. The one exception is Lin et al. (2022), which did report retrieval performance in their paper which is shown in our Table 1 (aka this is copied from their paper, not reproduced). We had to reproduce the retrieval results for Ozcelik et al. because the original paper reported reconstruction performance but not retrieval performance (this is mentioned in Appendix A.5). \n\nTherefore, one explanation for the discrepancy between our huge improvement to retrieval performance but only moderate improvement to reconstruction performance compared to previous work is that the authors of other papers were never aiming to improve retrieval performance. That is, other work never tried to decouple retrieval and reconstruction objectives.\n\nTo get such high retrieval performance we engineered separate retrieval and reconstruction submodules. This is necessary because the objective for retrieval performance is distinct from the objective for reconstruction: minimizing cosine similarity between paired samples does not translate to minimizing mean squared error of image latents. We discuss our evidence that these objectives trade-off with each other in lines 129-130 and 239-240 of the paper (the diffusion prior’s role cannot be fulfilled by simply adding MSE loss to the MLP projector, and using both contrastive and MSE losses to the MLP backbone does not work well). In other words, representations used for reconstruction and retrieval actually are expected to be quite different.\n\n> Q2: The effect of BiMixCo is not very convincing since as shown in Table 4, it seems to be helpful only for the retrieval task while not as much for the reconstruction task. Would there be some additional justification?\n\nA2: We expect the drop in reconstruction performance due to BiMixCo to be because of how mixup generates synthetic datapoints through linear interpolation. For the retrieval task, we do not need to mix the targets as the contrastive objective optimizes the relative distance of the model predictions with positive and negative samples. However, for the reconstruction task we need absolute targets for the mixed inputs. We generated new targets by mixing the original targets in the same ratio as the mixup inputs. This causes a slight shift in the distributions of target embeddings at train time, leading to a drop in test time performance. We now clarify this in the second-to-last paragraph of the “Contrastive Learning” section. Relatedly, other recent works (Liu & Wang, 2023; Yu, Wang, & Wu, 2021) have also shown that stopping mixup after a certain number of epochs improves performance by reducing the train test disparity (this is mentioned in L107 in the paper). \n\nAlso note that BiMixCo as a loss is only applied to the retrieval submodule. It doesn’t have a direct effect on the diffusion prior, except through the common MLP backbone and the slightly altered target distribution. We observe that BiMixCo gives the highest retrieval performance but slightly hurts reconstructions (Table 4). Our final schedule combining BiMixCo + SoftCLIP losses strikes the best balance between retrieval and reconstruction performance in a single model.\n\n> Q3: The exact process for the evaluation for the retrieval task is not very clearly described.\n\nA3: We have now updated the text to more clearly describe the retrieval evaluation process:\n\n“We followed the same procedure as Lin et al. [11] for calculating the retrieval metrics reported in Table 1. Brain retrieval performance was calculated according to the following procedure: for each test image, the image is converted to a CLIP image embedding and we compute the cosine similarity to both its respective ground truth disjointed CLIP fMRI embedding as well as 299 other randomly selected disjointed CLIP fMRI embeddings in the test set. For each test sample, success is determined if the cosine similarity is greatest between the ground truth CLIP embedding and its respective fMRI embedding (aka top-1 retrieval performance, chance=1/300). We average retrieval performance across all test samples and repeat the entire process 30 times to account for the variability in random sampling of batches. For image retrieval, the same procedure is used except image and brain samples are flipped such that the goal is to find the corresponding paired CLIP image embedding out of 300 possible CLIP embeddings in the batch. Lin et al. [11] refer to image retrieval as “forward retrieval” and brain retrieval as “backward retrieval” in their paper.”\n\nFor context, the above paragraph will replace the current explanation on page 5 that begins with \"To compare our retrieval performance to other papers we average ...\"\n"}, {"review_id": "DoMK0IxYAO", "reviewer": "Reviewer_M8r4", "summary": "This paper introduces a method for fMRI-to-image conversion that facilitates realistic image reconstruction from brain recordings. The proposed method comprises two stages: semantic (high-level) and perceptual (low-level) pipelines. In the semantic stage, fMRI voxels are projected into the image space using a Multilayer Perceptron (MLP) and a diffusion prior. Notably, the paper employs the contrastive learning of the CLIP model to map fMRI data into a pre-trained CLIP model. Subsequently, the diffusion prior is used to map CLIP text embedding to image embedding. On the other hand, in the perceptual pipeline, image embeddings are mapped using an MLP projector and a CNN decoder. The method's experimental results are derived using the NSD dataset.", "questions": "I wonder how the proposed model manages the domain shift when the structure of the training functional magnetic resonance imaging (fMRI) data significantly differs from that of the downstream task.  \n\nRegarding the architectural improvement ablation, what about potential implications of training the overarching concept using a more compact architecture with fewer parameters. It would be interesting to explore if a streamlined model could yield similar or improved results while offering more efficient computational processing and potentially mitigating the risk of overfitting. \n\nWhat about the computational complexity of the proposed method, specifically concerning image generation? How does this model compare to other methods in terms of computational demands? \n\nThe computation complexity would be crucial for understanding its practical applicability, particularly in settings where computational resources may be limited. \n\nFinally, I am curious about the choice of dataset. The paper utilizes NSD dataset for model training and evaluation. Could the model generalize well when applied to different datasets, specifically smaller ones? Moreover, when the dataset's size is reduced, can the model maintain its performance and possibly outperform comparative methods? \n", "limitations": "Yes, the paper discusses the possible limitation of the current work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method presented in the paper is clear, and the validity of the claims is compelling. \n\nI also appreciate the experimental evaluation conducted, which yields promising results both from a quantitative and qualitative standpoint. \n\nThe paper presents a commendable ablation analysis of various loss functions and data augmentation strategies.\n", "weaknesses": "I think the main weakness of the method is the use of pre-trained models and evaluations with large datasets only. I was wondering what will happen when we train a model without having a pretraining model.  \n\nHaving employed contrastive learning in the form of CLIP, with other proposed modules, I think the overall novelty of the paper might not be very high. However, I think the main contribution of the paper is clear and robust. \n\nThe presented method is quite large which makes it hard for building it with restricted resources. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: I think the main weakness of the method is the use of pre-trained models ... the overall novelty of the paper might not be very high.\n\nA1: MindEye relies on external state-of-the-art models that were trained on billions of image and text data samples. Critically, none of these existing models were trained with brain data and NSD provides less than 30,000 training fMRI samples per participant, which is orders of magnitude fewer data points than used for models like CLIP or Versatile Diffusion. Part of the novelty of our approach is to leverage models like CLIP that were trained with massive datasets as a teacher to guide the training of our brain models where we have a relative scarcity of data. \n\n> Q2: The presented method is quite large which makes it hard for building it with restricted resources. What about the computational complexity ...\n\nA2: Even though the parameter count for MindEye is high, MindEye (including both high- and low-level pipelines) can be trained on a single A100 in less than 18 hours. This efficiency is due to the bulk of the parameters stemming from MLPs, which are faster to compute than transformers or CNNs. We now mention these details in the “Diffusion Prior” section.\n\nRegarding image reconstruction, our generative model is pre-trained. Any off-the-shelf image generation method that accepts CLIP embeddings can be used with MindEye, meaning more efficient models can be swapped in depending on computational cost requirements. \n\nAlso note that at inference time, our diffusion prior (which is trained from scratch) can be dropped if only retrieval is needed, and as stated in Appendix A.2.1, our diffusion prior is faster than the DALLE-2 diffusion prior because we use 100 timesteps instead of 1000, and we modified its architecture to no longer have learnable queries and to instead directly predict denoised CLIP embeddings.\n\nRegarding comparison of computational demands with other reconstruction papers, we have added a new table to Appendix A.10 (see pdf attached to global response) comparing MindEye’s parameter counts with other methods. Lin et al. (2022) use 2 CNN mapper networks of size 1.17M each and also finetune a Lafite based StyleGAN. Takagi et al. (2022) use a linear model with 450M params for their high level pipeline and a linear model with 37M params for low level. Ozcelik et al. (2023) use 257 individual linear models of 12M params each for their high level pipeline and a linear model with 1.45B params for low level. In contrast, MindEye uses an MLP + diffusion prior approach totaling 996M params for the high level pipeline and an MLP with 206M params for our low level pipeline.\n\n> Q3: Could the model generalize well when applied to different datasets, specifically smaller ones? I wonder how the proposed model manages the domain shift ...\n\nA3: As an initial exploration into the importance of dataset size to MindEye performance, we now report performance for Subject 1 using models trained on reduced subsets of the complete training data in Appendix A.9 (attached pdf to our global response). \n\nThese results show that even with half the training samples, MindEye still achieves state-of-the-art retrieval performance and competitive reconstruction performance. Even with just 500 training image samples (less than 6% of the full training data), results remained competitive with previous models (albeit no longer state-of-the-art), as shown via the “2-Sessions” model evaluations.\n\nThis suggests that MindEye is viable for datasets with much less data than NSD, although future work is needed to directly test performance of this approach to new datasets. \n\nRegarding the topic of domain shift, a limitation that we discuss in the paper and which will be the focus of future work is that all MindEye models are subject-specific, meaning that we cannot train a model on NSD for Subject 1 and then evaluate that model on other subjects or non-NSD datasets. One reason for this is that each person has a differently shaped / sized brain and thus different numbers of voxels, resulting in different input dimensionalities and potential misalignment across voxels. Another reason is that different people have had different life experiences, so the functional organization of visual concepts in their brains could be different even if the brains are structurally aligned. Training on NSD and evaluating on non-NSD datasets would require that the same subject was likewise scanned in the separate fMRI dataset.\n\n> Q4: What about potential implications of training the overarching concept using a more compact architecture with fewer parameters?\n\nA4: Regarding the concern of overfitting with large models, note that our current MindEye model does not overfit despite its large size thanks to our training techniques. That said, we agree it is worthwhile to explore how more compact architectures could be used to achieve similar or improved model performance with reduced parameter count.\n\nIn our current approach, we predict all CLIP embeddings (size 257x768) from a hidden representation of size 4096, using a linear layer. This layer alone accounts for 808M parameters out of the 940M parameters in our backbone. Our initial attempts to reduce the size of this layer performed worse than the simple linear layer. \n\nIn future works, we will explore other methods to reduce the size of this layer by compressing the information in the 257 CLIP-image tokens into a smaller number of tokens. Another possible direction could be to reduce the size of the CLIP embedding dimension using PCA (see Ramesh et al., 2020). We now mention this future direction in the Conclusions section of the paper. Note that we cannot simply use the CLS token (size 1x768) as it does not contain all the necessary information (especially low-level image information) about the image. This is demonstrated by the inferior retrieval scores from the CLS token variant of MindEye as shown in Table 2.\n"}, {"review_id": "brGrZGmy0T", "reviewer": "Reviewer_Pn57", "summary": "The authors present MindEye, an innovative fMRI-to-image approach that utilizes contrastive learning and a diffusion prior for retrieval and reconstruction tasks. They conduct thorough comparisons, establishing MindEye's superiority over existing methods in terms of both qualitative and quantitative evaluations. The authors attribute the success of MindEye to its specialized submodules, improved training techniques, and models with increased parameterization. Additionally, they showcase MindEye's ability to preserve low-level image features in reconstructions through the use of img2img. Overall, MindEye represents a significant advancement in the field, pushing the boundaries of fMRI-based image retrieval and reconstruction.", "questions": "1. Exploring the applicability of the model to different patients would yield valuable insights. It raises the question of whether the model can provide meaningful results beyond the specific patient it was trained on.\n\n2. A sensitivity analysis investigating the relationship between image output and fMRI input would be highly intriguing. It would shed light on the crucial components of the input that contribute to generating the CLIP embedding and ultimately influence the quality of the reconstructed image.\n\n3. The authors suggest that increasing the number of parameters improves the results. It would be informative to know if they experimented with deeper networks to explore the potential benefits of a more complex architecture.\n", "limitations": "1. Lack of Methodological Originality: The study relies heavily on external state-of-the-art models, which diminishes the originality of the methodology. The authors predominantly employ simple MLPs and a pre-trained diffusion prior, which limits the novelty of their approach.\n\n2. Applicability to Different Patients: Exploring the generalizability of the model to diverse patient populations would be valuable. It is essential to understand if the model can yield meaningful results beyond the specific patient it was initially trained on.\n\n3. Sensitivity Analysis: Conducting a sensitivity analysis would provide valuable insights into the relationship between image output and fMRI input. Understanding which specific components of the input are crucial for generating the CLIP embedding and influencing the quality of the reconstructed image would enhance the understanding of the model's behavior.\n\n4. Deeper Networks and Parameterization: The authors suggest that increasing the number of parameters improves the results. It would be beneficial to know if they explored the use of deeper networks, as a more complex architecture may have potential benefits. Investigating the effects of different network depths could shed light on the impact of model complexity on performance.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors conducted a comprehensive comparison of MindEye with existing methods, employing both qualitative side-by-side comparisons and quantitative evaluations. The results demonstrate that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. Notably, MindEye excels at retrieving the exact original image even when faced with highly similar candidates, indicating that its brain embeddings retain fine-grained, image-specific information. This remarkable capability enables accurate image retrieval even from extensive databases such as LAION-5B. The paper is well written and the presentation is to the point.", "weaknesses": "The methodology employed in this study has some limitations in terms of originality, as a majority of the approach relies on external state-of-the-art models. The authors primarily train simple MLPs and utilize a pre-trained diffusion prior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback!\n\n> Q1: Lack of Methodological Originality: The study relies heavily on external state-of-the-art models, which diminishes the originality of the methodology. The authors predominantly employ simple MLPs and a pre-trained diffusion prior, which limits the novelty of their approach.\n\nA1: We now clarify that our diffusion prior is trained from scratch in the “Diffusion Prior” section. That is, our diffusion prior is trained from scratch to align our disjointed CLIP fMRI embeddings, which are then fed to a pretrained Versatile Diffusion model to output image reconstructions.  \n\nMindEye relies on external state-of-the-art models that were trained on billions of image and text data samples. Critically, none of these existing models were trained with brain data and NSD provides less than 30,000 training samples per participant, which is orders of magnitude fewer data points than used for models like CLIP or Versatile Diffusion. We argue that part of the novelty of our approach is to leverage models like CLIP that were trained with massive datasets as a teacher to guide the training of our brain models where we have a relative scarcity of data. We have now clarified this nuance in the “Conclusions” section.\n\n> Q2: Applicability to Different Patients: Exploring the generalizability of the model to diverse patient populations would be valuable. It is essential to understand if the model can yield meaningful results beyond the specific patient it was initially trained on.\n\nA2: A limitation that we discuss in the paper is that all MindEye models are subject-specific, meaning that they cannot be generalized to other subjects. One reason for the subject-specificity of the model is that each person has a differently shaped / sized brain and thus different numbers of voxels, resulting in different input dimensionalities and potential misalignment across voxels. Another reason is that different people have had different life experiences, so the functional organization of visual concepts in their brains could be different even if the brains are structurally aligned. A future goal our group is tackling is to change the model architecture to explicitly learn a shared-subject embedding space to potentially support across-subject decoding; based on prior “functional alignment” findings in the fMRI literature from Haxby, Ramadge, and others (e.g., “hyperalignment” and “shared response modeling” papers; Haxby et al., 2020, Chen et al., 2015) we think this approach has extensive promise. However, implementing the shared embedding space in MindEye requires substantially more work to complete and we do not have results yet for this exploration. \n\n> Q3: Sensitivity Analysis: Conducting a sensitivity analysis would provide valuable insights into the relationship between image output and fMRI input. Understanding which specific components of the input are crucial for generating the CLIP embedding and influencing the quality of the reconstructed image would enhance the understanding of the model's behavior.\n\nA3: Model interpretability methods like GradCAM (Selvaraju et al., 2016) and Network Dissection (Bau et al., 2017) could be used to determine the fMRI voxels that most strongly respond to the presence of certain image features. Similar work has already been done using the Natural Scenes Dataset in Sarch et al (2023), where they identify the most significant image features corresponding to every voxel. Their models are the reverse of MindEye, where an input image is used to predict the fMRI response for specific brain regions. Another direction could be to visualize reconstructions from synthetic fMRI inputs where voxels outside a given brain region are set to zero and voxels inside the brain region are set to an arbitrarily high value and fed into the pretrained MindEye model. Resulting reconstructions would emphasize the image qualities important to that brain region (see Lin et al., 2022; Ozcelik et al., 2023). Such sensitivity analysis could help visualize the functional specialization of the brain and the most important voxels for reconstructions. We now discuss the above prior work and this research direction in the Conclusions section.\n\n> Q4: Deeper Networks and Parameterization: The authors suggest that increasing the number of parameters improves the results. It would be beneficial to know if they explored the use of deeper networks, as a more complex architecture may have potential benefits. Investigating the effects of different network depths could shed light on the impact of model complexity on performance.\n\nA4: We observe diminishing returns from training larger models, because of the limited dataset size. The performance when going from 2 to 4 resblocks only improves when using skip connections, and even then the improvement is around 1% with a 33M increase in parameter count (Table 2). This suggests that a larger dataset would be needed for bigger models to demonstrate noticeable improvements."}, {"review_id": "dGuq4DHY5o", "reviewer": "Reviewer_pqrP", "summary": "This paper presents a novel fMRI-to-image approach (MindEye) that can achieve excellent reconstructions of natural scenes. The main idea is to retrieve and reconstruct viewed images from fMRI with brain activity information. The model consists of two parallel submodules. The retrieval submodule uses contrastive loss to retrieve images and generate features with high cosine similarity with the corresponding image embeddings. In the reconstruction submodule, the diffusion prior and CLIP image embedding are used to generate aligned semantic features. In order to better reconstruct the image, a separate encoder is trained to generate low-level features, and jointly output the reconstructed images with the previous semantic features. The experimental results also demonstrate the superior performance of the method.", "questions": "1. How about the comparisons with other methods in terms of computational cost and model parameters? \n2. For data augmentation method, is using generative methods to generate more samples an alternative?\n3. Is the diffusion prior module pre-trained? Also, I wonder what is the benefit of using a diffusion prior?", "limitations": "Please refer to the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Interesting idea. This paper provides a novel method to achieve fMRI-to-image task, and explores this task by introducing retrieval, diffusion prior and low-level information. Aside from the framework, the experimental results also fully illustrate its performance.", "weaknesses": "1. In this paper, most of the modules are existing models. The novelty of the paper requires further elaboration.\n2. Lack of theoretical proof, most of them are descriptions. For example, the aligned embeddings mentioned in the paper, whether there is a theoretical proof of the rationality of the aligned embeddings for image reconstruction.\n3. The network structure uses multiple large-scale models, such as Clip and Diffusion, which are relatively time-consuming, while the paper lacks an explanation of the computational cost.\n4. Limited datasets. Only NSD dataset is used, and the generalization of the model is not considered.\n5. The experiment is valid, only the performance of each sub-module is considered. The performance after removing a certain submodule is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Most of the modules are existing models. The novelty of the paper requires further elaboration. \n\nA1: MindEye relies on models trained on billions of image/text samples. NSD provides <30,000 fMRI samples per participant. We argue that part of the novelty of our approach is to use models like CLIP trained with massive datasets as a teacher to guide training of our brain models where we lack data. Note our diffusion prior is trained from scratch and that other novelties of our paper are summarized in our global response.\n\n> Q2: Lack of theoretical proof ... for example, ... theoretical proof of the rationality of the aligned embeddings for image reconstruction.\n\nA2: We now discuss existing proof for the rationality of aligned embeddings:\n\nMultimodal contrastive learning will always produce disjointed embeddings because of the “modality gap” phenomenon whereby encoding modalities into a shared space restricts the effective embedding space to a narrow cone in geometric space. Liang et al. (2022) show theoretically and empirically that contrastive learning induces a modality gap phenomenon. This modality gap phenomenon explains why our CLIP image embeddings and CLIP fMRI embeddings are isolated to different regions of the same shared embedding space (UMAP plots in Appendix A.4). We use pre-trained models that expect CLIP image embeddings as input, thus motivating our training of a diffusion prior to align disjointed embeddings.\n\n> Q3: How about the comparisons with other methods in terms of computational cost and model parameters?\n\nA3: MindEye is modular and doesn’t need all the large scale models at all times. CLIP is used only during training. Similarly, Versatile Diffusion is only needed during inference. Even though our parameter count is high, MindEye can be trained on a single A100 in less than 18 hours. This is because our models primarily consist of MLPs which are faster to compute than transformers or CNNs. \n\nAt inference time, the diffusion prior can be dropped if only retrieval is needed. As stated in Appendix A.2.1, our diffusion prior is also faster than the DALLE-2 diffusion prior as it only needs 100 timesteps instead of 1000. It is also more computationally efficient because we modified the architecture to not have learnable queries and to instead directly predict denoised CLIP embeddings. For reconstruction, any off-the-shelf image generation method that accepts CLIP embeddings can be used, depending on computational cost requirements. \n\nWe have added a new table to Appendix A.10 comparing MindEye’s parameter counts with other methods. \n\n> Q4: Limited datasets. Only NSD dataset is used, and the generalization of the model is not considered.\n\nA4: We agree that generalization is important – our ultimate goal is to show results that are comparable to our NSD findings in other, independently-collected datasets. The main challenge here is that all other fMRI datasets of this kind are much smaller than NSD, by an order of magnitude, and the sheer size of NSD is a major contributor to the very high quality of the results shown here (see Appendix A.9, attached as a pdf to our global response, to see how performance scales as a function of training set size). To obtain comparable performance on smaller datasets, we will need to find a suitable way of combining data across subjects; this would allow us, e.g., to train on large datasets like NSD and then fine-tune and test on a smaller dataset incorporating different subjects.  This “across-subject alignment” problem in fMRI data analysis is difficult because subjects differ in brain structure (which can lead to different input dimensionalities and potential misalignment across voxels) and in life experiences (which can lead to functional organization of visual concepts in their brains being different, even if their brains are structurally aligned). For the present work, we focused on optimizing single-subject decoding in the “high data” setting of NSD; going forward, we will be changing the model architecture to explicitly learn a shared-subject embedding space that supports across-subject decoding (we now discuss this in the Conclusions). Based on other alignment findings in the fMRI literature (e.g., “shared response modeling”, Chen et al., 2015), we think this approach has extensive promise. However, implementing the shared embedding space in MindEye requires substantially more work to complete and we do not have results yet for this exploration.\n\n> Q5: The performance after removing a certain submodule is not clear.\n\nA5: Performance with and without each of the 2 submodules is mentioned in Table 4. We also show the corresponding reconstructions for these ablations in Figure 8 in the Appendix. We also show retrieval and reconstruction performance when using just the backbone, backbone + projector, backbone + prior, and backbone + prior + projector.\n\n> Q6: For data augmentation method, is using generative methods to generate more samples an alternative?\n\nA6: Our model needs paired fMRI and visual stimulus data for which there are limited datasets. Establishing generative models for fMRI data is an important research problem in its own right, and at present there are not well-established methods for doing this. This is why we use mixup to create synthetic fMRI responses for MindEye.\n\nWe considered augmenting image samples by randomly replacing the prior’s target with embeddings of generated image variations, instead of the ground truth image. We expect this could make the prior better at modeling the distribution of target images. But this augmentation would also slow down the training pipeline considerably. \n\n> Q7: Is the diffusion prior module pre-trained? Also, I wonder what is the benefit of using a diffusion prior?\n\nA7: The diffusion prior is not pre-trained. We show benefits of using a diffusion prior in Table 4. Figure 9 in the Appendix shows UMAP projections of CLIP image embeddings before and after the diffusion prior.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their thorough comments, thoughts, and suggestions on our manuscript. We have done our best to answer all questions and concerns. We summarize some of the core revisions and clarifications below.

* New Appendix A.9 (see attached pdf to this response) table shows how MindEye performs given reduced subsets of the complete training data. Competitive performance was still observed even when training with less than 6% of the full training data. We also discuss how future work should explore generalization to other subjects and datasets and how this is not straightforward to immediately implement due to the nature of working with brain samples, which are intrinsically unique to the individual.
* Clarification on the novelties of our paper, especially in regards to the concern that there was a lack of novelty due to the use of large-scale pre-trained models. We note that Natural Scenes Dataset contains less than 30,000 training samples compared to the billions of samples used to train CLIP and Versatile Diffusion, and that figuring out the best way to leverage these larger models as teachers to guide the training of our brain models with relative scarcity of data is one of the novelties of this paper.
* Clarification that our diffusion prior is actually trained from scratch, and is much faster during inference than the DALLE-2 diffusion prior. This is because we only use 100 timesteps instead of 1000 timesteps, and we also implemented architectural changes (Appendix A.2.1), including no learnable queries and direct prediction of denoised CLIP embeddings, which allow the diffusion prior to be performed on a single A100. We also clarify that MindEye can work with any off-the-shelf image generation method that accepts CLIP embeddings depending on computational cost requirements. 
* Description of the computational resources used for training MindEye and comparison of model size with other reconstruction approaches (see new Appendix A.10 in attached pdf). Notably, we mention how MindEye, despite having a very large parameter count, is actually quite computationally efficient due to the majority of parameters stemming from linear layers (MLPs are more computationally efficient than transformers or CNNs). MindEye can be fully trained on a single A100 in less than 18 hours.
* We elaborate on the theoretical motivation for aligning disjointed CLIP embeddings by referring to the “modality gap” geometric phenomenon. This describes the underlying reason behind why contrastive learning produces disjointed embeddings and thus motivates our use of a diffusion prior for alignment.

Because a few reviewers brought up the concern of novelty, below we summarize the novel advances put forth by our paper (:

1. Our novel implementation of separate submodules within a single model was shown to be critical for attaining simultaneous state-of-the-art reconstruction and retrieval metrics.
2. Contrary to common expectations within the neuroimaging community, using a deep MLP with a parameter count orders of magnitude higher than the number of training samples did not produce overfitting and instead benefitted model performance.
3. We introduce a novel bidirectional version of mixup contrastive data augmentation that seems to work very well in our low sample setting.
4. Mapping voxels to Stable Diffusion’s VAE latent space produces state-of-the-art image reconstructions in terms of low-level image metrics.
5. We are the first paper to attempt large-scale image retrieval (using LAION-5B) from fMRI brain data inputs.

Meta-review: This paper showcases an exciting use of contrastive learning and diffusion to respectively identify the image seen by an individual from their brain activity and to reconstruct the image from the brain activity. While these two goals are not new, even to neurips, the current work has been found to both be cleanly implemented and have good qualitative and quantitative performance. The authors are expected to incorporate the new changes into their paper.
Decision: Accept (spotlight)