Title: Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors
Venue: NeurIPS 2023
Accepted: yes
Keywords: fMRI

computational neuroscience

mind reading

diffusion models
Abstract: We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.

Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors

Abstract

We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. Wecomprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates, indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.

1 Introduction

The problem of decoding environmental inputs and cognitive states from brain activity is fundamental to the field of neuroscience, where improved computational approaches allow for further understanding of brain mechanisms [1]. A neuroimaging methodology that has seen significant success in this domain is functional magnetic resonance imaging (fMRI), where neural activity is measured by detecting changes in blood oxygenation. fMRI decoding is already being used in

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

real-time clinical domains [2] and has potential for novel mind reading applications in brain-computer interfaces. Previous works mapped fMRI activity to the embeddings of image generation models via relatively simple mappings, usually ridge regression [3-5]. Here we propose MindEye, a novel approach that involves mapping via large-scale multilayer perceptrons (MLPs), contrastive learning, and diffusion models to achieve state-of-the-art image reconstruction. See Figure 1 for select samples of reconstructions. 1

MindEye learns to map flattened spatial patterns of fMRI activity across voxels (3-dimensional cubes of cortical tissue) to the image embedding latent space of a pretrained CLIP [7] model. MindEye has an MLP backbone and 2 specialized submodules for retrieval and reconstruction. The retrieval submodule is contrastively trained and produces 'disjointed CLIP fMRI' embeddings that have high cosine similarity with the corresponding image embeddings but differ in magnitude. To reconstruct images, we train a diffusion prior [8] from scratch to take in the outputs from the MLP backbone and produce aligned embeddings suitable as inputs to any pretrained image generation model that accepts CLIP image embeddings. In order to ensure that our reconstructions also match the original images' low-level features (e.g., color, texture, spatial position), we train a separate encoder that directly maps voxels to the embedding space of Stable Diffusion's [9] variational autoencoder (VAE), obtaining blurry image reconstructions that lack high-level semantic content but perform state-of-the-art on lowlevel image metrics. Combining the high-level 'semantic' pipeline with the low-level 'perceptual' pipeline in an img2img [10] setting allows MindEye to output state-of-the-art reconstructions across both low- and high-level image metrics.

In addition to image reconstruction metrics, our disjointed CLIP fMRI embeddings attain state-ofthe-art performance on image retrieval and brain retrieval metrics. Image retrieval refers to finding the original seen image out of a pool of other images given a brain sample, while brain retrieval refers to finding the brain sample given an image. MindEye finds exact (top-1) matches in the pool of NSD test samples with >90% accuracy for both image and brain retrieval, outperforming previous state-of-the-art [11, 4] which showed <50% retrieval accuracies. These results demonstrate that MindEye brain embeddings possess fine-grained exemplar-level signal.

Our main findings are: (1) Specialized submodules for retrieval (using contrastive learning) and reconstruction (using a diffusion prior) enable a single model to achieve state-of-the-art results across both tasks even though the objectives exhibit a tradeoff. (2) Mapping to a deep MLP with a parameter count orders of magnitude higher than previous methods does not produce overfitting and instead directly benefits model performance. (3) A novel bidirectional version of mixup contrastive data augmentation further improves model performance in this low-sample setting. (4) State-of-the-art reconstructions for low-level image metrics can be obtained by independently mapping to Stable Diffusion's VAE latent space. (5) fMRI-to-image retrieval can find the exact original image even among highly similar candidates, suggesting that fine-grained image-specific information is contained in brain embeddings, thus allowing retrieval to be scaled up to large-scale databases like LAION-5B to output images without generative models.

N images

fMRI voxels

CLIP ViT-L/14

N x 257 x 768

Diffusion prior

N x 15000

Low-level (perceptual) pipeline

fMRI voxels

N x 4 x 64 x 64

MLP

Reconstruction

Versatile

Diffusion

Image

Variations

High-level (semantic) pipeline

N x 257 x 768

MLP projector

MSE loss

Retrieval from LAION-5B

CLIP-fMRI

CLIP-Image

Contrastive loss

Aligned CLIP-fMRI (for reconstruction)

Disjointed CLIP-fMRI (for retrieval)

N x 257 x 768

N x 64 x 16 x 16

Low-level

reconstruction

Stable

Diffusion

image

encoder

N

x

4

x

64

x

64

N x 257 x 768

MSE loss

Upsampling

N x 15000

Stable

Diffusion

image

decoder

img2img

MLP backbone

Autoencoder latent

2 MindEye

MindEye consists of two pipelines (see Figure 2), a high-level (semantic) pipeline where fMRI voxels are mapped to the CLIP ViT-L/14 image space and a low-level (perceptual) pipeline where the voxels are mapped to the image embedding space of a VAE. Both pipelines follow a common structure: a residual MLP backbone followed by two task-specific submodules. For the high-level pipeline the submodules are an MLP projector and a diffusion prior. For the low-level pipeline the submodules are an MLP projector and a CNN decoder that performs 4x upsampling. For both pipelines we observe that training the projector submodule with a contrastive loss and the second submodule with mean squared error (MSE) loss gives the best performance.

2.1 High-Level (Semantic) Pipeline

The high-level pipeline is the core of MindEye as it maps voxels to CLIP image space to be fed through pretrained image generation models. We refer to it as a 'high-level' pipeline because CLIP embeddings are inherently more semantic than perceptual, given that CLIP image encoders were trained to maximize similarity with text captions (low-level features like color and object location are not typically preserved in these captions). MindEye can be used without the low-level pipeline, which simply aids to better preserve low-level image features during reconstruction.

The MLP backbone for our high-level pipeline maps flattened voxels to an intermediate space of size 257 × 768 , corresponding to the last hidden layer of CLIP ViT/L-14 (see Appendix 1 for PyTorch model code). The backbone consists of a linear layer followed by 4 residual blocks and a final linear projector. The embeddings from the backbone are fed to an MLP projector and a diffusion prior in parallel. The whole model is trained end-to-end with the prior getting an MSE loss and the projector getting a bidirectional CLIP loss. The projector outputs can be used for retrieval tasks and the diffusion prior outputs can be used by generative models to reconstruct images.

Contrastive Learning: Contrastive learning is an effective method for learning representations across modalities by maximizing cosine similarity for positive pairs while minimizing similarity for negative pairs. Previous work has shown the potential benefits of using contrastive learning alongside neural data [12, 13]. CLIP [7] is an example of a multimodal contrastive model that maps

images and text captions to a shared embedding space. MindEye is trained to introduce fMRI as an additional modality to the embedding space of a pretrained CLIP model, keeping the CLIP image space frozen as done with locked-image text tuning (LiT) [14]. We use the CLIP loss [7] as our contrastive objective. This loss is bidirectional and helps improve both image and brain retrieval.

Recent work [15-18] has explored novel data augmentation techniques that offer several benefits like improving performance, increasing robustness, and reducing training data requirements. Mixup [15] is one such technique that trains models on synthetic data created through convex combinations of two datapoint-label pairs [19]. Kim et al. [20] introduce MixCo, an extension of mixup that uses the InfoNCE loss, and show that MixCo improves classification performance in a semi-supervised setting. Based on the same principle, we modify the bidirectional CLIP loss to use MixCo. While Kim et al. [20] observed that MixCo gives largest performance benefit for smaller models, we observe that it also helps large models in low data regimes.

To combine MixCo with CLIP loss, we mix voxels using a factor λ sampled from the Beta distribution with α = β = 0 . 15 .

Here, x i and y i represent the i -th fMRI sample and image respectively. k i ∈ [1 , N ] is an arbitrary mixing index for the i -th datapoint and f represents the combined MLP and projector. p ∗ , p and t are L2-normalized. The CLIP loss with MixCo is defined as:

We term this bidirectional loss as BiMixCo. Here τ is a temperature hyperparameter, and N is the batch size.

Recent works [21, 22] have shown that stopping mixup augmentation after a certain number of epochs leads to better classification performance. As per these findings, we stop using mixup and switch from a hard contrastive loss to a soft contrastive loss one-third of the way through training. This improves our reconstructions without harming our retrieval performance (see Table 4). BiMixCo gives the highest retrieval performance but slightly hurts reconstructions (Table 4) likely due to how the reconstruction task needs absolute targets for the mixed inputs (which we generate by mixing the original targets in the same ratio as the mixup inputs), causing a slight shift in the distribution of target embeddings. Our final schedule combines BiMixCo and soft contrastive loss to strike the best balance between retrieval and reconstruction performance in a single model.

Our soft contrastive loss is inspired by knowledge distillation [23], where the authors argue that the softmax probability distribution produced by a powerful teacher model acts as a better teaching signal for a student than hard labels. To generate the soft labels we take the dot product of CLIP image embeddings in a batch with themselves. The loss (with bidirectional component omitted for brevity) is calculated between CLIP-CLIP and Brain-CLIP matrices as:

Diffusion Prior: Using a diffusion model to align the outputs of a contrastive learning model was inspired by DALL-E 2 [8], where a 'diffusion prior' was used to map CLIP text embeddings to CLIP

image space before using an unCLIP decoder to reconstruct images. Here we train our own diffusion prior from scratch to map CLIP fMRI embeddings to CLIP image space, which are then fed into a pretrained Versatile Diffusion model to generate image reconstructions. We modified an open-source implementation of the DALL-E 2 diffusion prior available on GitHub (see Appendix A.3.1). We used the same prior loss as Ramesh et al. [8]. Our total end-to-end loss is defined as:

We use α = 0 . 3 and switch from BiMixCo to SoftCLIP after one-third of the train cycle. All our models are trained on a single A100 GPU for 240 epochs with a batch size of 32. Despite a high parameter count, MindEye (including both high- and low-level pipelines) can be trained on a single A100 in less than 18 hours. This efficiency is due to the bulk of the parameters stemming from MLPs, which are faster to compute than transformers or CNNs.

The diffusion prior is critical for reconstruction because contrastive learning only incentivizes the CLIP fMRI embeddings to match the vector direction of the associated CLIP image embeddings. This generates disjointed embeddings as observed by Ramesh et al. [8]. Theoretically, multimodal contrastive learning will always produce disjointed embeddings because of the 'modality gap' phenomenon whereby encoding modalities into a shared space restricts the effective embedding space to a narrow cone in geometric space [24]. We use pre-trained models that expect CLIP image embeddings as input, thus motivating our training of a diffusion prior to align disjointed embeddings.

To rectify this issue, the diffusion prior learns a distribution of CLIP image embeddings conditioned on CLIP fMRI embeddings. UMAP [25] plots of disjointed CLIP fMRI embeddings next to aligned CLIP fMRI embeddings in Appendix A.5 show how the diffusion prior addresses the disjointed embedding spaces problem. We observe that the prior's role cannot be fulfilled by simply adding MSE loss to the MLP projector in Table 4. This is because there is a tradeoff between reconstruction and retrieval objectives and a model cannot effectively learn a single embedding space that does well on both.

2.2 Low-Level (Perceptual) Pipeline

The low-level pipeline maps voxels to the embedding space of Stable Diffusion's VAE. The output of this pipeline can be fed into the V AE decoder to produce blurry image reconstructions that lack high-level semantic content but exhibit state-of-the-art low-level image metrics. We use img2img [10] to improve our final image reconstructions in terms of low-level metrics, with minimal impairment to high-level metrics, such that we start the diffusion process from the noised encodings of our blurry reconstructions rather than pure noise.

The MLP backbone for our low-level pipeline follows the same architecure as our high-level pipeline, except that the final outputs are of size (16 , 16 , 64) . These are upsampled to (64 , 64 , 4) by a CNN upsampler. An MLP projector projects the backbone outputs to a 512 dimensional space where an auxiliary contrastive loss is applied. For more information on the low-level pipeline see Appendix A.3.2. See Appendix Figure 7 for example blurry reconstructions and Appendix Table 5 to see the effect of changing img2img strength on subsequent reconstruction metrics.

3 Results

For all experiments, we used the Natural Scenes Dataset (NSD) [26], a public fMRI dataset containing the brain responses of human participants passively viewing natural scenes from MS-COCO [27]. By utilizing MS-COCO, this dataset provides measured brain responses to rich naturalistic stimuli, allowing us to study how well low- and high-level image features are reconstructed by MindEye. We used the same standardized train/test splits as other NSD reconstruction papers [3, 4, 28], training subject-specific models for each of 4 participants. We averaged across three same-image repetitions for the test set (leaving 982 test samples) but not the training set (24,980 training samples), similar to Takagi and Nishimoto [3]. For more information on NSD and data preprocessing see Appendix A.2; for single-trial and reduced dataset results see Appendix A.9 and Appendix A.10.

Given a brain sample:

Find the corresponding image that was presented

fMRI voxels

1 x 15000

MindEye

1 x 257 x 768

MindEye finds the exact Top-1 image pair from

the test set of 982 images with 93.2% accuracy

(random chance = 0.1%)

Select image with highest CLIP cosine similarity

Retrieve nearest neighbor in CLIP space in LAION-5B database

Seen image

Retrieval

Seen image

Retrieval

Seen image

Retrieval

Seen image

Retrieval

Seen image

Top 1

Top 2

Top 3

Seen image

Top 1

Top 2

Top 3

Seen image

Top 1

Top 2

Top 3

3.1 Image/Brain Retrieval

Image retrieval evaluations reveal the level of fine-grained image-specific information contained in the predicted brain embeddings. For example, if the model is given a dozen pictures of zebras and the brain sample corresponding to viewing one of those zebras, can the model correctly find the corresponding zebra? If the model can correctly deduce that the brain sample corresponds to an image of a zebra but cannot deduce the specific image amongst various candidates, this would suggest that category-level information but not exemplar-specific information is preserved in the CLIP fMRI embedding. MindEye not only succeeds in this zebra example but also demonstrates 93.2% overall accuracy for Subject 1 in finding the exact original image within the 982 test images (see Figure 3).

Although we use the full test dataset for retrievals in Figure 3, we followed the same procedure as Lin et al. [11] for calculating the retrieval metrics reported in Table 1. Brain retrieval performance was calculated according to the following procedure: for each test image, the image is converted to a CLIP image embedding and we compute the cosine similarity to both its respective ground truth disjointed CLIP fMRI embedding as well as 299 other randomly selected disjointed CLIP fMRI embeddings in the test set. For each test sample, success is determined if the cosine similarity is greatest between the ground truth CLIP embedding and its respective fMRI embedding (aka top-1 retrieval performance, chance=1/300). We average retrieval performance across all test samples and repeat the entire process 30 times to account for the variability in random sampling of batches. For image retrieval, the same procedure is used except image and brain samples are flipped such that the goal is to find the corresponding paired CLIP image embedding out of 300 possible CLIP embeddings in the batch. Lin et al. [11] refer to image retrieval as 'forward retrieval' and brain retrieval as 'backward retrieval' in their paper.

We can scale up image retrieval using a pool of billions of image candidates. In Figure 3 we show results querying the LAION-5B dataset [29] using our CLIP fMRI embeddings. The final layer CLIP ViT-L/14 embeddings for all 5 billion images are available at knn.laion.ai, and can be queried for K-nearest neighbor lookup via the CLIP Retrieval client [30]. For each test sample, we first retrieve 16 candidate images using this method (using a variant of MindEye that maps voxels to the final

(2023)

(2023)

Takagi et al.

(2022)

MindEye

(ours)

Image seen

in MRI

(2023)

MindEye

(ours)

Image seen

in MRI

layer of CLIP, see Appendix A.7). The best image is then selected based on having the highest CLIP embedding cosine similarity to the CLIP fMRI embedding. This image retrieval approach is especially well-suited for tasks involving fine-grained classification, and can be used as an alternative to image reconstruction without a generative model (evaluations in Table 1).

3.2 fMRI-to-Image Reconstruction

The diffusion prior outputs from MindEye are aligned CLIP fMRI embeddings that can be used with any pretrained image generation model that accepts latents from CLIP image space. We evaluate the outputs of MindEye reconstructions across several models including Versatile Diffusion [6], Stable Diffusion (Image Variations) [31], and Lafite [32, 11]. Here we report results from Versatile Diffusion since it yielded the best results, and we report results from the other models in Appendix A.7. We qualitatively compare our reconstructions side-by-side with outputs from other fMRI-to-image reconstruction models in Figure 4 and quantitatively compare against other models in Table 1, demonstrating state-of-the-art MindEye reconstructions.

For each subject, for each test brain sample, we output 16 CLIP image embeddings from MindEye and feed these embeddings through the image variations pipeline of Versatile Diffusion. This produces 16 image reconstructions per brain sample. For our reconstructions we use 20 denoising timesteps with UniPCMultistep noise scheduling [33] and start the denoising process from the noised output of our low-level pipeline (img2img). We then select the best of 16 reconstructions by computing last hidden layer CLIP embeddings and picking the image with the highest cosine similarity to the disjointed CLIP fMRI embedding. This automatic second-order selection was inspired by DALL-E 2 [8], which used a similar process of selecting the best of 2 generated samples.

Two-way identification refers to percent correct across comparisons gauging if the original image embedding is more similar to its paired brain embedding or a randomly selected brain embedding. Comparison was performed for AlexNet [37] (second and fifth layers), InceptionV3 [38] (last pooling layer), and CLIP (final layer of ViT-L/14). We use the same settings as Ozcelik and VanRullen [4] for our metrics. For more details refer to Appendix A.6.

3.3 Ablations

In this subsection we try to explain where MindEye performance improvements come from through ablations. To study the effects of architectural changes and training strategies we train only the retrieval pipeline (no diffusion prior) for 120 epochs with batch size 300. All models in this section are trained on Subject 1. Table entries with * correspond to the final version of MindEye's settings.

Architectural Improvements: To study the effect of model depth and parameter count we train multiple MLPs of various sizes (Table 2). Among models that map to the last hidden layer of CLIP ViT-L/14, we observe a clear trend of increased performance with added residual blocks. For 2 blocks, the effect of skip connections is not too significant but at 4 blocks the model does significantly worse without them, indicating that skip connections are important for training deeper models.

Table 1: Quantitative comparison of MindEye retrieval and reconstruction performance against other models. Top and middle sections average across the same 4 participants (see Appendix A.8 for individual subject models), except Lin et al. [11] which only analyzed Subject 1. Middle section reflects outputs from only the high- or low-level pipeline, and metrics when evaluating images retrieved from LAION-5B. Bottom section compares our low-level reconstructions to the low-level reconstructions from Ozcelik and VanRullen [4] which only reported metrics for Subject 1. Image retrieval refers to the percent of the time the correct image was retrieved out of 300 candidates, given the associated brain sample (chance=0.3%); vice-versa for brain retrieval. PixCorr=pixelwise correlation between ground truth and reconstructions; SSIM=structural similarity index metric [34]; EfficientNetB1 ('Eff') [35] and SwAV-ResNet50 ('SwAV') [36] refer to average correlation distance; all other metrics refer to two-way identification (chance = 50%). Missing values are from papers not reporting all metrics or metrics being non-applicable. We followed the same image preprocessing as Ozcelik and VanRullen [4]. Previous state-of-the-art Ozcelik and VanRullen [4] results are directly comparable to MindEye as the same test set and Versatile Diffusion model were used. Bold indicates best performance within sections.

We also show a comparison with a 4-resblock model that maps to the final layer of CLIP (only the CLS classification token). This model has 7 × fewer parameters and does much worse than all other models. This indicates two things: (1) MindEye strongly benefits from a large parameter count MLP backbone and does not overfit even in the sample constrained settings of the NSD dataset, and (2) the fMRI voxels contain fine-grained information about images, allowing us to effectively predict all 257 CLIP image embeddings instead of just the CLS token.

Training Strategies (Losses and Data Augmentations): We observe that with InfoNCE, MindEye only does well on brain retrieval (Table 3). A similar trend was observed in Lin et al. [11]. We attribute this to InfoNCE being a one-sided loss that only optimizes for one retrieval objective. Simply replacing InfoNCE with CLIP loss significantly improves image retrieval. MixCo augmentation helps both unidirectional and bidirectional losses.

We also show the effect of training with our SoftCLIP loss. SoftCLIP improves over hard CLIP loss for brain retrieval but performs worse than BiMixCo. Our training regime combining SoftCLIP with BiMixCo gives the best image retrieval performance.

Reconstruction Strategies: To demonstrate the need for a separate diffusion prior, we train a version of MindEye where both contrastive and MSE losses are applied to the ouputs of the MLP backbone. We observe that this model does poorly in terms of retrieval metrics, indicating a tradeoff between retrieval and reconstruction objectives where it is difficult to learn a single embedding space that satisfies both objectives. Inspired by recent works in self-supervised learning [39-42], we decouple these losses using a separate MLP projector, where MSE loss is applied to the outputs of the MLP backbone and contrastive loss is applied to the outputs of the projector. This model does slightly worse in terms of reconstruction but is much better at retrieval. Finally, we train a model with a diffusion prior but no MLP projector. Contrastive loss is computed for the MLP backbone and MSE loss is computed for the diffusion prior. This model is comparable to high-level MindEye in terms of reconstruction but does worse in retrieval, giving further evidence of a tradeoff. Example reconstructions for these models are in Appendix Figure 8.

4 Related Work

In the 2000s, researchers demonstrated that visual information, such as spatial position [43], orientation [44, 45], and coarse image category [46, 47] could be decoded from fMRI signals using linear classifiers. With the introduction of generative adversarial networks [48], more sophisticated decoding became feasible as researchers mapped brain activity to the latent space of these models to reconstruct handwritten digits [49], human faces [50, 51], and natural scenes [52, 5, 53]. More recently, with the release of multimodal contrastive models like CLIP [7], diffusion models [54, 55] like Stable Diffusion [9], and new large-scale fMRI datasets [26], fMRI-to-image reconstructions have reached an unprecedented level of quality [4, 3, 28].

Lin et al. [11] reconstructed NSD images by mapping voxels to CLIP space (see also Wang et al. [56]) and fed outputs through a fine-tuned Lafite [32] GAN (MindEye reconstructions using Lafite in Appendix A.7). Differences from MindEye include using a convolutional model, no projector to separate contrastive loss from MSE loss, InfoNCE instead of CLIP loss, fine-tuning of a pretrained GAN, no diffusion prior, and mapping to both CLIP image and text space. Ozcelik and VanRullen [4] used a low- and high-level pipeline with Versatile Diffusion [6]. Differences include mapping to CLIP space via ridge regression, no contrastive learning or diffusion prior, and mapping to a VDVAE [57] for low-level reconstructions. Gu et al. [28] used a low- and high-level pipeline and extended on Ozcelik et al. [5] by reconstructing with IC-GAN [58]; they did not flatten voxels and mapped to SwAV [36] features with surface-based convolutional networks. Takagi and Nishimoto [3] used ridge regression to map to Stable Diffusion latents and CLIP text latents, using different voxel selections for different components. Mind-Vis [59] did not use the Natural Scenes Dataset but tackled the fMRI-to-image problem from the unique angle of first pre-training a masked brain model on a separate large-scale dataset. This enabled the authors to use a more informative latent as the input to their image reconstruction model. Mind-Video [60] subsequently extended on the Mind-Vis approach by reconstructing video rather than images. Overall, MindEye is unique in its use of reconstruction and retrieval submodules, a deep MLP backbone with 940 million parameters (parameter size comparison in Table 10), a diffusion prior for more accurate translation across brain and image modalities, and state-of-the-art reconstuction and retrieval results.

5 Conclusion

We present MindEye, a novel mental decoding approach that achieves state-of-the-art reconstructions of natural scenes presented to humans in the MRI machine. These reconstructions retain semantic and perceptual similarity to the original images due to the use of a combined high-level and low-level pipeline. The novel use of specialized submodules for contrastive-based retrieval and diffusion-based reconstruction allows MindEye to learn mappings for both tasks in parallel. MindEye brain latents contain fine-grained image-specific signal as demonstrated by the ability to select ground truth images out of a set of nearly 1,000 possible images (see Figure 3). We leveraged pretrained CLIP [7], a model trained with billions of image and text data samples, as a teacher to guide MindEye where we have a relative scarcity of brain data (less than 30,000 training samples per participant). Our diffusion prior submodule is trained from scratch and allows for accurate translation of brain embeddings into pretrained CLIP space such that any model that accepts CLIP image embeddings can be provided with CLIP fMRI embeddings without fine-tuning. This flexibility suggests that MindEye reconstructions will continue to improve as newer, more powerful image generation models are released.

Privacy Concerns & Societal Benefits: The ability to accurately reconstruct perception from brain activity prompts questions about broader societal impacts. For instance, it should be possible to generalize current reconstruction models from perception to mental imagery without training a new model [61-64]. However, current models are not capable of across-subject decoding and each NSD participant spent up to 40 hours in the MRI machine to procure sufficient training data (see Appendix 9 for results using a subset of the total training data). Furthermore, non-invasive neuroimaging methods in general require compliance because participants can easily resist decoding by moving their head or thinking about unrelated information [65]. MindEye is also limited to natural scenes such as those in MS-COCO; for other image distributions additional data collection and specialized generative models would be required. While high-quality image reconstruction via non-invasive neuroimaging is not currently practical for real-world applications, technology is constantly improving and it is important that brain data be carefully protected and companies collecting such data be transparent with their use.

Image reconstruction from brain activity can enable various potential societal benefits. Reconstructions are expected to be systematically distorted due to mental state, neurological conditions, etc. This could potentially enable novel clinical diagnosis and assessment approaches. For example, patients suffering from major depressive disorder might produce reconstructions where emotionally negative aspects of images are more salient [66]. MindEye results also suggest potential for improved locked-in (pseudocoma) patient communication via fine-grained visual communication beyond simple classification [67], as well as brain-computer interface performance if adapted to real-time fMRI analysis [68] or non-fMRI neuroimaging modalities.

Future Directions: Future directions we wish to explore include mapping individual subjects to a shared embedding space to enable training models that are generalizeable across people (e.g., [69, 70]), and exploring model intepretability approaches like GradCAM [71] or Network Dissection [72] to identify the fMRI voxels that most strongly respond to the presence of certain image features.

6 Open Research: 100% Transparent Volunteer-Driven Science

MindEye was openly developed through volunteer contributions in the MedARC Discord server. Source code was always accessible via a public GitHub repository throughout the lifespan of the project. Research discussions were held via public Discord channels, and weekly video conference calls were recorded and shared publicly. We continue to extend a global invitation to contribute to MedARC Neuroimaging & AI Lab projects to cultivate an internationally diversified, volunteer-driven research team composed of members from varied backgrounds possessing a wide array of expertise. We contend that fully transparent open-research initiatives such as this and others like EleutherAI, LAION, OpenBioML, and ML Collective could redefine the traditional framework of scientific research, democratizing entry into machine learning and medical research through the harnessing of crowd-sourced collective intelligence and community collaboration.

7 Author Contributions

For detailed author contributions see Appendix A.1.

References

Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fMRI. Neuroimage , 56(2):400-410, 2011. Publisher: Elsevier.

David Linden. Introduction. In fMRI Neurofeedback , pages 161-169. Elsevier, 2021. ISBN 978-0-12822421-2. doi: 10.1016/B978-0-12-822421-2.00008-9. URL https://linkinghub.elsevier.com/ retrieve/pii/B9780128224212000089 .

Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. preprint, Neuroscience, November 2022. URL http://biorxiv.org/lookup/ doi/10.1101/2022.11.18.517004 .

Furkan Ozcelik and Rufin VanRullen. Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion, March 2023. URL http://arxiv.org/abs/2303.05334 . arXiv:2303.05334 [cs, q-bio].

Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs, February 2022. URL http://arxiv.org/abs/2202.12692 . arXiv:2202.12692 [cs, eess, q-bio].

Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile Diffusion: Text, Images and Variations All in One Diffusion Model, March 2023. URL http://arxiv.org/abs/2211. 08332 . arXiv:2211.08332 [cs].

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv. org/abs/2103.00020 . arXiv:2103.00020 [cs].

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022. URL http://arxiv.org/abs/2204.06125 . arXiv:2204.06125 [cs].

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, April 2022. URL http://arxiv.org/abs/2112.10752 . arXiv:2112.10752 [cs].

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, January 2022. URL http://arxiv.org/abs/2108.01073 . arXiv:2108.01073 [cs].

Sikun Lin, Thomas Sprague, and Ambuj K. Singh. Mind Reader: Reconstructing complex images from brain activities, September 2022. URL http://arxiv.org/abs/2210.01769 . arXiv:2210.01769 [cs, eess, q-bio].

Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech from non-invasive brain recordings, August 2022. URL http://arxiv.org/abs/2208.12266 . arXiv:2208.12266 [cs, eess, q-bio].

Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature , 617(7960):360-368, May 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06031-6. URL https://www.nature.com/articles/s41586-023-06031-6 . Number: 7960 Publisher: Nature Publishing Group.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-Shot Transfer with Locked-image text Tuning, June 2022. URL http: //arxiv.org/abs/2111.07991 . arXiv:2111.07991 [cs].

Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv , abs/1710.09412, 2017.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6022-6031, 2019.

Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. ArXiv , abs/1805.09501, 2018.

Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ArXiv , abs/1912.02781, 2019.

Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Naumovich Vapnik. Vicinal risk minimization. In NIPS , 2000.

Sungnyun Kim, Gihun Lee, Sangmin Bae, and Seyoung Yun. Mixco: Mix-up contrastive learning for visual representation. ArXiv , abs/2010.06300, 2020.

Zixuan Liu and Ziqiao Wang. Over-training with mixup may hurt generalization. ArXiv , abs/2303.01475, 2023.

Hao Yu, Huanyu Wang, and Jianxin Wu. Mixup without hesitation. ArXiv , abs/2101.04342, 2021.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv , abs/1503.02531, 2015.

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning, October 2022. URL http: //arxiv.org/abs/2203.02053 . arXiv:2203.02053 [cs].

Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, September 2020. URL http://arxiv.org/abs/1802.03426 . arXiv:1802.03426 [cs, stat].

Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience , 25(1):116-126, January 2022. ISSN 1097-6256, 1546-1726. doi: 10.1038/s41593-021-00962-x. URL https://www.nature.com/articles/s41593-021-00962-x .

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 , Lecture Notes in Computer Science, pages 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1. doi: 10.1007/978-3-319-10602-1_48.

Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Decoding natural image stimuli from fMRI data with a surface-based convolutional network, March 2023. URL http://arxiv.org/abs/2212. 02409 . arXiv:2212.02409 [cs, q-bio].

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION5B: An open large-scale dataset for training next generation image-text models, October 2022. URL http://arxiv.org/abs/2210.08402 . arXiv:2210.08402 [cs].

Romain Beaumont. Clip Retrieval: Easily compute clip embeddings and build a clip retrieval system with them, 2022. URL https://github.com/rom1504/clip-retrieval . publicationType: misc; publisher: GitHub; journal: GitHub repository.

Justin Pinkney. Lambda Diffusers, 2022. URL https://github.com/LambdaLabsML/ lambda-diffusers . publicationType: misc; publisher: GitHub; journal: GitHub repository.

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. LAFITE: Towards Language-Free Training for Text-to-Image Generation, March 2022. URL http://arxiv.org/abs/2111.13792 . arXiv:2111.13792 [cs].

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models, February 2023. URL http://arxiv.org/abs/ 2302.04867 . arXiv:2302.04867 [cs].

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, April 2004. ISSN 1941-0042. doi: 10.1109/TIP.2003.819861. Conference Name: IEEE Transactions on Image Processing.

Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, September 2020. URL http://arxiv.org/abs/1905.11946 . arXiv:1905.11946 [cs, stat].

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, January 2021. URL http://arxiv.org/abs/2006.09882 . arXiv:2006.09882 [cs].

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/hash/ c399862d3b9d6b76c8436e924a68c45b-Abstract.html .

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2818-2826, June 2016. doi: 10.1109/CVPR.2016.308. ISSN: 1063-6919.

Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. ArXiv , abs/2006.07733, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9630-9640, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv , abs/2002.05709, 2020.

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. ArXiv , abs/2206.02574, 2022.

Bertrand Thirion, Edouard Duchesnay, Edward Hubbard, Jessica Dubois, Jean-Baptiste Poline, Denis Lebihan, and Stanislas Dehaene. Inverse retinotopy: Inferring the visual content of images from brain activation patterns. NeuroImage , 33(4):1104-1116, December 2006. ISSN 10538119. doi: 10.1016/j.neuroimage. 2006.06.062. URL https://linkinghub.elsevier.com/retrieve/pii/S1053811906007373 .

John-Dylan Haynes and Geraint Rees. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature Neuroscience , 8(5):686-691, May 2005. ISSN 1097-6256, 1546-1726. doi: 10.1038/nn1445. URL http://www.nature.com/articles/nn1445 .

Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience , 8(5):679-685, May 2005. ISSN 1097-6256, 1546-1726. doi: 10.1038/nn1444. URL http://www.nature.com/articles/nn1444 .

David D Cox and Robert L Savoy. Functional magnetic resonance imaging (fMRI) 'brain reading': detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage , 19(2):261-270, June 2003. ISSN 10538119. doi: 10.1016/S1053-8119(03)00049-1. URL https: //linkinghub.elsevier.com/retrieve/pii/S1053811903000491 .

James V. Haxby, M. Ida Gobbini, Maura L. Furey, Alumit Ishai, Jennifer L. Schouten, and Pietro Pietrini. Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex. Science , 293(5539):2425-2430, September 2001. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.1063736. URL https://www.science.org/doi/10.1126/science.1063736 .

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014. URL http://arxiv.org/ abs/1406.2661 . arXiv:1406.2661 [cs, stat].

Sanne Schoenmakers, Markus Barth, Tom Heskes, and Marcel van Gerven. Linear reconstruction of perceived images from human brain activity. NeuroImage , 83:951-961, December 2013. ISSN 10538119. doi: 10.1016/j.neuroimage.2013.07.043. URL https://linkinghub.elsevier.com/retrieve/pii/ S1053811913007994 .

Rufin VanRullen and Leila Reddy. Reconstructing faces from fMRI patterns using deep generative neural networks. Communications Biology , 2(1):193, December 2019. ISSN 2399-3642. doi: 10.1038/ s42003-019-0438-y. URL http://www.nature.com/articles/s42003-019-0438-y .

Thirza Dado, Ya˘ gmur Güçlütürk, Luca Ambrogioni, Gabriëlle Ras, Sander Bosch, Marcel van Gerven, and Umut Güçlü. Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space. Scientific Reports , 12(1):141, December 2022. ISSN 2045-2322. doi: 10.1038/ s41598-021-03938-w. URL https://www.nature.com/articles/s41598-021-03938-w .

Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLOS Computational Biology , 15(1):e1006633, January 2019. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1006633. URL https://dx.plos.org/10.1371/journal.pcbi.1006633 .

K. Seeliger, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M.A.J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage , 181:775-785, November 2018. ISSN 10538119. doi: 10.1016/j.neuroimage.2018.07.043. URL https://linkinghub.elsevier. com/retrieve/pii/S105381191830658X .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. URL http://arxiv.org/abs/2006.11239 . arXiv:2006.11239 [cs, stat].

Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models, October 2020. URL http://arxiv.org/abs/2006.09011 . arXiv:2006.09011 [cs, stat].

Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex, September 2022. URL https://www.biorxiv.org/content/10.1101/2022.09.27.508760v1 . Pages: 2022.09.27.508760 Section: New Results.

Rewon Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images, March 2021. URL http://arxiv.org/abs/2011.10650 . arXiv:2011.10650 [cs].

Arantxa Casanova, Marlène Careil, Jakob Verbeek, Michal Drozdzal, and Adriana RomeroSoriano. Instance-Conditioned GAN, November 2021. URL http://arxiv.org/abs/2109.05070 . arXiv:2109.05070 [cs].

Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding, March 2023. URL http://arxiv.org/abs/2211.06956 . arXiv:2211.06956 [cs].

Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity, May 2023. URL http://arxiv.org/abs/2305.11675 . arXiv:2305.11675 [cs].

Mark Stokes, Russell Thompson, Rhodri Cusack, and John Duncan. Top-Down Activation of ShapeSpecific Population Codes in Visual Cortex during Mental Imagery. Journal of Neuroscience , 29(5): 1565-1572, February 2009. ISSN 0270-6474, 1529-2401. doi: 10.1523/JNEUROSCI.4657-08.2009. URL https://www.jneurosci.org/content/29/5/1565 . Publisher: Society for Neuroscience Section: Articles.

Rainer Goebel, Rick van Hoof, Salil Bhat, Michael Lührs, and Mario Senden. Reading Imagined Letter Shapes from the Mind's Eye Using Real-time 7 Tesla fMRI. In 2022 10th International Winter Conference on Brain-Computer Interface (BCI) , pages 1-3, February 2022. doi: 10.1109/BCI53720.2022.9735031. ISSN: 2572-7672.

Thomas Naselaris, Cheryl A. Olman, Dustin E. Stansbury, Kamil Ugurbil, and Jack L. Gallant. A voxelwise encoding model for early visual areas decodes mental images of remembered scenes. NeuroImage , 105:215-228, January 2015. ISSN 1053-8119. doi: 10.1016/j.neuroimage.2014.10.018. URL https: //www.sciencedirect.com/science/article/pii/S1053811914008428 .

Leila Reddy, Naotsugu Tsuchiya, and Thomas Serre. Reading the mind's eye: Decoding category information during mental imagery. NeuroImage , 50(2):818-825, April 2010. ISSN 1053-8119. doi: 10. 1016/j.neuroimage.2009.11.084. URL https://www.sciencedirect.com/science/article/pii/ S1053811909012701 .

Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience , pages 1-9, May 2023. ISSN 1546-1726. doi: 10.1038/s41593-023-01304-9. URL https://www.nature.com/articles/ s41593-023-01304-9 . Publisher: Nature Publishing Group.

Anne C Mennen, Kenneth A Norman, and Nicholas B Turk-Browne. Attentional bias in depression: understanding mechanisms to improve training and treatment. Current Opinion in Psychology , 29: 266-273, October 2019. ISSN 2352-250X. doi: 10.1016/j.copsyc.2019.07.036. URL https://www. sciencedirect.com/science/article/pii/S2352250X19300016 .

Martin M. Monti, Audrey Vanhaudenhuyse, Martin R. Coleman, Melanie Boly, John D. Pickard, Luaba Tshibanda, Adrian M. Owen, and Steven Laureys. Willful Modulation of Brain Activity in Disorders of Consciousness. New England Journal of Medicine , 362(7):579-589, February 2010. ISSN 00284793. doi: 10.1056/NEJMoa0905370. URL https://doi.org/10.1056/NEJMoa0905370 . Publisher: Massachusetts Medical Society _eprint: https://doi.org/10.1056/NEJMoa0905370.

Grant Wallace, Stephen Polcyn, Paula P. Brooks, Anne C. Mennen, Ke Zhao, Paul S. Scotti, Sebastian Michelmann, Kai Li, Nicholas B. Turk-Browne, Jonathan D. Cohen, and Kenneth A. Norman. RTCloud: A cloud-based software framework to simplify and standardize real-time fMRI. NeuroImage , 257:119295, August 2022. ISSN 10538119. doi: 10.1016/j.neuroimage.2022.119295. URL https: //linkinghub.elsevier.com/retrieve/pii/S1053811922004141 .

Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge. A Reduced-Dimension fMRI Shared Response Model. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://papers.nips.cc/paper_files/ paper/2015/hash/b3967a0e938dc2a6340e258630febd5a-Abstract.html .

James V. Haxby, J. Swaroop Guntupalli, Andrew C. Connolly, Yaroslav O. Halchenko, Bryan R. Conroy, M. Ida Gobbini, Michael Hanke, and Peter J. Ramadge. A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex. Neuron , 72(2):404-416, October 2011. ISSN 08966273. doi: 10.1016/j.neuron.2011.08.026. URL https://linkinghub.elsevier.com/ retrieve/pii/S0896627311007811 .

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. International Journal of Computer Vision , 128(2):336-359, February 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007/s11263-019-01228-7. URL http://arxiv.org/abs/1610.02391 . arXiv:1610.02391 [cs].

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations, April 2017. URL http://arxiv.org/abs/1704. 05796 . arXiv:1704.05796 [cs].

Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets, May 2022. URL http://arxiv.org/abs/2202.00273 . arXiv:2202.00273 [cs].

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized Image Modeling with Improved VQGAN, June 2022. URL http://arxiv.org/abs/2110.04627 . arXiv:2110.04627 [cs].

Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife , 11:e77599, November 2022. ISSN 2050-084X. doi: 10.7554/eLife.77599. URL https://doi.org/10.7554/ eLife.77599 . Publisher: eLife Sciences Publications, Ltd.

Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging , 3:47-57, 2017.

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 1132-1140, 2017.

Kevin Turner. Decoding latents to rgb without upscaling. https://discuss.huggingface.co/t/ decoding-latents-to-rgb-without-upscaling/23204/2 , 2022. Accessed: 2023-05-23.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. ArXiv , abs/2210.01571, 2022.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. ArXiv , abs/2105.04906, 2021.

A Appendix

A.1 Author Contributions

PSS devised the project, led the team, developed the models, drafted the manuscript, and otherwise contributed to all parts of MindEye development. AB drafted the manuscript, developed the models, and contributed to all parts of MindEye development including creating the low-level pipeline, conception of BiMixCo and soft CLIP loss, and modification of the DALL-E 2 diffusion prior. JG developed the models, tracked/compared model variants, and significantly contributed to the MindEye codebase. SS conceived of and implemented LAION-5B retrieval using the CLIP Retrieval client and conducted various exploratory experiments. AN implemented the Lafite pipeline for MindEye reconstructions. EC conducted various initial explorations into using a diffusion prior for aligning voxels to CLIP space. AJD created the initial webdatasets used to train MindEye and created various model architectures to compare different mapping approaches. NV conducted various exploratory experiments mapping voxels to StyleGAN-XL [73] latent space. EY shared code to automatically identify identical images for qualitative comparisons and added code to ensure LAION-5B retrieval did not retrieve ground truth images. DW conducted various exploratory experiments and helped with project discussions. KAN oversaw the project and contributed valuable feedback. TMA oversaw the project, conducted initial explorations using VQGAN [74], and helped keep the project on-track through MedARC and Stability AI communication.

A.2 Additional Dataset Information

The Natural Scenes Dataset (NSD) [26] is a public 7-Tesla fMRI dataset containing the brain responses of several human participants each spending up to 40 hours in the MRI machine passively viewing images. These square-cropped images of natural scenes were sourced from the MS-COCO dataset [27]. Each of 9,000-10,000 unique images was presented for three seconds at a time, shown three times across 30-40 scanning sessions, totaling 22,000-30,000 trials of fMRI responses per participant. fMRI responses correspond to session-wise z-scored single-trial betas output from GLMSingle [75]. Following the procedure used in other reconstruction studies that used NSD [5, 28, 3], we train individual-subject models for the four participants who completed all scanning sessions (participants 1, 2, 5, and 7) and used a test set corresponding to the shared 1,000 images presented to every participant. This yields a dataset consisting of 24,980 training samples and 2,770 test samples-we average across the three same-image repetitions for the test set (leaving 982 test samples) but not the training set, similar to Takagi and Nishimoto [3]. We use preprocessed flattened fMRI voxels in 1.8-mm native volume space corresponding to the 'nsdgeneral' brain region, defined by the NSD authors as the subset of voxels in posterior cortex most responsive to the visual stimuli presented (between 13,000 to 16,000 voxels per participant). MindEye was developed using a training and validation set of Subject 1's data, with the test set (and other subjects' data) untouched until final training of models.

A.3 MindEye Architecture

PyTorch code for the MLP backbone and projector is depicted in Algorithm 1. Specifics on how we modified the open-source implementation of the DALL-E 2 diffusion prior are discussed in A.3.1.

A.3.1 Modifications from DALL-E 2 Diffusion Prior

The inputs for the diffusion prior are 257 backbone embeddings, 1 timestep embedding, and 257 noised CLIP image embeddings, and the output is 257 denoised CLIP image embeddings. Unlike the DALL-E 2 prior, we do not use learnable queries and instead directly predict denoised CLIP embeddings from the noised embeddings. This significantly saves on memory and allows us to train the backbone and prior end-to-end on a single GPU. We observe that adding absolute positional embeddings to the noised CLIP embeddings improves performance in the absence of learnable queries. We also observe that our prior can work with just 100 timesteps instead of 1000 as used in DALL-E 2. This makes our prior much faster at inference time. We conducted experiments with both causal and bidirectional attention and did not observe any significant difference in reconstruction performance. For simplicity we use bidirectional attention in our final model.

Algorithm 1 PyTorch code for MindEye MLP backbone and MLP projector

A.3.2 Low-Level Pipeline: Mapping to Stable Diffusion Variational Autoencoder

To map to Stable Diffusion's VAE latent space we use a low-level pipeline with the same architecture as the high level pipeline. We use a separate residual MLP backbone with 4 residual blocks that maps flattened voxels to a 16 × 16 × 64 dimensional latent space. The reconstruction submodule in the low-level pipeline is a CNN upsampler that upsamples these latents by 4 × to create embeddings of size (64 , 64 , 4) . The CNN upsampler uses a similar architecture to Stable Diffusion's VAE decoder, which does an 8 × upsampling. To create the targets for the upsampler we upsample NSD images to 512 × 512 through bilinear interpolation and encode them with the SD VAE encoder. The resulting (64 , 64 , 4) embeddings form the targets for the high-level pipeline.

Recent works in low-level vision (super-resolution, denoising, deblurring, etc.) have observed that mean absolute error performs better than mean squared error for pixel-level metrics like PSNR and SSIM [76, 77] due to better convergence properties. It has been shown that the 4-channel SD latent space effectively compresses images, and latents can be converted to RGB images with a linear mapping from latent space to pixel space [78]. We observe that the problem of mapping to SD embedding space follows the same properties as low-level vision tasks, such that mean absolute error performs better than mean squared error. We also experiment with using a "full reconstruction" loss where we reconstruct complete images using the SD VAE decoder and apply the loss in pixel space. This performs worse than only applying the loss in latent space and also requires significantly more GPU memory.

The contrastive submodule in the low-level pipeline acts as an auxiliary loss to improve the performance of the reconstruction submodule. It uses an MLP projector that maps the (16 , 16 , 64) backbone outputs to (16 , 16 , 512) . Since we do not care about retrieval performance for the low-level pipeline, we simply use SoftCLIP loss without BiMixCo. To maximize low-level performance we distill the knowledge of VICRegL [79] ConvNext-XXL instead of CLIP ViT. VICRegL with α = 0 . 75 is specialized for low-level tasks and achieves state-of-the-art linear segmentation results, unlike CLIP which has been trained with high-level text guidance.

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Retrieval

Seen image

Retrieval

Seen image

Retrieval

Seen image

Retrieval

Seen image

Retrieval

A.4 More Reconstructions / Retrievals

Images containing all 982 reconstructions and retrievals for each subject are on GitHub. Figure 5 depicts a subset of randomly selected reconstruction examples from Subject 1 (first try random selection of 30 samples). Figure 6 likewise depicts randomly selected examples from LAION-5B retrieval. Figure 7 depicts randomly selected example reconstructions from the low-level pipeline. Figure 8 depicts randomly selected reconstructions for the models described in 3.3.

A.5 UMAP Comparison

As depicted in Figure 9, the CLIP image and CLIP fMRI embedding spaces are disjointed before being fed through the diffusion prior. While the MLP projector does improve alignment compared to the outputs of the MLP backbone, the diffusion prior does a much better job at aligning the two spaces as shown by decreased euclidean distance between data points following UMAP dimensionality reduction.

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Only MLP

backbone

Backbone +

Projector

Backbone +

Prior

MindEye

(only BiMixCo)

MindEye

(0.33 BiMixCo)*

UMAP 1

UMAP 1

UMAP 2

UMAP 1

CLIP Image x MLP backbone

CLIP Image x MLP projector

CLIP Image x Diffusion prior

Avg.euclideandistance=5.815

Avg.euclideandistance=0.414

Avg.euclideandistance=0.180

A.6 Reconstruction Evaluations: Additional Information

Two-way identification was performed in the same manner as Ozcelik and VanRullen [4]. For each model, we computed the Pearson correlation between embeddings for the ground truth image and the reconstructed image, as well as the correlation between the ground truth image and a different reconstruction elsewhere in the test set. If the correlation for the former was higher than the latter, this was marked as correct. For each test sample, performance was averaged across all possible pairwise comparisons using the other 981 reconstructions to ensure no bias from random sample selection. This yielded 982 averaged percent correct outputs, which we averaged across to obtain the metrics reported in Table 1.

Retrieval evaluations for Ozcelik and VanRullen [4] were not reported in the original paper; we calculated image/brain retrieval ourselves with the help of the Brain-Diffuser GitHub repository.

A.7 Reconstructions from Stable Diffusion (Image Variations) and Lafite

We also attempted reconstructions using Stable Diffusion (Image Variations) [31] and Lafite [32] rather than Versatile Diffusion. Reconstructions from these models for Subject 1 are depicted in Figure 10, with metrics reported in Table 6.

For Stable Diffusion (Image Variations) we use the same approach as MindEye + Versatile Diffusion except we map from voxels to the 1 × 768 final layer outputs of ViT-L/14 (same architecture as "4 ResBlocks + Only CLS" in Table 2). For the diffusion prior we fine-tune an open-sourced implementation of the DALL-E 2 prior that was trained to generate CLIP image embeddings from CLIP text embeddings using 250M image-caption pairs from LAION-Aesthetics. We note that using this pretrained prior works much better than training from scratch, suggesting that a similar large-scale pretrained prior for Versatile Diffusion might further improve fMRI reconstructions. We

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Stable Diffusion (Image Variations)

Lafite (CLIP-ViT/B-32)

use this MindEye model variant both for reconstructing via Stable Diffusion (Image Variations) and for retrieving the top-16 nearest neighbors in CLIP space for LAION-5B image retrieval. This is because the CLIP Retrieval client [30] only has precomputed CLIP embeddings for the final layer of CLIP, not the last hidden layer as used by Versatile Diffusion.

For Lafite we tried to replicate the same approach as Lin et al. [11] but with inputs from the MindEye MLP backbone. Lafite is a conditional image generation pipeline that uses the CLIP-aligned voxel embeddings as "condition vectors". In particular, Lafite leverages a StyleGAN that uses the CLIP embeddings as "style" vectors to generate images. Lafite's discriminator is trained to distinguish generated images from ground truth images and also to semantically align the CLIP embedding of the generated image with the condition vector using contrastive learning. Here we train two mapping models f mi and f mc that map voxels to the final layer of CLIP ViT-B/32, where f mi is contrastively aligned with CLIP image embeddings and f mc is contrastively aligned with CLIP text embeddings. We used the same contrastive learning schedule as MindEye with BiMixCo for the first one-third of the training cycle and SoftCLIP for the rest. Note that Lafite doesn't require training a prior so we only train the MLP backbone. Once the mapping models f mi and f mc are trained, we follow Lin et al. [11] to fine-tune a pretrained language-free Lafite model provided by [32]. Finally, we use a low-level "perceptual" pipeline by aligning layer-2 ResNet features of the generated image with those of the ground truth image using contrastive learning. The ResNet was trained using a self-supervised VICReg loss [80].

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Subject 2

Subject 3

Subject 4

M

A.8 Subject-Specific Results

Here we depict reconstructions across the other 3 NSD participants in Figure 11, with individual subject evaluations metrics in Table 7.

A.9 Single-Trial Results

In the main paper we report results from the test dataset following the standard approach of averaging voxels across the three same-image repetitions. Reconstruction evaluations using only one brain sample for each image is shown in Table 8, with example reconstructions in Figure 12.

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

Seen image

Reconstruction

A.10 Performance with varying dataset size

A.11 Model size comparison with other methods

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel fMRI-to-image approach (MindEye) that can achieve excellent reconstructions of natural scenes. The main idea is to retrieve and reconstruct viewed images from fMRI with brain activity information. The model consists of two parallel submodules. The retrieval submodule uses contrastive loss to retrieve images and generate features with high cosine similarity with the corresponding image embeddings. In the reconstruction submodule, the diffusion prior and CLIP image embedding are used to generate aligned semantic features. In order to better reconstruct the image, a separate encoder is trained to generate low-level features, and jointly output the reconstructed images with the previous semantic features. The experimental results also demonstrate the superior performance of the method.\n\nSTRENGTHS:\nInteresting idea. This paper provides a novel method to achieve fMRI-to-image task, and explores this task by introducing retrieval, diffusion prior and low-level information. Aside from the framework, the experimental results also fully illustrate its performance.\n\nWEAKNESSES:\n1. In this paper, most of the modules are existing models. The novelty of the paper requires further elaboration.\n2. Lack of theoretical proof, most of them are descriptions. For example, the aligned embeddings mentioned in the paper, whether there is a theoretical proof of the rationality of the aligned embeddings for image reconstruction.\n3. The network structure uses multiple large-scale models, such as Clip and Diffusion, which are relatively time-consuming, while the paper lacks an explanation of the computational cost.\n4. Limited datasets. Only NSD dataset is used, and the generalization of the model is not considered.\n5. The experiment is valid, only the performance of each sub-module is considered. The performance after removing a certain submodule is not clear.\n\nQUESTIONS:\n1. How about the comparisons with other methods in terms of computational cost and model parameters? \n2. For data augmentation method, is using generative methods to generate more samples an alternative?\n3. Is the diffusion prior module pre-trained? Also, I wonder what is the benefit of using a diffusion prior?\n\nLIMITATIONS:\nPlease refer to the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present MindEye, an innovative fMRI-to-image approach that utilizes contrastive learning and a diffusion prior for retrieval and reconstruction tasks. They conduct thorough comparisons, establishing MindEye's superiority over existing methods in terms of both qualitative and quantitative evaluations. The authors attribute the success of MindEye to its specialized submodules, improved training techniques, and models with increased parameterization. Additionally, they showcase MindEye's ability to preserve low-level image features in reconstructions through the use of img2img. Overall, MindEye represents a significant advancement in the field, pushing the boundaries of fMRI-based image retrieval and reconstruction.\n\nSTRENGTHS:\nThe authors conducted a comprehensive comparison of MindEye with existing methods, employing both qualitative side-by-side comparisons and quantitative evaluations. The results demonstrate that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. Notably, MindEye excels at retrieving the exact original image even when faced with highly similar candidates, indicating that its brain embeddings retain fine-grained, image-specific information. This remarkable capability enables accurate image retrieval even from extensive databases such as LAION-5B. The paper is well written and the presentation is to the point.\n\nWEAKNESSES:\nThe methodology employed in this study has some limitations in terms of originality, as a majority of the approach relies on external state-of-the-art models. The authors primarily train simple MLPs and utilize a pre-trained diffusion prior.\n\nQUESTIONS:\n1. Exploring the applicability of the model to different patients would yield valuable insights. It raises the question of whether the model can provide meaningful results beyond the specific patient it was trained on.\n\n2. A sensitivity analysis investigating the relationship between image output and fMRI input would be highly intriguing. It would shed light on the crucial components of the input that contribute to generating the CLIP embedding and ultimately influence the quality of the reconstructed image.\n\n3. The authors suggest that increasing the number of parameters improves the results. It would be informative to know if they experimented with deeper networks to explore the potential benefits of a more complex architecture.\n\nLIMITATIONS:\n1. Lack of Methodological Originality: The study relies heavily on external state-of-the-art models, which diminishes the originality of the methodology. The authors predominantly employ simple MLPs and a pre-trained diffusion prior, which limits the novelty of their approach.\n\n2. Applicability to Different Patients: Exploring the generalizability of the model to diverse patient populations would be valuable. It is essential to understand if the model can yield meaningful results beyond the specific patient it was initially trained on.\n\n3. Sensitivity Analysis: Conducting a sensitivity analysis would provide valuable insights into the relationship between image output and fMRI input. Understanding which specific components of the input are crucial for generating the CLIP embedding and influencing the quality of the reconstructed image would enhance the understanding of the model's behavior.\n\n4. Deeper Networks and Parameterization: The authors suggest that increasing the number of parameters improves the results. It would be beneficial to know if they explored the use of deeper networks, as a more complex architecture may have potential benefits. Investigating the effects of different network depths could shed light on the impact of model complexity on performance.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for fMRI-to-image conversion that facilitates realistic image reconstruction from brain recordings. The proposed method comprises two stages: semantic (high-level) and perceptual (low-level) pipelines. In the semantic stage, fMRI voxels are projected into the image space using a Multilayer Perceptron (MLP) and a diffusion prior. Notably, the paper employs the contrastive learning of the CLIP model to map fMRI data into a pre-trained CLIP model. Subsequently, the diffusion prior is used to map CLIP text embedding to image embedding. On the other hand, in the perceptual pipeline, image embeddings are mapped using an MLP projector and a CNN decoder. The method's experimental results are derived using the NSD dataset.\n\nSTRENGTHS:\nThe method presented in the paper is clear, and the validity of the claims is compelling. \n\nI also appreciate the experimental evaluation conducted, which yields promising results both from a quantitative and qualitative standpoint. \n\nThe paper presents a commendable ablation analysis of various loss functions and data augmentation strategies.\n\nWEAKNESSES:\nI think the main weakness of the method is the use of pre-trained models and evaluations with large datasets only. I was wondering what will happen when we train a model without having a pretraining model.  \n\nHaving employed contrastive learning in the form of CLIP, with other proposed modules, I think the overall novelty of the paper might not be very high. However, I think the main contribution of the paper is clear and robust. \n\nThe presented method is quite large which makes it hard for building it with restricted resources.\n\nQUESTIONS:\nI wonder how the proposed model manages the domain shift when the structure of the training functional magnetic resonance imaging (fMRI) data significantly differs from that of the downstream task.  \n\nRegarding the architectural improvement ablation, what about potential implications of training the overarching concept using a more compact architecture with fewer parameters. It would be interesting to explore if a streamlined model could yield similar or improved results while offering more efficient computational processing and potentially mitigating the risk of overfitting. \n\nWhat about the computational complexity of the proposed method, specifically concerning image generation? How does this model compare to other methods in terms of computational demands? \n\nThe computation complexity would be crucial for understanding its practical applicability, particularly in settings where computational resources may be limited. \n\nFinally, I am curious about the choice of dataset. The paper utilizes NSD dataset for model training and evaluation. Could the model generalize well when applied to different datasets, specifically smaller ones? Moreover, when the dataset's size is reduced, can the model maintain its performance and possibly outperform comparative methods?\n\nLIMITATIONS:\nYes, the paper discusses the possible limitation of the current work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper improves the fMRI reconstruction method using contrastive learning strategy and diffusion prior model. The concept is relative simple but the details of the proposed method, which is the key to make difference, is well-implemented. First, the BiMixCo implements a contrastive loss between the fMRI voxel representation and the CLIP image representation with the utilization of mixup augmentation of the fMRI vectors. Second, the Diffusion prior is implemented on the fMRI representation after the MLP backbone (which is the input to the MLP projector of which output is the input the contrastive loss) for the generating the representation for reconstructing the image. Moreover, the paper implements additional low-level pipeline based on VAE and Stable Diffusion so that an initialization of reconstruction can be attained. The overall reconstruction and retrieval performance significantly outperforms the baselines with large margin, particularly for the retrieval performance.\n\nSTRENGTHS:\n- While the problem is not new and the proposed methods are clever combinations of existing techniques rather than brand new ones, the overall fMRI-based reconstruction and retrieval performances are impressive. \n- Both retrieval and reconstruction results are presented with thorough ablation studies to justify the modeling choice. \n- The visualization results are compelling and convincing.\n\nWEAKNESSES:\n- The performance gaps between MindEye and other baselines are vast while those for reconstructions are not as much. It seems like the reconstruction results for the baselines are copied from the original papers, while the retrieval results are reproduces..Is that correct? What is the main reason for such huge difference? The representations used for reconstruction and retrieval should not be that different, so such a wide gap is little mysterious. In case the retrieval results for the baselines are reproduces, why not also reproduce the results for the reconstructions? \n- The effect of BiMixCo is not very convincing since as shown in Table 4, it seems to be helpful only for the retrieval task while not as much for the reconstruction task. Would there be some additional justification?\n\nQUESTIONS:\nPlease see my comments in Weakness.\n\nLIMITATIONS:\n- The exact process for the evaluation for the retrieval task is not very clearly described. \n- The performance gap between the reconstruction and retrieval is not clearly explained.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}