MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

Authors: paul s scotti
mihir  tripathy
cesar  kadir
torrico  villanueva
reese  kneeland
  tong chen
ashutosh  narang
charan  santhirasegaran
jonathan  xu
thomas  naselaris
kenneth a norman
tanishq mathew abraham
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-01 16:55:01

Abstract: Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility.

Main paper: 
 introduction: 
 Spurred by the open releases of deep learning models such as CLIP Radford et al. (2021) and Stable Diffusion Rombach et al. (2022) , along with large-scale functional magnetic resonance imaging (fMRI) datasets such as the Natural Scenes Dataset Allen et al. (2022) where human participants were scanned viewing tens of thousands of images, there has been an influx of research papers demonstrating the ability to reconstruct visual perception from brain activity with high fidelity Scotti et al. (2023) ; Ozcelik and VanRullen (2023) ; Ozcelik et al. (2022) ; Takagi and Nishimoto (2022; 2023) ; Gu et al. (2023) ; Kneeland et al. (2023a; b; c) ; Ferrante et al. (2023a) ; Thual et al. (2023) ; Chen et al. (2023a; b) ; Sun et al. (2023) ; Gaziv et al. (2022) ; Mai and Zhang (2023) ; Xia et al. (2023) . FMRI indirectly measures neural activity by detecting changes in blood oxygenation. These patterns of fMRI brain activity are translated into embeddings of pretrained deep learning models and used to visualize internal mental representations Beliy et al. (2019) ; Shen et al. (2019a; b) ; Seeliger et al. (2018) ; Lin et al. (2019) .Visualization of internal mental representations, and more generally the ability to map patterns of brain activity to the latent space of rich pretrained deep learning models, has potential to enable novel clinical assessment approaches and brain-computer interface applications. However, despite all the recent research demonstrating high-fidelity reconstructions of perception, the practical adoption of such approaches to these settings has been limited if not entirely absent. A major reason for this is that the high-quality results shown in these papers use single-subject models that are not generalizable across people, and which have only been shown to work well if each subject contributes dozens of hours of expensive fMRI training data. MindEye2 introduces a novel functional alignment procedure that overcomes these barriers by pretraining a shared-subject model that can be fine-tuned using limited data from a held-out subject and generalizing to held-out data from that subject. This approach yields similar reconstruction quality to a single-subject model trained using 40× the training data. See Figure 1 for selected samples of reconstructions obtained from just 1 hour of data from subject 1 compared to their full 40 hours of training data in the Natural Scenes Dataset.In addition to a novel approach to shared-subject alignment, MindEye2 builds upon the previous SOTA approach introduced by MindEye1 Scotti et al. (2023) . In terms of similarities, both approaches map flattened spatial patterns of fMRI activity across voxels (3-dimensional cubes of cortical tissue) to the image embedding latent space of a pretrained CLIP Radford et al. (2021) model with the help of a residual MLP backbone, diffusion prior, and retrieval submodule. The diffusion prior Ramesh et al. (2022) is used for reconstruction and is trained from scratch to take in the outputs from the MLP backbone and produce aligned embeddings suitable as inputs to any pretrained image generation model that accepts CLIP image embeddings (hereafter referred to as unCLIP models). The retrieval submodule is contrastively trained and produces disjointed CLIP-fMRI embeddings that can be used to find the original (or nearest neighbor) image in a pool of images, but is not used to reconstruct a novel image. Both MindEye2 and MindEye1 also map brain activity to the latent space of Stable Diffusion's Rombach et al. (2022) variational autoencoder (VAE) to obtain blurry reconstructions that lack high-level semantic content but perform well on low-level image metrics (e.g., color, texture, spatial position), which get combined with the semantically rich outputs from the diffusion prior to return reconstructions that perform well across perceptual and semantic features.MindEye2 innovates upon MindEye1 in the following ways: (1) Rather than the whole pipeline being independently trained per subject, MindEye2 is pretrained on data from other subjects and then fine-tuned on the held-out target subject. (2) We map from fMRI activity to a richer CLIP space provided by OpenCLIP ViT-bigG/14 Schuhmann et al. (2022) ; Ilharco et al. (2021) , and reconstruct images via a fine-tuned Stable Diffusion XL unCLIP model that supports inputs from this latent space. (3) We merge the previously independent high-and low-level pipelines into a single pipeline through the use of submodules. (4) We additionally predict the text captions of images to be used as conditional guidance during a final image reconstruction refinement step.The above changes support the following main contributions of this work: (1) Using the full fMRI training data from Natural Scenes Dataset we achieve state-of-the-art performance across image retrieval and reconstruction metrics. (2) Our novel multi-subject alignment procedure enables competitive decoding performance even with only 2.5% of a subject's full dataset (i.e., 1 hour of scanning). 
mindeye2: 
 MindEye2 involves pretraining and then fine-tuning a single model where brain activity is mapped to the embeddings of pretrained deep learning models. During inference, these embeddings predicted from the brain are fed into frozen image generative models that translate from model space to pixel space. Our strategy to reconstruct seen images from brain activity using minimal training data is to first pretrain the model using data from 7 subjects (30-40 hours of scanning data each) and then to fine-tune the model using data from a held-out 8th subject. The full MindEye2 pipeline is depicted in Figure 2 .Single-subject models were trained/fine-tuned on a single 8xA100 80Gb GPU node for 150 epochs with a batch size of 24. Multi-subject pretraining was done with a batch size of 63 (9 samples per each of 7 subjects). Models were trained with Huggingface Accelerate Gugger et al. (2022) and DeepSpeed Rajbhandari et al. (2020) Stage 2 with CPU offloading.Every subject has a uniquely shaped brain with different functional organization, meaning that there needs to be an initial alignment step to ensure the model can handle inputs from different brains. Unlike anatomical alignment where every subject's brain is mapped to the same brain template Talairach and Tournoux (1990) ; Mazziotta et al. (2001) , we remain in subjects' native brain space and functionally align flattened spatial patterns of fMRI activity to a shared-subject latent space using subject-specific ridge regression. That is, each subject has a separate linear layer with weight decay to map the input fMRI voxels (13, 000 to 18, 000 voxels depending on the subject) to a 4096-dim latent.Following this initial linear layer, the rest of the model pipeline is shared across subjects without any subject-specific mappings. The alignment to shared-subject space is not trained independently-the whole pipeline is trained end-to-end where pretraining involves each batch containing brain inputs from all subjects. That is, we do not pretrain a model for each subject; rather, we pretrain a single model equally sampling across all the subjects except the held-out subject used for fine-tuning.Two strengths of this novel functional alignment procedure are in its simplicity and flexibility. Using a simple linear mapping for alignment can provide robust, generalizeable performance in low-sample, high-noise settings because simple mappings are less likely to overfit to noise. Also, unlike typical functional alignment approaches that require subjects to process a shared set of images Haxby et al. (2011) , our approach has the flexibility to work even when subjects are viewing entirely unique images in the training data. This is critical for the Natural Scenes Dataset, where 90% of the seen images are unique to the subject and the 10% that were seen across subjects are relegated to the test set. Further, this approach holds advantages for subsequent data collection of a new subject, where such data collection does not need to be restricted to showing a predefined set of images.Aside from linear mapping to a shared-subject space, mapping to OpenCLIP ViT-bigG/14 rather than CLIP ViT-L/14, and adding a low-level MLP submodule, the MindEye2 model is trained identically to MindEye1. Flattened spatial patterns of brain activity are first linearly mapped to the shared-subject space using an output dimensionality of 4096. Then, these latents are fed through an MLP backbone with 4 residual blocks, followed by a linear mapping that goes from 4096-dim to 256 × 1664 dimensionality of OpenCLIP ViT-bigG/14 image token embeddings. These backbone embeddings are then simultaneously fed through a diffusion prior Ramesh et al. (2022) and two MLP projectors (retrieval and low-level submodules).MindEye2 has three losses that are summed, stemming from the diffusion prior, retrieval submodule, and low-level submodule. The end-to-end loss, with α 1 = .033 and α 2 = .016, is defined as:L = L prior + α 1 • L BiMixCo|SoftCLIP + α 2 • L lowlevel (1) 2.2.1 DIFFUSION PRIORUsing a diffusion prior to align outputs from a contrastive learning model was inspired by DALL-E 2 Ramesh et al. (2022) , where a "diffusion prior" maps CLIP text embeddings to CLIP image space before using an unCLIP decoder to reconstruct images. Here we trained our own diffusion prior from scratch to map fMRI latents to the OpenCLIP ViT-bigG/14 image space, which was kept frozen as done with locked-image text tuning (LiT) Zhai et al. (2022) . We used the same prior loss as Ramesh et al. (2022) , implemented with the same code as MindEye1 which used modified code from the DALLE2-pytorch repository.MindEye1 observed a tradeoff if using contrastive loss and MSE loss on the outputs of the diffusion prior directly, such that the model could not effectively learn a single embedding to satisfy both objectives. Instead, applying MSE loss on the diffusion prior and applying contrastive loss on the outputs from an MLP projector attached to the MLP backbone effectively mitigated this tradeoff because the objectives no longer shared identical embeddings. We adopted the same approach here, with the retrieval submodule contrastively trained to maximize cosine similarity for positive pairs while minimizing similarity for negative pairs. We used the same BiMixCo and SoftCLIP losses used in MindEye1 Scotti et al. (2023) , which involved the first third of training iterations using bidirectional MixCo data augmentation Kim et al. (2020) and the last two-thirds of training iterations using soft labels (generated from the dot product of CLIP image embeddings in a batch with themselves) without data augmentation.MindEye1 used an independent low-level pipeline to map voxels to the latent space of Stable Diffusion's variational autoencoder (VAE) such that blurry reconstructions were returned that lacked semantic information but performed well on low-level metrics. Here, we reimplement this pipeline as a submodule, similar to the retrieval submodule, such that it need not be trained independently.The MLP projector feeds to a CNN upsampler that upsamples to the (64, 64, 4) dimensionality of SD VAE latents with L1 loss as well as an additional MLP to the embeddings of a teacher linear segmentation model VICRegL Bardes et al. (2022) ConvNext-XXL (α = 0.75) for an auxilliary SoftCLIP loss (soft labels from VICRegL model).EQUATIONTo predict image captions from brain activity we convert the diffusion prior's predicted ViT-bigG/14 embeddings to CLIP ViT/L-14 space and feed those outputs through a frozen pretrained Genera-tiveImage2Text (GIT) model Wang et al. (2022) . The use of GIT to caption images from brain activity in the Natural Scenes Dataset was previously shown to be viable by Ferrante et al. (2023b) .We independently trained a linear model to convert from OpenCLIP ViT-bigG/14 embeddings to CLIP ViT-L/14 embeddings (see Appendix A.9), which was necessary because there was no existing GIT model that accepted OpenCLIP ViT-bigG/14 embeddings as inputs. Image caption prediction from brain activity lends further flexibility to such decoding approaches and can help refine image reconstructions to match desired semantic content.2.4 FINE-TUNING STABLE DIFFUSION XL FOR UNCLIP CLIP Radford et al. (2021) is an example of a multimodal contrastive model that maps images and text captions to a shared embedding space. unCLIP (or image variations) models go from this shared embedding space back to pixel space, and have been used for the creative application of returning variations of a given reference image. As such, previous unCLIP models prioritized replication of high-level semantics over low-level structures. These models can be trained by fine-tuning a base image generation model to accept CLIP image embeddings instead of, or in addition to, text embeddings. Outputs are diffused from pure noise just like the base model, unlike image-to-image models Meng et al. (2022) that start the diffusion process from a reference image mixed with noise.Contrary to previous unCLIP models, our goal was to train a model that returns images as close as possible to the reference image across both low-level structure and high-level semantics. This is because our use-case was to exactly return the original image given its CLIP image embedding predicted from the brain.The base Stable Diffusion XL (SDXL) Podell et al. (2023) model uses text conditionings from both OpenCLIP ViT-bigG/14 and CLIP ViT-L/14. They condition cross-attention layers on the penultimate text encoder outputs and additionally condition on pooled text embeddings from OpenCLIP ViT-bigG/14 by adding it to the timestep embedding. Here, we fine-tuned the cross-attention layers using the OpenCLIP ViT-bigG/14 image embeddings corresponding to all 256 patch tokens and we dropped the additional conditioning on pooled text embeddings. We opted to only condition on image embeddings because we observed that incorporating any text conditioning worsened the fidelity of the unCLIP reconstructions.We evaluate the fidelity of our SDXL unCLIP model to reconstruct images from ground truth OpenCLIP ViT-bigG/14 image embeddings in Appendix A.8, showing that reconstructions are nearly identical to the original images. We fine-tuned SDXL on one 8xA100 80GB GPU node using an internal dataset for 110, 000 optimization steps at a resolution of 256 × 256 pixels and a batch size of 8 with offset-noise The pipeline for reconstruction inference is depicted in Figure 2 . First, the diffusion prior's predicted OpenCLIP ViT-bigG/14 image latents are fed through our SDXL unCLIP model to output a pixel image. We observed that these reconstructions were often distorted ("unrefined") due to an imperfect mapping to bigG space (see Figure ?? ). This may be explained by the increased versatility allowed from mapping to the larger dimensionality OpenCLIP bigG latent space. To increase image realism, we feed the unrefined reconstructions from SDXL unCLIP through base SDXL via image-to-image Meng et al. ( 2022) with text conditioning guidance from MindEye2's predicted image captions (section 2.3). We skip the first 50% of denoising diffusion timesteps, starting the process from the noised image encoding of the unrefined reconstruction. We simply take the first samples output from these stochastic models without any special 2nd-order selection. Refinement using base SDXL subjectively improves the quality of image outputs without strongly affecting low or high-level image metrics.The final "refined" reconstructions come from combining the outputs from base SDXL with the pixel images output from the low-level submodule via simple weighted averaging (4:1 ratio). This weighted averaging step increases performance on low-level image metrics while minimally affecting reconstructions' subjective appearance.For retrieval inference, only the retrieval submodule's outputs are necessary. Nearest neighbor retrieval can be performed via cosine similarity between the submodule's OpenCLIP ViT-bigG/14 embeddings and all the ViT-bigG/14 embeddings corresponding to the images in the desired image pool. 
results: 
 We used the Natural Scenes Dataset (NSD) Allen et al. (2022) , a public fMRI dataset containing the brain responses of human participants viewing rich naturalistic stimuli from COCO Lin et al. (2014) . The dataset spans 8 subjects who were each scanned for 30-40 hours (30-40 separate scanning sessions), where each sesssion consisted of viewing 750 images for 3 seconds each. Images were seen 3 times each across the sessions and were unique to each subject, except for a select 1,000 images which were seen by all the subjects. We follow the standardized approach to train/test splits used by other NSD reconstruction papers Takagi and Nishimoto (2022) ; Ozcelik and VanRullen (2023) ; Gu et al. (2023) which is to use the shared images seen by all the subjects as the test set. We follow the standard of evaluating model performance across low-and high-level image metrics averaged across the 4 subjects who completed all 40 scanning sessions. We averaged across same-image repetitions for the test set (1,000 test samples) but not the training set (30,000 training samples). For more information on NSD and data preprocessing see Appendix A.1.Critically, models trained on a subset of data were selected in chronological order. That is, models trained from only 1 hour's worth of data come from using the subject's first scanning session of 750 image presentations. This means our model must be able to generalize to test data collected from scanning sessions entirely held-out during training.First, we report performance of MindEye2 when training on the full NSD dataset. We quantitatively compare reconstructions across fMRI-to-image models in Table 1 , demonstrating state-of-the-art MindEye2 performance across nearly all metrics. We compare to both the previous MindEye1 results as well as other fMRI-to-image approaches that were open-sourced such that we could replicate their pipelines using the recently updated NSD (which includes an additional 3 scanning sessions for every subject).MindEye2 refined reconstructions using the full NSD dataset performed SOTA across nearly all metrics, confirming that our changes to shared-subject modeling, model architecture, and training procedure benefitted reconstruction and retrieval performance (explored more in section 3.5). Interestingly, we observed that high-level metrics for the unrefined MindEye2 reconstructions outperformed the refined reconstructions across several metrics despite looking visibly distorted. This suggests that the standard evaluation metrics used across fMRI-to-image papers should be further scrutinized as they may not accurately reflect subjective interpretations of reconstruction quality.We also report performance for MindEye2 fine-tuned with only 1 hour of data in the same Table 1 .We qualitatively compare reconstructions side-by-side with models trained on only 1 hour's worth of data in Figure 4 , depicting improvements in reconstruction quality for MindEye2. We report more evaluations in the Appendix: see A.3 for MindEye2 results without pretraining, A.6 for evaluations with varying amounts of training data across all models, A.7 for single-subject evaluations, and A.12 for MindEye2 evaluations with varying selection of pretraining subjects.The overarching goal of the present work is to showcase high-quality reconstructions of seen images from a single visit to an MRI facility. Figure 5 shows reconstruction performance across MindEye2 models trained on varying amounts of data from subject 1. There is a steady improvement across both pretrained and non-pretrained models as more data is used to train the model. "Non-pretrained" refers to single-subject models trained from scratch. The pretrained and non-pretrained results became Method Low-Level High-Level Retrieval Results from all previous work were recalculated using their respective public codebases using the full 40 sessions of NSD data, which was not released until the recent completion of the 2023 Algonauts challenge. Image retrieval refers to the percent of the time the correct image was retrieved out of 300 candidates, given the associated brain sample (chance=0.3%); vice-versa for brain retrieval. PixCorr=pixelwise correlation between ground truth and reconstructions; SSIM=structural similarity index metric Wang et al. (2004) ; EfficientNet-B1 ("Eff") Tan and Le (2020) and SwAV-ResNet50 ("SwAV") Caron et al. (2021) refer to average correlation distance; all other metrics refer to two-way identification (chance = 50%). Two-way identification refers to percent correct across comparisons gauging if the original image embedding is more similar to its paired brain embedding or a randomly selected brain embedding (see Appendix A.11). Missing values are from metrics being non-applicable. Bold indicates best performance, underline second-best performance.PixCorr ↑ SSIM ↑ Alex(2) ↑ Alex(5) ↑ Incep ↑ CLIP ↑ Eff ↓ SwAV ↓ Image ↑ Brain ↑ MindEye2 0.increasingly more similar as more data was added. The 1-hour setting offers a good balance between scan duration and reconstruction performance, with notable improvements from pretraining. The non-pretrained models trained with 10 or 30 minutes of data suffered significant instability. These models may have experienced mode collapse where outputs were similarly nonsensical regardless of input. Such reconstructions coincidentally performed well on SSIM, indicating SSIM may not be a fully representative metric. (2022) . MindEye2 captioning performance outperformed previous models across all metrics except one, suggesting high-quality image captions from brain activity.Image retrieval metrics help quantify the level of fine-grained image information contained in the fMRI embeddings. There are many images in the test set that contain similar semantic content (e.g., 14 images of zebras), so if the model can identify the exact image corresponding to a given brain sample, that demonstrates such fMRI embeddings contain fine-grained image content. MindEye2 improves upon MindEye1's retrieval evaluations by reaching near-ceiling performance on the retrieval benchmarks used in previous papers Lin et al. (2022) ; Scotti et al. (2023) (Table 1 ). Further, retrieval performance remained competitive when MindEye2 was trained with only 1 hour of data. Computing the retrieval metrics in Table 1 involved the steps described in Appendix A.10.To measure whether a reconstruction is faithful to the original brain activity that evoked it, we examine whether it accurately predicts that brain activity when input to a encoding model pretrained to predict brain activity from images Gaziv et al. (2022) . Encoding models provide a more comprehensive analysis of the proximity between images and brain activity Naselaris et al. (2011) , providing a unique measure of reconstruction quality that is perhaps more informative than the image metrics traditionally used for assessment. This alignment is measured independently of the stimulus image, allowing it to be used to assess reconstruction quality when the ground-truth image is unknown, making it extendable to new data in a variety of domains including covert visual content such as mental images. Given that human judgment is grounded in human brain activity, it could also be the case that brain correlation metrics provide increased alignment with the judgments of human observers. Here we explain where MindEye2 improvements over MindEye1 come from through ablations. MindEye2 outperforms MindEye1 even without pretraining on other subjects (see Appendix A.3), suggesting improvements in model architecture and training procedure. Our ablation results compare models trained from scratch in reduced capacity (1024-dim shared-subject latent space), skipping base SDXL refinement, using 10 sessions of data solely from subject 1.Two core differences between MindEye2 and MindEye1 are (1) we used a linear layer, rather than an MLP with dropout, for the initial mapping of voxels to the dimensionality of the residual MLP backbone, and (2) we map to OpenCLIP bigG image latents rather than CLIP L latents. Our ablations in Appendix A.4 show that these changes improve performance across all metrics, suggesting that a linear layer with L2 regularization is a more effective means of initially mapping voxels into model space, and that bigG is the richer, more effective CLIP space to map fMRI activity into.Ablations in Appendix A.5 show evaluations from models trained with various combinations of components. Retrieval metrics were worst when MindEye2 was trained with the diffusion prior and low-level submodules removed, and reconstruction metrics were worst when trained with the retrieval submodule and low-level submodule removed. More fine-grained comparison reveals that reconstruction metrics benefitted substantially from the retrieval submodule, although the low-level submodule did not considerably impact performance. 
related work: 
 It is common for fMRI analyses to align subjects' brains to a shared space for the purposes of increasing statistical power and/or assessing generality of scientific findings. Such alignment is difficult because structural and functional topography differs substantially across people Talairach and Tournoux (1990) ; Mazziotta et al. (2001) . There are many approaches to functional alignment but typically they involve subjects experiencing shared stimuli and then using responses to these stimuli to learn an alignment mapping Chen et al. (2015) ; Haxby et al. (2011) ; Huang et al. (2021) ; Nastase et al. ( 2019); Busch et al. (2021) . While it is useful to conduct such experiments to identify sources of shared signal across subjects, it is also limiting in that new subjects would need to be scanned using the same experimental protocol. Other functional alignment approaches avoid such limitations by using self-supervised learning to identify an initial generalizable embedding space with outputs suitable for downstream tasks Schneider et al. (2023) ; Chen et al. (2023a; b) . Closest to our alignment approach are models that adopt both shared-subject and subject-specific mappings in their model architecture Défossez et al. (2022) ; Benchetrit et al. (2023) ; Yang et al. (2023) . Ferrante et al. (2023a) previously showed across-subject image reconstruction via ridge regression by training a linear subject-specific decoding model and then separately mapping other subjects to this space via ridge regression. This is similar to our approach in that both involve ridge regression to a shared space, but is distinct in that their approach is capped by the performance of the initial singlesubject model from which other subjects are mapped into, is restricted to only linear fine-tuning, and was demonstrated only with a reduced training dataset of images seen by all subjects. MindEye2 is unique in its demonstration that a single neural network model can be pretrained across subjects experiencing unique stimuli and robustly fine-tuned to a new subject with few data points. 
conclusion: 
 We introduce MindEye2, a modeling approach that outputs reconstructions of seen images from fMRI activity with a similar quality to previous approaches using only a fraction of the training data. MindEye2 further achieves SOTA across reconstruction and retrieval metrics when supplied with the full training data. Our approach pretrains a model using data from multiple subjects, which is then fine-tuned on scarce data from a held-out subject. Patterns of fMRI activity are mapped to CLIP space and images are reconstructed with the help of our unCLIP model fine-tuned from Stable Diffusion XL. Our work shows the potential to apply deep learning models trained on large-scale neuroimaging datasets to new subjects with minimal data.The present work demonstrates that it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality reconstructions of their visual perception.Such image reconstructions from brain activity are expected to be systematically distorted due to factors including mental state, neurological conditions, etc. This could potentially enable novel clinical diagnosis and assessment approaches, including applications for improved locked-in (pseudocoma) patient communication Monti et al. (2010) and brain-computer interfaces if adapted to real-time analysis Wallace et al. (2022) or non-fMRI neuroimaging modalities. Some limitations of this work are that fMRI is extremely sensitive to movement and requires subjects to comply with the task: decoding is easily resisted by slightly moving one's head or thinking about unrelated information Tang et al. (2023) . MindEye2 has also only been shown to work on natural scenes such as those in COCO; additional data and/or specialized generative models would likely be required for other image distributions. As technology continues to improve, we note it is important that brain data be carefully protected and companies collecting such data be transparent with their use. 
 
 Appendix: A.1 ADDITIONAL DATASET INFORMATION fMRI responses correspond to normalized single-trial betas output from GLMSingle Prince et al. (2022) . We use preprocessed flattened fMRI voxels in 1.8-mm native volume space corresponding to the "nsdgeneral" brain region, defined by the NSD authors as the subset of voxels in posterior cortex most responsive to the visual stimuli presented (between 13,000 to 16,000 voxels per participant). MindEye2 was developed using a training and test set of subject 1's data, with other subjects' data untouched until final training of models. The fMRI data from both the training and test set was normalized using a voxel-wise Z-scoring procedure using the mean and standard deviation calculated using only the training set. Despite the shared1000 test trials being distributed across the scanning sessions for each subject, we chose to keep the test set consistent no matter the number of sessions being used for training. We also adjusted the number of training sessions after the normalization step, allowing us to keep the statistical properties of the shared1000 test set consistent between experiments with varying amounts of training data. This may inadvertently give a small normalization advantage to models trained with fewer training sessions, as the models are normalized with additional data not made available for training.We adopt the same caption metrics reported in the previous work. ROUGE Lin (2004) and METEOR Banerjee and Lavie (2005) capture aspects of text structure and composition. CLIP Radford et al. (2021) and SentenceTransformer ("all-MiniLM-L6-v2") Reimers and Gurevych ( 2020) are higherlevel metrics that provide insight into textual context, relationships, and semantics. All metrics except ROUGE were calculated using the same code as Ferrante et al. (2023b) . Table 4 shows how MindEye2 outperforms MindEye1 even without pretraining on other subjects. Models were trained using the full 40 sessions of training data from subject 1. This suggests that improvements from MindEye1 to MindEye2 are not explained solely from pretraining on other subjects, but that benefits also come from improved model architecture and training procedure.Ablations results in A.7 SINGLE-SUBJECT EVALUATIONS Tables 7 and 8 show more exhaustive evaluation metrics computed for every subject individually using 40-hours and 1-hour of fine-tuning data respectively. ). This difference is visually obvious as shown in Figure 9 . Note that while we observed distortions in our unrefined fMRI-to-image reconstructions using our unCLIP model fine-tuned from SDXL, such distortions were rare when using the ground truth CLIP embeddings.The ability for this unCLIP model to nearly perfectly return the original image also indicates that OpenCLIP ViT-bigG image embeddings effectively preserve the majority of the information inherent in the original pixel image, retaining both low-level structure and high-level semantic details.A.9 OPENCLIP BIGG TO CLIP L CONVERSION A.10 COCO RETRIEVALThe goal for brain retrieval is to identify the correct sample of brain activity that gave rise to the seen image out of a pool of brain samples. The seen image is converted to an OpenCLIP image embedding (or CLIP image embedding, depending on the contrastive space used in the paper) and cosine similarity is computed between its respective fMRI latent (e.g., from the retrieval submodule) as well as 299 other randomly selected fMRI latents in the test set. For each test sample, success is determined if the cosine similarity is greatest between the ground truth OpenCLIP/CLIP image embedding and its respective fMRI embedding (aka top-1 retrieval performance, chance=1/300). We specifically used 300 random samples because this was the approach used in previous work. We averaged retrieval performance across test samples and repeated the entire process 30 times to account for the variability in random sampling of batches. For image retrieval, the same procedure is used except image and brain samples are flipped such that the goal is to find the corresponding seen image in the image pool from the provided brain sample.MindEye1 scaled up image retrieval using a pool of billions of image candidates contained in the LAION-5B dataset Schuhmann et al. (2022) . This was possible because all LAION images were already converted to CLIP L embeddings and made available for nearest neighbor lookup via the CLIP Retrieval client Beaumont (2022) . We were not able to use this approach for MindEye2 because it would require converting all images to the 256 × 1664 dimensionality bigG latent space which was not feasible. That said, cursory investigation with the comparatively smaller MS-COCO dataset suggests that retrieval from a pool of images not containing the original image may not work as well with OpenCLIP bigG embeddings compared to the CLIP L embeddings used in MindEye1. To test retrieval, we used FAISS Douze et al. (2024) for k-nearest neighbor search through an index of flattened OpenCLIP bigG embeddings of 73,000 MS-COCO images. We found that for incorrect retrievals, the 3 nearest neighbors usually were dissimilar to the original image both semantically and in low-level appearance. This could be due to the latents corresponding to the 256 image patch tokens of OpenCLIP bigG representing a more complex combination of different levels of information. This could cause the OpenCLIP bigG embeddings to not be as effective for nearest neighbor retrieval in terms of subjective intepretation, as the last layer of CLIP ViT-L/14 is highly semantic but lacks in low-level image content. Although we demonstrated improved retrieval performance for MindEye2 compared to MindEye1 using random subsets of 300 images for MindEye2 compared to MindEye1 (Table 5 ), we suggest that mapping to the last layer of CLIP ViT-L/14 image space would work better if the intended application is to find semantically related nearest neighbors in a large image pool.A.11 RECONSTRUCTION EVALUATIONS: ADDITIONAL INFORMATION Two-way comparisons were performed for AlexNet Krizhevsky et al. (2012) (second and fifth layers), InceptionV3 Szegedy et al. (2016) (last pooling layer), and CLIP (final layer of ViT-L/14). We followed the same image preprocessing and the same two-way identification steps as Ozcelik and VanRullen (2023) and Scotti et al. (2023) . For two-way identification, for each model, we computed the Pearson correlation between embeddings for the ground truth image and the reconstructed image, as well as the correlation between the ground truth image and a different reconstruction elsewhere in the test set. If the correlation for the former was higher than the latter, this was marked as correct. For each test sample, performance was averaged across all possible pairwise comparisons using the other 999 reconstructions to ensure no bias from random sample selection. This yielded 1,000 averaged percent correct outputs, which we averaged across to obtain the metrics reported in Table 1 .A.12 PRETRAINING WITH LESS SUBJECTS To determine the relative impact of using additional subjects for pretraining, we separately fine-tuned a MindEye2 model for subject 1 (using 1 hour of their training data) that was pretrained only on subjects 2, 5, and 7 (these are the subjects who completed all 40 scanning sessions), as well as only on subject 5 (the subject whose single-subject model performed the best). Results in Table 10 show similar performance for these models compared to pretraining on the full set of available subjects, suggesting that the number of pretraining subjects does not seem to play a major role in subsequent fine-tuning performance.As discussed in Scotti et al. (2023) , UMAP dimensionality reduction McInnes et al. (2020) plots of disjointed CLIP fMRI embeddings next to aligned CLIP fMRI embeddings visualize how the diffusion prior effectively addresses the disjointed embedding spaces problem. Theoretically, multimodal contrastive learning will always produce disjointed embeddings because of the "modality gap" phenomenon whereby encoding modalities into a shared space restricts the effective embedding space to a narrow cone in geometric space Liang et al. (2022) . 



Reviews: review: title: title: Improved sample efficiency in visual perception reconstruction from fMRI through inter-subject alignment
main_review: main_review: This work aims to improve the sample efficiency in the reconstruction of visual perception from fMRI recordings. In particular, it proposes a pre-training heuristic for alignment of asynchronous fMRI responses from multiple individuals into a shared representational space, coupled with a mapping to CLIP space. Images are reconstructed with the help of an "unCLIP" model. After fine-tuning to a new subject with limited data, the procedure achieves good generalisation to held-out data from that subject across several metrics.

The proposed method is simple and the findings appear interesting. My main reservation is that some aspects of this work may be explained more clearly. The method introduced here has several advantages with respect to standard subject-alignment methods for neuroimaging, where "new subjects would need to be scanned using the same experimental protocol"; it is however worth remarking that those usually come with statistical guarantees, and that the proposed method is rather heuristic in comparison. Sections 2.2.1 to 2.2.3 are not easy to parse for readers without the required prior knowledge. 

> The retrieval submodule is contrastively trained and produces disjointed CLIP-fMRI embeddings that can be used to find the original (or nearest neighbor) image in a pool of images, but is not used to reconstruct a novel image

It would be interesting to have more discussion of pros and cons, and the respective importance, of retrieval vs. reconstruction tasks in the context of neuroimaging.

The results look interesting and indicate an improvement in sample efficiency based on the proposed subject-alignment procedure. This may be an interesting contribution to the workshop.

Typos: 

?? in Section 2.5
strength_weakness: strength_weakness: The use of asynchronous inter-subject response alignment seems interesting and relevant to the workshop topic; the method appears to have potential.
title: title: Awesome work!
main_review: main_review: This work proposes a new version (MindEye2) of MindEye1, a model that can reconstruct a person’s visual perception during an fMRI scan. The new model both outperforms MindEye1 more generally, and 
is able to generalize to a new subject after pre-training on other subjects, with only 1 hour of data from the new subject, as opposed to 40.

**Clarity** \
I think the paper is generally clearly written, and fairly easy to understand. The paper stands out on all other fronts, but this is the main point of improvement. Especially section 2, where the authors go over all of the submodules that make up their model. I believe that for a more general audience, it may be important to intuitively explain all of the steps in the model. The text comes across as rather jargon-y at times, and this can reduce the method’s interpretability for less familiar readers. Specifically, you use two CLIP models full names and it is hard for a reader to distinguish between the two (OpenCLIP ViT-bigG/14 and CLIP ViT-L/14) after reading about them many times. Moreover, much of this section reads as rather descriptive and detailed, whereas a more general reviewer is likely helped by a more intuitive explanation, rather than an exact explanation. You can refer to a more exact explanation in the Appendix if you don’t have enough space. A few examples are: P4 “We used the same BiMixCo and SoftCLIP losses used in MindEye1, which involved the first third of training iterations using bidirectional MixCo data augmentation and the last two-thirds of training iterations using soft labels…”. This sentence essentially assumes you have read Mindeye1, which is fine in the appendix, but I think the explanation can be more intuitive. Another example: P4 “The MLP projector feeds to a CNN upsampler that upsamples to be (64, 64, 3) dimensionality of SD VAE latents with L1 loss as well as an additional MLP to the embeddings of a teacher linear segmentation model VICRegL ConvNext-XXL for an auxilary SoftCLIP loss…”. This happens more often throughout the text, and apart from the run-on sentence structure sometimes, I believe the authors should focus more on why they did certain things and what the mechanistic advantage is. 

**Correctness** \
The analyses in the paper are correct as far as I can tell, and so are the conclusions from their experiments. It is also commendable that the authors have made the code available online already.

**Novelty** \
I think the paper is novel in that it clearly improves on a previous model with ablations to show that each of their extensions have a positive impact, and the fact that they were able to figure out pre-training in this type of setting such that their model only requires 1 hour of fMRI data, which is quite unique.

**Interest to the community** \
I believe this paper is of very direct interest to the community.

**Grammatical/spelling errors (these may be personal preference so feel free to ignore them)** \
- P4, “auxillary” -> auxilliary
- P5 “We simply take the first samples output from these stochastic models…” -> first few samples from these
- P6, “We also report performance…” -> We also report the performance
- P6, “… of data suffered significant instability” -> suffered instabilities/significant instabilities
- P8, “… whether it accurately predicts that brain activity when input to a encoding model …” -> when used as input to an encoding model
- P8, “… using protocol from …” -> using the protocol from
- P8, “Here we explain where MindEye2 improvements over MindEye1 come from through ablations.” -> In this section we explain, with ablations, which additions lead to MindEye2’s improvement over MindEye1
- P11, the link in the reference at the bottom of the page goes out of screen on the right
strength_weakness: strength_weakness: I think this is an important piece of work to highlight at the workshop, and although it is not directly looking at the alignment of biological and artificial representations, it does showcase a method with which we can interpret and extract representations from noisy, indirect, and whole-brain measurements of the human brain. We can potentially compare the representational alignment to artificial neural networks in the future.
title: title: Not the best aligned paper, but a pretty nice contribution nonetheless!
main_review: main_review: Clarity 5/5 
Correctness 5/5
Novelty 3/5
Interest/Relevance 2/5

The authors introduce MindEye2, a model that can reconstruct images from fMRI activity using only limited training data per subject. The model achieves state-of-the-art results on the Natural Scenes Dataset using full training data, and retains competitiveness with just 1 hour of data.
strength_weakness: strength_weakness: Yet, it is not too far removed from the ideas that form the central theme. The contribution is a new level of performance, the methods are thoroughly analyzed and the paper is very well written, and so I recommend acceptance.