BrainRAM: Cross-Modality Retrieval-Augmented Image Reconstruction from Human Brain Activity

Authors: dian  xie
peiang  zhao
jiarui  zhang
kangqi  wei
xiaobao  ni
jiong  xia
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 12:49:10

Abstract: Reconstructing visual stimuli from brain activities is crucial for deciphering the underlying mechanism of the human visual system. While recent studies have achieved notable results by leveraging deep generative models, challenges persist due to the lack of large-scale datasets and the inherent noise from non-invasive measurement methods. In this study, we draw inspiration from the mechanism of human memory and propose BrainRAM, a novel two-stage dual-guided framework for visual stimuli reconstruction. BrainRAM incorporates a Retrieval-Augmented Module (RAM) and diffusion prior to enhance the quality of reconstructed images from the brain. Specifically, in stage I, we transform fMRI voxels into the latent space of image and text embeddings via diffusion priors, obtaining preliminary estimates of the visual stimuli's semantics and structure. In stage II, based on previous estimates, we retrieve data from the LAION-2B-en dataset and employ the proposed RAM to refine them, yielding high-quality reconstruction results. Extensive experiments demonstrate that our BrainRAM outperforms current state-of-the-art methods both qualitatively and quantitatively, providing a new perspective for visual stimuli reconstruction.

Main paper: 
 introduction: 
 "What you see is what you get. " The cognitive function of humans is closely intertwined with the visual system [51] . People capture information about the external world through their eyes and obtain corresponding perceptions through prior knowledge and memory within their minds [4, 12] . Understanding the intricate visual process occurring in the human brain has long been a hot topic in cognitive neuroscience. Within this overarching topic, decoding visual stimuli from functional magnetic resonance imaging (fMRI) represents one of the most challenging issues that has garnered extensive attention [10, 12, 20, 29, 30, 36, 44] .Early attempts [18, 29, 32] demonstrated the feasibility of decoding visual images. However, the results were primitive and abstract due to the limited expressive ability of traditional regression methods. Over the last few years, with the prosperity of Artificial Intelligence Generated Content (AIGC), a series of methods [3, 9, 20, 43, 44] adopt generative models to decode visual activities. Shen et al. [44] and Ozcelik et al. [34] have employed Generative Adversarial Network (GAN) [16] to reconstruct stimulus. More recent works [8, 23, 28, 42, 47, 49, 55] leverage diffusion models [11, 19, 39, 56] to generate high-quality images corresponding to the original visual stimuli. However, fMRI is a transient response to human brain activity, which does not adequately reflect the role of past human knowledge and memory in the cognitive process. It also inherently contains some physiological noise unrelated to visual stimuli. As shown in Fig 2,  relying solely on fMRI for decoding will inevitably lead to semantic and structural inconsistency.The recent study by Scotti et al. [42] observed that images based on the retrieval of brain activities tend to be more realistic than the reconstructed ones. However, this observation has not been thoroughly explored and leveraged. Meanwhile, Retrieval-Augmented Generation (RAG) has achieved impressive success in generative models [5, 45, 46, 52, 54, 57] . It introduces an information retrieval process that enhances generated results by retrieving relevant objects from a large dataset, thereby improving accuracy and robustness. This parallels human cognitive and memory functions, where individuals with rich knowledge and experience tend to describe objects more accurately than those with limited experience. Moreover, the introduction of RAG can compensate for limited training samples with the support of massive data. Therefore, we believe that RAG is an effective means to improve the quality of reconstruction.In this paper, we propose a novel framework for visual stimuli reconstruction called BrainRAM, which aims to produce more faithful reconstructions by imitating human memory mechanisms. In stage I, we project fMRI voxels into the representation space of images and text separately, transforming them into corresponding features using diffusion priors. These features serve as preliminary estimates of visual stimuli. To further enhance alignment and alleviate inconsistency, we introduce the Retrieval-Augmented Module (RAM) to select and integrate features between preliminary estimates and retrieved samples. In stage II, we retrieve samples from the LAION-2B-en dataset [41] based on prior output features. These features, along with the retrieved samples, collaboratively predict the image and text features of the visual stimuli in RAM. We illustrate the differences between BrainRAM and previous methods briefly in Figure 1 . Extensive experiments have demonstrated that our method outperforms state-of-the-art methods. Two examples in Figure 2 also showcase the superiority of BrainRAM.In summary, our main contributions are formulated as follows:  
related works 2.1 visual stimuli reconstruction: 
 Neural decoding from visual stimuli has captivated researchers many years. Several early studies have demonstrated the possibility of decoding visual information from brain activity patterns measured using functional magnetic resonance imaging (fMRI). Naselaris et al. [31] showed that natural images can be reconstructed from fMRI data using a linear decoder. Kay et al. [21] used a support vector machine to decode the orientation of gratings based on activity patterns in the early visual cortex. Nishimoto et al. [32] used natural scences as visual stimuli. However, traditional visual stimuli reconstruction methods [10, 14, 21, [30] [31] [32] only rely on linear regression models to fit fMRI data with paired images. The results were abstract and blurry due to the limited expressive capabilities of regression models. In recent years, a series of studies [3, 20, 26, 43, 44] have delved into the potential of harnessing deep learning algorithms to unravel visual information within the human brain. Beliy et al. and Gaziv et al. [3, 15] employed encoder-decoder structure to align image representations with fMRI voxels. Shen et al. [44] fed the predicted features into GAN to reconstruct original visual stimuli. Ozcelik et al. [34] used a regression model to extract fMRI features and fine-tuned a pre-trained conditional big-GAN. The introduction of diffusion model [11, 19] and Contrastive Language-Image Pre-training (CLIP) [37] provides powerful tools for this task. Takagi et al. [49] use regression model to mapping fMRI to image and text features of Stable Diffusion [39] . Chen et al. [8] use Masked Aucoencoder (MAE) to extract fMRI feature and fine-tuned a LDM to reconstruct stimulus. Lu et al. [28] controlled the semantic and structure of the results by mapping fMRI voxels to CLIP features and VQ-VAE latent features. Scotti et al. [42] leverage diffusion prior from DALL-E 2 [38] to transform fMRI to image representations. These methods have achieved impressive results in visual stimuli reconstruction. But the above methods have limitations in image fidelity and semantic accuracy, resulting in unreliable reconstruction results.Recently, the use of external memory to enhance large models has attracted attention. Typically, adding more training data can improve model's performance. But it is inefficient to collecting data and retraining a model with a large number of parameters. In natural language processing (NLP), Wu et al. [54] proposed a memory transformer to store information from past inputs. Querying in the memory component could improve performance many downstream tasks. Several studies [46, 52] also introduced retrieval-augmented generation in computer vision. They use the retrieved data to generate high fidelity and faithful images. RetrieveGAN [52] used a differentiable retrieval module to generate images based on scene descriptions. IC-GAN [7] trained a GAN using the neighborhood of the training image and generate samples by adjusting individual instances from the training data. KNN-diffusion [45] trained diffusion based models using large-scale retrieval methods without any textual data. The model is conditioned on text or image feature extracted by CLIP, and use k retrieved embeddings from a large dataset to generate images. RDM [5] combines small diffusion or autoregressive models with large external image datasets to form a semi parametric model. The model retrieves a set of nearest neighbors for each training image from the dataset and adjusts the diffusion model based on their CLIP embeddings. These studies have proved that retrieving samples from large datasets can enhance the generation quality. But prior works of neural decoding only focus on aligning fMRI with image or text features directly. As far as we know, BrainRAM is the first method to integrate retrieval-augmentation into visual stimuli reconstruction, which utilize retrieved samples to achieve higher fidelity and semantic accuracy in reconstructions. 
method 3.1 overview: 
 In this subsection, we give a detailed analysis of the fMRI data and visual-linguistic representations that inspire our method's design.As a non-invasive measurement, fMRI measures the blood-oxygenlevel-dependent (BOLD) level in every brain voxels, indirectly reflecting the intensity of activity in brain regions. The neural activity in the brain often exhibits certain clustering characteristics in space, where similar neurons participate in processing similar information. Therefore, when a brain region is activated, it usually leads to changes in the blood oxygen levels of that region and its surrounding areas, resulting in signals with similar amplitudes between adjacent voxels in fMRI. Therefore, the spatial representation of fMRI is sparse. Besides, it is inherently noisy because fMRI not only records brain activity induced by visual stimuli, but also signals from other physiological and cognitive processes. Therefore, we need to compress fMRI voxels into a denser representation space and utilize RAG on general visual-linguistic data to overcome the inherent noise of fMRI.Driven by the analysis above, we propose our two-stage dualguided framework, which illustrated in Fig 3 . In stage I, the fMRI signals are projected and transformed into image/text representations with diffusion prior. The predicted embedding will serve as a query to find k-nearest-neighbour of image-text pairs in LAION-2B-en dataset in stage II. Then, the retrieved data and the predicted embedding will be fed into the retrieval-augmentation module to generate a further refined embedding. Finally, the refined image/text embeddings will jointly guide the image generation in Versatile Diffusion dual-guided pipeline.As mentioned in 3.1, fMRI signals have sparse representations just like images. Therefore, it is necessary to project it onto a tighter, shared feature space through contrastive learning. The success of Contrastive Language-Image Pre-Training (CLIP) [37] proves that images and text could essentially share same representations. However, the final representation space of CLIP itself is too small (1 × 768), making it difficult to recover the original details from shared features. Aligning tightly on a larger representation space will be more feasible for our goal. Meanwhile, previous studies have proved the effectiveness of diffusion prior to transfer features and reconstruct between modalities [13, 38, 42] . As a result, we adopt diffusion prior as backbone to maintain the consistency between fMRI and image/text representations.Consider the triplet dataset Ω = {𝑆 𝑖 , 𝑉 𝑖 ,𝑇 𝑖 } 𝑛 𝑖=1 of { fMRI, image, captions }. The fMRI data 𝑆 𝑖 ∈ R 1×𝑁 is pre-processed fMRI beta values. The beta value is obtained by weighted average on the blood oxygen level response throughout the entire experiment, avoiding processing more sparse and lengthy temporal data in subsequent analysis. The fMRI data was flatten as a 1D series based on the region of interest (ROI) on visual cortex, 𝑁 denotes the number of voxels in the ROI. 𝑉 𝑖 ∈ R 𝐻 ×𝑊 ×3 denotes visual stimuli presented to the subject, and 𝑇 𝑖 is corresponding COCO captions of the visual stimuli.We adopt the pre-trained image encoder E 𝑖𝑚𝑔 (•) and text encoder E 𝑡𝑥𝑡 (•) of CLIP to extract image and text features. However, instead of direct using the final outputs of the encoder (1 × 768), we use the last hidden layers as the mapping target of fMRI (257 × 768 for image feature, and 77 × 768 for text feature), avoiding overly tight spatial constraints on expression ability.At first, the fMRI data will pass a MLP projection layer and subsequent mamba blocks Φ 𝑝𝑟𝑜 𝑗 (•) to projected into a intermediate space (e.g. 257 × 768 or 77 × 768). Then the diffusion prior will generate corresponding target feature (E 𝑖𝑚𝑔 (𝑉 𝑖 ) or E 𝑡𝑥𝑡 (𝑇 𝑖 )) from Gaussian noise conditioned on Φ 𝑝𝑟𝑜 𝑗 (𝑆 𝑖 ). In forward diffusion process, the sample at each time point is defined as: where x 0 is target GT feature, 𝛼 𝑡 represents the noise variance schedule and 𝑡 ∈ {0, 1, . . . ,𝑇 } [39] . The inverse diffusion process will apply a attention-based U-Net 𝑓 𝜃 (x 𝑡 , 𝑡, Φ 𝑝𝑟𝑜 𝑗 (𝑆 𝑖 )) to gradually recover the original input x 0 from noisy x 𝑇 . Specifically, the conditional information Φ 𝑝𝑟𝑜 𝑗 (𝑆 𝑖 ) is incorporated in the cross-attention module of 𝑓 𝜃 :EQUATIONEQUATIONwhere 𝜑 𝑖 (x 𝑡 ) indicates intermediate features of U-Net [40] andEQUATION𝑉 denote trainable attention projection matrices.We use InfoNCE loss [33] to align the representations of fMRI to target features:EQUATIONwhere 𝑝 denotes the output of diffusion prior, 𝑐 denotes aligned target, 𝜏 is a temperature hyperparameter. It has 1 positive and 𝐵 -1 negative samples in batch size of 𝐵. We also incorporate Mean Square Error (MSE) loss to improve retrieval performance in subsequent stage, with a hyperparameter 𝛾 to balance these two losses in Eq 6.Due to the limitation of limited number of training samples of brain-image-text triplet compared to the contrastive learning in CLIP, we adopt the MixCo [22] augmentation strategy to improve the robustness. Given two fMRI data 𝑆 𝑖 and 𝑆 𝑘 , the convex mixture are given by:EQUATIONwhere 𝑆 𝑘 is a arbitrary sample shared the same batch with 𝑆 𝑖 . Let 𝑝 * 𝑖 be the diffusion prior output from 𝑆 𝑚𝑖𝑥 𝑖,𝑘 , and 𝑐 is the corresponding target feature, the MixCo loss is given by:EQUATIONWith introduced hyperparameter 𝛽 to control the intensity of L MixCo , the total loss of stage I can formulated as:EQUATIONIn this stage, we proposed Retrieval-Augmentation Module (RAM), which illustrated in Fig. 3 (c), to incorporate massive real-world data into the generated results. Unlike other retrieval-augmented image generative models, which require to train the entire diffusion model, we utilize RAM in embedding level that can directly applied to the existing model. To enable the module adaptively select feature from the retrieved data, we perform an automatic selection contain two operations: Divide and Fuse [25] . Note that the diffusion prior and fMRI projection block are frozen in this stage. Divide. Let 𝑝 ∈ R 𝑇 ×𝐷 be the output of diffusion prior, where 𝑇 is the number of tokens (257 for image and 77 for text) and 𝐷 is the dimension of tokens (typically 768). The 𝑝 will serve as a query in LAION-2B-en, and retrieve 𝐾 embeddings {𝑐 𝑖 } 𝐾 𝑖=1 based on the similarity between 𝑐 𝑖 and 𝑝. Every 𝑐 𝑖 is performed crossattention with 𝑝 separately, with query from 𝑝, key and value from 𝑐 𝑖 . Note that 𝑝 also performed self-attention too. These operations will generate 𝑘 + 1 attention maps, denoted as:EQUATIONwhere 𝑢 𝑝 , 𝑢 𝑐 𝑖 ∈ R 𝑇 ×𝐷 , U ∈ R (𝐾+1) ×𝑇 ×𝐷 . The weights are shared between cross-attention and self-attention. Fuse. To integrate information from 𝐾 + 1 features, we averaged them firstly:EQUATIONThen, we apply different tiny MLP F 𝑖 (•) to extract sample-specific information from the compact ū separately:EQUATION)where Z ∈ R (𝐾+1) ×𝑇 .To acquire the guidance for adaptive selection from different samples, we apply softmax to the first dimension of Z: Z = softmax(Z). (10) where the first dimension of Z are weights for feature from different tokens. Let 𝑞 ∈ R 𝑇 ×𝐷 be the final output of RAM, every element in 𝑞 is given by:EQUATIONwhich represents weighted average for every token on different 𝑢.To ensure the reconstruction results won't get worse with RAM, we consider the optimization objective of 𝑞 as residual between prior output 𝑝 and ground truth embedding 𝑐, which means 𝑐 = 𝑝 +𝑞 in the most ideal situation. Besides, the initial weight of the last dense linear layer in Attention is zero-initialized, which means the initial output of RAM is zero, to protect 𝑝 from the damage of random initialization. Considering that the residual on embedding is relatively small, we use the loss with smaller 𝛾 value compared to stage I:EQUATION) 
experiments 4.1 dataset: 
 We employ the Natural Scenes Dataset (NSD) [1] for this study. NSD is the largest known dataset that using 7T scanner to acquire visual-evoked fMRI response from 8 subjects. Each subject was presented about 10,000 different natural images multiple times during 30 to 40 sessions, with whole-brain gradient-echo EPI at 1.8mm isotropic resolution and 1.6s TR for scanning. The natural images in the paradigm are sourced from Common Objects in Context dataset (COCO) [27] , which have 5 or 6 captions correspondingly for each image. In this study, we only focused on 4 subjects (subj01, subj02, subj05, and subj07) out of 8, since they have completed all scanning sessions. Same as other experiment, we use nsdgeneral mask to select ROI, which provide a general ROI that covering voxels responsive to the NSD experiment in the posterior aspect of cortex.We follow the same train-test splits in NSD, with 8,859 image stimuli for training and 982 for testing for each subject. Also, we averaged response for image with multiple fMRI trials, and the target text feature are averaged on multiple captions.In stage I, we training image diffusion prior and text diffusion prior separately for 180 epochs with batch size of 32. We use AdamW optimizer with default parameters and one-cycle learning rate 2.5 × 10 -4 . The loss hyperparameter 𝛾 and 𝛽 are set as 0.8, 0.3 respectively. In stage II, we use clip-retrieval tool [2] to retrieve data in LAION-2B-en. Due to the randomness of diffusion prior output, we reconstruct 4 priors per image stimuli, and each prior was retrieved for another 4 samples. We apply zero-initialization to the last dense linear layer of Attention in RAM. Training for 120 epochs with batch size of 256, and default AdamW optimizer with fixed learning rate 2.5 × 10 -4 . The loss hyperparameter 𝛾 is 0.2. Due to inherent noise in current measurement method, it is impossible to reconstruct all details of the image from fMRI with complete accuracy. As a result, we mainly focus on semantic consistency in this task. But for the sake of fairness, we still use some low-level metrics in accordance with previous studies [35, 42] . Specifically, for high-level metrics, we use two-way comparison of the last hidden layer of CLIP image encoder E 𝑖𝑚𝑔 (•), denoted as CLIP [37] . EffNet-B and SwAV indicate the average correlation distance gathered from EfficientNet-B1 [50] and SwAV-ResNet50 [6] , correspondingly. Inception represents the two-way comparison of the last pooling layer of InceptionV3 [48] . In low-level metrics, the Structural Similarity Index (SSIM) [53] measures the similarity between two images. Pix-Corr indicates pixel-level correlation between the reconstructed and ground-truth images. AlexNet(2) and AlexNet(5) represent twoway comparisons of the second and fifth layers of AlexNet [24] , respectively. 
reconstruction results: 
 In this section, we compared our BrainRAM with other five stateof-the-art methods: Lin et al. [26] , Takagi et al. [49] , Gu et al. [17] , Brain-diffuser [35] , and MindEye [42] . Due to some methods' resultswere based on experiments conducted on subj01, for the sake of fairness, if not specified, all our experiments were conducted on subj01. The quantitative performance of each model are presented in Tab 1. Our method outperforms in all high-level metrics and part of low-level metrics. Especially in CLIP similarity, BrainRAM earn 4.1%, 20.3%, and 32.8% performance gains over previous stateof-the-art MindEye [42] , Takagi et al. [49] , and Brain-diffuser [35] , which proves that our method has achieved significant success in high-precision semantic information extraction.In addition to quantitative metrics, several visual cases are also illustrated in Fig 4 that are consistent with numerical findings. For an example, in the first row of Fig 4 , other methods did not accurately capture semantic information, which misidentified the elephant in GT as a dog (Takagi et al. [49] ) or zebra (MindEye [42] ). Alternatively, Brain-diffuser [35] can't accurately reconstruct the structure of the elephant. Besides, for the Red double-decker bus, other methods can only reconstruct it as a public transportation vehicle. In contrast, BrainRAM precisely reconstructs the red double-decker bus with its unique structure and semantic information. These results indicate that BrainRAM could produce results that are consistent with the visual stimuli in terms of semantics and structure. In this subsection, we will demonstrate the performance of test set retrieval and LAION retrieval separately. Test set retrieval. Test set retrieval contain fMRI retrieval and image retrieval. Image retrieval means retrieving the image embedding with the highest CLIP similarity based on predicted output embedding from fMRI in the test set. The image retrieval is considered correct if a paired image embedding is retrieved, and vice-versa for fMRI retrieval. Follow the same settings of previous studies [26, 42] , we randomly select 300 image-fMRI pairs in the test set of NSD, and computing image embeddings and corresponding output embeddings from fMRI voxels. The quantitative results are presented in Tab 2. Compared to previous studies, our retrieval accuracy improved significantly for both image and brain retrieval ways, with 3.6% and 5.7% superiority over MindEye [42] .LAION retrieval. LAION retrieval refers to image retrieval on LAION-2B-en [41] , which can reflect the model's potential to capture visual information from fMRI from another perspective. For an output embedding, we could query K-nearest neighbours to retrieve images most relevant to the model's prediction. Given the emphasis on semantic relevance in embedding-based retrieval, we only focus high-level image evaluation metrics. We calculate 16 output embeddings per fMRI data, and retrieve 16 images with highest CLIP similarity in correspondence to the outputs. Experiment results in Tab 5 shows that our BrainRAM outperforms MindEye [42] In this subsection, we further conduct comprehensive experiments to verify the validity of each component in the model. All quantitative performance can be found in Tab 3.Effect of model architecture. Obviously, as shown in the Tab 3, the introduction of RAM significantly increases the reconstruction performance of the model. It can also be observed that even using predicted features directly to reconstruct visual stimuli in stage I can surpass MindEye in most of high-level metrics. We also conduct experiments to compare our proposed RAM with other feature fusion methods: Average, 1x1 Convolution, and Cross-Attention. Average means directly average on all features. 1x1 Convolution represents use a 1x1 convolution kernel to merge all features. Cross-Attention means the prior output serves as query, and other retrieved embeddings serve as keys and values. Presented in Tab 4, RAM surpasses all other feature fusion methods in all metrics. RAM leverages Divide and Fuse operations to select features from retrieved embeddings and prior output, thus enhance reconstruction using retrieved information. It also ensures that the performance of the model will not decrease after adding RAM.As mentioned in Sec 3. Intuitively, text contains richer semantic information, thus could enhance the semantic fidelity of reconstructed image. Results from the third and fourth row in Tab 3 also verified that idea. Our BrainRAM maintain higher semantic superiority than solely relied on image guidance. It is also impractical to reconstruct image with text guidance only. Since text is on semantic-level, which may not be able to provide details in the image. Furthermore, due to the ambiguity of language representation, RAM has limited effectiveness in this situation.Effect of retrieval dataset. When using RAM, we default to retrieve sample on LAION-2B-en. To examine the impact of retrieval datasets on model performance, we conduct experiments with different retrieval dataset. Specifically, we re-trained RAM on LAION-400M, and results could be found in the second to the last row of Tab 3. We observed that when retrieving on LAION-400M, the model's performance also increased, but not significant like retrieving on LAION-2B-en. On the one hand, smaller data size reduces data diversity in the retrieved samples. If retrieving on a larger dataset, it may even be possible to retrieve ground-truth stimuli based on prior output (like in Fig 5) , which would be very helpful for reconstruction. On the other hand, captions from LAION-2B-en are only in English. LAION-400M contains multi-language captions that may cause confusion for CLIP text encoders, since CLIP are trained on image-text pairs with English captions. 
conclusion: 
 In this study, we propose a two-stage dual-guided visual stimuli reconstruction framework called BrainRAM. In stage I, we first project the fMRI signal and then convert it into CLIP text image features through diffusion prior. In stage II, we retrieve samples from LAION-2B-en to refine prior outputs, and use Versatile Diffusion to reconstruct visual stimuli. Experiments have shown that BrainRAM outperforms current state-of-the-art methods both qualitatively and quantitatively in reconstruction and retrieval. In addition, we provide a completely new perspective for this task: introducing retrieval-augmented generation into visual stimuli reconstruction can help improve image fidelity and semantic accuracy. For future work, we believe that cross-subject decoding would be a promising direction. By utilizing the responses of different subjects in multiple datasets, we can further understand the visual mechanisms of entire humanity. 
 
 Appendix:  


Decision: The rebuttal letter well address the concerns from the reviewers. All reviewers tend to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a two-stage dual-guided framework for fMRI visual decoding and image reconstruction. This method uses fMRI as conditional embeddings, leveraging the prior knowledge of diffusion models to align fMRI with both images and text separately, and refines it through retrieval enhancement. Ultimately, the image is reconstructed using versatile diffusion.
strength_weakness: strength_weakness: 1. This method simulates the human memory mechanism, providing a new perspective for studies on the understanding and reconstruction processes of human visual cognition.

2. The method effectively utilizes diffusion priors and versatile diffusion, which are absent in previous similar works.

3. It is a novel work, and the experiments are solid.

4. The paper is well-written.
limitations: limitations: 1. In Sec. 3.2, the loss hyperparameter 'α' in equation (6) overlaps somewhat with the variable in the noise variance schedule in equation (1).

2. It is recommended to supplement the implementation details with the hardware platforms used in the experiments.

3. The motivation for using versatile diffusion instead of other models seems not clear enough.
paper_summary: paper_summary: This paper introduced a novel method to reconstruct images from brain activities represented by fMRI. The authors retrieve images from a large image dataset using fMRI embeddings aligned with images and texts respectively. The retrieved images will be combined with the fMRI embeddings as guidance for a versatile diffusion model. Superior performance was shown compared with the previous work.
strength_weakness: strength_weakness: The paper is solid as an ML paper with interesting ideas and solid experiments. The organization and writing are easy to follow.
limitations: limitations: 1. The high-level semantics metric only uses a 2-way test, which has a chance level of 50%. More distinguishable metrics like a 50-way or 100-way test should also be used as a supplement. 

2. The claim that the proposed RAM method considers the impact of human memory on visual perception should be careful and better elaborated. This claim may not be scientifically correct. I agree that human memory could have an impact on visual perception, but the proposed RAM method did not consider the "actual human memory", as the decoding is also based on transient fMRI signals. If I understand correctly, retrieving images from a large dataset using fMRI does not consider the human memory encoded in other areas of our brain. The proposed RAM method is more of an analogy of human memory rather than actually considering human memory.   

3. Using retrieved images in the reconstruction directly could be controversial because the retrieval dataset may contain groundtruth images.   According to my knowledge, Scotti et al. [41] used the retrieval task to show that the reconstruction is not just a classification.  

4. Some minor typos: 
- Line 278. Should it be "embedding will be fed into.."?
- Line 458. Should it be "require to train" ?

5. Could the author elaborate more on Line 503? Why the reconstruction would get worse with RAM?
paper_summary: paper_summary: The manuscript introduces BrainRAM, an innovative dual-stage framework for visual stimulus reconstruction from brain activity. Drawing inspiration from memory mechanisms, the study employs a Retrieval-Augmented Module (RAM) and diffusion priors to enhance the quality of reconstructed images. The methodology involves an initial mapping of fMRI voxels into a latent space for semantic and structural estimation, followed by data retrieval from the LAION-2B-en dataset and RAM-based refinement, achieving superior reconstruction quality.
The study demonstrates that BrainRAM outperforms current state-of-the-art methods both qualitatively and quantitatively, providing a novel perspective for research in visual reconstruction. However, challenges such as the scarcity of large-scale datasets and the noise inherent in non-invasive measurement techniques are acknowledged and warrant further investigation. Overall, the research significantly contributes to understanding the neural mechanisms of the human visual system and paves the way for future advancements in the field.
strength_weakness: strength_weakness: 1. This paper presents a two-stage dual-guided framework for visual stimuli reconstruction, achieving state-of-the-art (SOTA) performance on the largest neuroimaging dataset NSD.

2. The experimental results demonstrate that BrainRAM is capable of reconstructing realistic natural images.

3. The framework proposed by the authors has achieved commendable results in both reconstruction and retrieval tasks.

4. The authors have conducted thorough ablation studies to validate the effectiveness of each component proposed, as well as to demonstrate that utilizing a larger database can enhance the model's performance.
limitations: limitations: 1. The authors have compared their work against baselines that do not employ retrieval augmentation techniques within the paper. It is recommended that the authors also benchmark their method against the performance of the work that utilizes retrieval augmentation, as detailed in the provided reference (https://arxiv.org/pdf/2312.07705).

2. The manuscript's writing quality is subpar, particularly in the 'Contribution' section (lines 150-166), which appears to merely describe the model architecture and performance. It is challenging to discern the authors' true contributions. It is recommended that the authors consolidate these points into a single, clear statement that explicitly highlights their distinct contributions.

3. The section 3.1 'Overview' contains inaccuracies. Among all methods of recording brain responses, fMRI is considered to have a relatively high spatial resolution. However, its temporal resolution is low due to the fact that it captures the slower Blood Oxygen Level Dependent (BOLD) response. Additionally, in the second paragraph, you mention the low signal-to-noise ratio of the fMRI signal, yet the manuscript does not propose a superior denoising algorithm to address this issue. Furthermore, the transition with 'Driven by...' in the third paragraph is inappropriate and requires careful revision.

4. The authors have not made the code publicly available, which raises concerns regarding the reproducibility of the findings presented in the manuscript.