Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
Authors: Jingyuan Sun
Mingxiao Li
Zijiao Chen
Yunhao Zhang
Shaonan Wang
Marie-Francine Moens
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural decoding
brain machine interface
mind reader
visual reconstruction
vision decoding

Abstract: Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception. However, the task is challenging due to the noisy nature of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder. The optimized fMRI feature learner then conditions a latent diffusion model to reconstruct image stimuli from brain activities. Experimental results demonstrate our model's superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-art methods by 39.34% in the 50-way-top-1 semantic classification accuracy. The code implementations is available at https://github.com/soinx0629/vis_dec_neurips/.

Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
Jingyuan Sunt# KU Leuven jingyuan.sum kuleuven.be
Mingxiao Lif KU Leuven mingxiao. @ kuleuven.be
Zijiao Chen
Yunhao Zhang
National University of Singapore zijiao. chen @ e.uuu.edu
Chinese Academy of Science zhangyunhao2021 ia..ac.cn
Shaonan Wang
Farre-Francine Moens
Chinese Academy of Science shaonan. wang @ @lppria..acco
KU Leuven sien.moens @ kuleuven.be
Abstract
Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception. However, the task is challenging due to the noisy na- of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed contrasttn Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from image auto-encoder. The optimized fMRI feature learner then conditions latent diffusion model reconstruct image stimuli from brain activities. Experimental results demonstrate our model's s superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-an the-art methods by 39.34% in the 10-wwyyt semantic classification accuracy. The code implementations will be available at httpttt::/// h/ggtttttbb.///e wwtp///iimo00///ss.deeee...o _ecuuuurppse///////eeeeeee
Introduction
Reconstructing visual stimuli from neural imaging data represents a promising interdisciplinary area between cognitive neuroscience and machine learning (I] A system capable of accurately decoding neural responses to visual input can help illuminate the complex mechanisms underlying the brain't visual perception 3. Furthermore,
elucidate the relationships between human visual systems and computational vision models [ b). Such system also holds the potential to assist patients, particularly those with motor disabilities, in expressing their thoughts and intentions through brain signals.
Functional Magnetic Resonance Imaging (fMRI), non-invasive method to measure brain activity, However,
has been extensively utilized to decipher perceptions from neural responses
'Equal Contribution
*Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

despite its utiliiyy, achieving reliable and robust visual reconstructions from fMRI recordings remains a significant challenge 99. Primarily. the fMRI data is inherently noisy. The recorded signals encompass not only the specific responses elicited by visual stimuli but also incorporate additional sources of noise arising from various cognitive, physiological processes, and scanner operations [1. These noises can obscure the neural activation patterns associated with the stimulus, thereby complicating the direct decoding of visual information from the fMRI data. Moreover, the process by which visual stimulus arouses neural response is dynamic, intricate and multifaceted. It involves multiple stages of neural processing 12 131, from the initial perception in the retina to higher-order cognitive operations within the brain's 's visual and associative regions [1 [5]. The resulting fMRI signal is a highly convolved representation of these distinct processing stages, rather than a linear one [16]. It is thus non-trivial to reverse this process and disentangle the convolved representations to achieve visual decoding.
Existing methods for vision decoding tend to struggle with such challenging complexity. Previous pioneering work has relied on traditional statistical approaches, such as ridge regression, to map [MRI signals back to the corresponding stimuli 117 [8) However, these oversimplified methods often fall short of capturing the non-linear relationship between the stimulus and the neural responses. More recently, deep learning methods such as Generative Adversarial Networks (GAN) [ [9] and latent diffusion models (LDMs) 5 have been adopted to model the non- linearity and yield better results. But the difficulty of disentangling vision-related brain activities from noises still hinders these methods from decoding images with optimal accuracy.
To navigate these challenges, we propose double-phase fMRI representation learning framework. In Phase 1, we pre-train the fMRI feature learner on large-scale unlabeled fMRI data with a novel Double-contrastive Masked Auto-encode (DC-MAE). The DC-MAE helps discern common patterns of brain activities shared among populations over individual noises. In Phase 2, we further tune the feature learner on the IMRI-image parallel data with image auto-encoder. The pixel-level guidance from the image auto-encoder teaches the IMRI auto-encoder to attend to brain signals that are most informative for image reconstruction. The trained fMRI feature learner is then used to condition an LDM to reconstruct image stimuli from brain activities. Experimental results demonstrate that the proposed model generates high-resoluttin and semantically accurate images, outperforming the previous state-of-the. thr- method by 39.3 39.34% in way-top-1 accuracy. Our research paves the way for further exploration of the potential of decoding tasks.
2 Related Works
2.1 Visual Decoding from fMRI
Driven by the substantial potential, recent years have witnessed growing interest in reconstructing visual experiences from fMRI data. This task has been examined in various contexts, including explicitly presented visual stimuli 20]. perceived emotions [21) and even imagined content [I]. Though studies on this task keep emerging, the challenges presented by the low signal-to-noise ratio (SNR) of IMRI data and the high complexity of brain visual representations still exist. In the initial stage in this field, efforts to identify or reconstruct visual images from fMRI primarily utilized handcrafted features 17 and traditional regression models [18] 22]. Nonetheless, oversimplified methods can not fully account for the intricate patterns of brain visual representations, generating blurry and semantically meaningless images.
Recent research has shifted towards artificial neural network representations and deep generative models 23 Typically, such models mapped fMRI signals to image features and fine-tuned pre-
trained generative models like Generative Adversarial Networks (GANs) 19 or Diffusion Models [7 25 26 (2) to generate images from the mapped features. For example, (28) decoded IMRI data to hierarchical image features extracted by pre-trained VGG and fed the predicted features to GAN. But (28). like other parallel work 229 I. used fMRI directly for training and decoding. Without explicitly denoising the fMRI representations, though these works outperformed traditional regression-basst models, they still fell short by generating implausible images. In contrast, our framework contains an individual phase to learn denoised fMRI representations with DC-MAE. 7 adopted a naive MAE to learn fMRI representations. But compared with 77. our framework further introduces pixel-level guidance from the image auto-encoder to help disentangle the vision-related neural activities from background noises. Our model is also not similar to other works [3 31 which

---

directly map fMRI to the image feature space and inevitably face data incompatibility between the two very different modalities. We encourage the emergence of cross- cross-modality representation space by guiding the fMRI auto-encoder with pre-trained image auto-encoder. Experimental results prove the superiority of our methods over these related baselines.
2.2 Diffusion Probabilistic Models
Diffusion probabilistic models have been empirically established as powerful generative models for image generation surpassing GANs (32 in terms of both diversity and fidelity. The diffusion model was first proposed in [33] and further improved by [26] 34] The quality of generated image could also be enhanced by training-fres methods [35 36]. The initial diffusion models were primarily applied
in the pixel space. They achieve impressive success in generating images of high quality, but also suffer from significant drawbacks such as prolonged inference time and substantial training costs [26). To address these two issues, (27) introduce the latent diffusion model (LDM), also referred to as stable diffusion (SD). This approach adopts a pretrained Vector Quantized Generative Adversarial Network (VQGAN) 37 or Variational auto-encoder (VAE) (38) to construct latent image space, within which optimization and evaluation are performed. The LDM not only generates images of high quality, but also alleviates the computational burden. Moreover, the incorporation of a cross-attention mechanism within the attention block of the diffusion UNet model permits the LDM model to offer a broad spectrum of controls in image synthesis. This includes textual controls [39 40 41] controls over images in various domains [42 43 such as depth maps. sketch maps, or candy maps. Such versatility and adaptability give the LDM model substantial advancement in the field of image synthesis.
3 Methods
3.1 Motivation and Overview
In this subsection, we provide a concise analysis of the fMRI data and brain visual representations that motivate the design of our method.
First, the IMRI recordings are inherently noisy, subject to various sources physiological and reanner-related noises 44. FMRI records not only responses to visual stimuli but also signals from other cogmitive processes. Second, fMRI quantifies changes in the blood-oxygen -eae----yg--eee---eeeeee level-dependent (BOLD) signal. Adjacent voxels are often found display similar magnitudes, suggesting the spatial redundancy of fMRI data (2]. Third, neural responses to identical stimuli can exhibit significant across populations.
divergence 45 46
Considering these analyses altogether, we propose double-phase fMRI representation learning framework. In Phase 1, we pre-train an MAE with contrastive loss to learn IMRI representations from unlabeled data. The masking which sets a certain portion of the input data to zero targets the spatial redundancy of fMRI data. The calculation of recovering the original data from the remaining after masking suppresses noises. Optimization of the contrastive loss discerns common patterns of brain activities over individual variances. After pre- treitraining in Phase we tune the fMRI auto-encoder with an image auto-encoder. We expect the pixel- guidance from the image auto-encoder to support the fMR auto-encoder in disentangling and attending to brain signals related to vision processing. After FRL Phase 1 and Phase 2, we apply the representation learned by the fMRI auto-encoder conditions to tune the LDM and generate the image from the brain activities.
3.2 fMRI Representation Learning (FRL)
Phase 1: Pre-training Double-Contrastive Masked Auto-Encoder (DC-MAE) We introduce
method termed the Crouble-Contrastir Masked auto-encc DC-MAE has been specifically designed to pre-train fMRI representations from unlabeled data, as inspired by previous work in visual contrastive learning [7]. As shown in Figure 1, the DC-MAE consists of an encoder Ep and a decoder Dr. EF takes a masked version of the fMRI signal as the input, and Dr is trained to predict the unmasked fMRI. The term
implies that the model optimizes contrastive losses and engages two separate contrasting processes during the representation learning of an fMRI example.
3

---

FRI Phase 1
sugalise
MRI Encoder
MMR Decoder Dp
Ef
O O
fMRI Encoder
fMRI Decoder
patch made
Ef
Dj
FRI Phase
rotch made
MMR Encoder
hig
CA₃
fMRI Decoder
Ep
Dr
30
paich& mook
Image Encoder
E
CA2
Image Decoder
predict
Ej
D₁
U4
Figure 1: The Phase 1 and Phase 2 of the fMRI Representation Learning framework. After the fMRI feature learner is pre-trained in Phase 1, it will be tuned with an image auto-encoo unto-encoder in Phase 2. The definitions of depicted variables in this figure are detailed in Section 3.1.
Specifically, in the first contrasting, each sample (i S n within batch of fMRI examples undergoes two separate instances of random masking, This procedure yields two distinct masked versions of U4 referred to v and oi These masked versions establish positive sample pair for the first contrasting, Subsequently, 1D convolutional layer, characterized by a stride that matches the patch size, tokenizes oith and E." into embeddings. These embeddings are then fed independently into the same fMRI encoder Ep. The decoder Dr. in turn, receives each of the encoded latent representations as input and generates predictions u, and ddm The first contrastive loss, denoted as the cross-contrastive loss, is thus calculated through an InfoNCE loss [8] as follows:
exp (u.m) ueme /T)
Le log
exp
dony
1i /T) Ee exp (oii D. dowi
dong
/T)
In the second contrasting, an unmasked original image vz(i S n)) and its corresponding masked image w form an inherent positive sample pair. Here, vftn denotes the predicted image output from the decoder Dr. The second contrastive loss, referred to the self-contrastive loss, is computed through as:
exp (v-m 2//T)
Cs log
exp (vehm w//) Ejzi exp (154m og
Optimizing the self-contrastive loss Ls can also achieve mask-reconstruction For both Le and Ls. the negative examples (j * i) are sourced from the same batch of instances. Lo and Ls are optimized jointly follows:
L L=voLe + 7sLs
In this equation, the hyper-parameters T0 and "ys regulate the weight of each loss.
Note that this a 1-dimensional vector, not time series, as we averaged the data across time dimension, This results in a spatial pattern of fMRI signal over the visual cortex for each picture viewed by the subject. We then employ ID convolutional model to transform this ID spatial pattern of the 1MRI signal, v. into an embedding.

---

Phase 2: Tuning with Cross Modality Guidance Considering the relatively low signal-to- noise ratio (SNR) and highly convolved nature of fMRI recordings it important for the fMRI feature learner to attend to the brain activation patterns most relevant for visual processing and most informative for reconstruction.
So as shown in Figure 1, after pre-training in Phase I, the fMRI auto-encoder is tuned to reconstruct IMRI with the aid of an MAE for image, and vice versa in Phase 2. Specifically, denote one image from a batch of n samples as ( I n) and the fMRI recorded neural responses to u, as U,. and 2 go through patching and random masking to be 2," and ot" u," and og" are respectively fed in 22 the image encoder E₁ and fMRI encoder EF, getting RE = E, (u.") and h = Ey(,,"). ). For reconstructing the fMRI 14 Eg, and Ep are merged with a cross- cross-attention module as follows:
O7 = wowwing hg 1Qi; Kp- = W hy + bbrr Vp -W NE + 6%
CAr(QT KY. Vp softmar O (KF))) dd
(4)
IV
W and denote the weights and biases of corresponding linear layers. Vdg is the scaling factor and de is the dimension the key vectors. CAA is the abbreviation of cross-attention, CAp(QT + K V is then added to Ep and fed into the fMRI decoder to reconstruct eú as uf.
Dp(hEp CAF(QU) K3 ))
similar computation is also conducted in the image auto-encoder. The output hy, of the image encoder E, is merged with the output of EP through a cross-attention module C'2,, and then used to decode an image u, as ud:
2 = (h + KY VV))))
(6)
The IMRI and image auto-encoders are trained jointly by optimizing the following loss:
L = >- )² + 1u - )²
3.3
Image Generation with Latent Diffusion Model (LDM)
After the IMRI feature learner is trained through FRL Phase and Phase 2, we use its encoder Er to condition a LDM to generate images from brain activities. As displayed in Figure 2, the diffusion model consists of forward diffusion process and a reverse denoising process. The forward process gradually degrades an image to a normal Gaussian noise by incrementally introducing Gaussian noise of variable variance. This process can be mathematically formulated as q(22][_z_)) N( Vazz-1, 1, (1 ox) I), where / denotes the temporal step and a encompasses predefined noise schedule parameters. During the training phase, the diffusion model, specifically the U-Net [49 model, is optimized with the loss function below to learn the noise e, added at each time step in the forward process.
paimple Er, I||e+ co (
(8)
In the backward process, an image is synthesized by progressively eliminating noise from a randomly initialized standard Gaussian noise. The Latent Diffusion Model (LDM) [27] performs both forward and backward processes in low-dimensional latent space which is constructed using pre-trained VQ-VAE model (38) This approach significantly mitigates the computational complexity while simultaneously maintaining the generated image's quality. We leverage the LDM as the backbone of our image generation model.
Decoding MMRI to a natural image can be seen conditional image generation task. Given the relatively low SNR inherent in IMRI, combined with the limited quantity of IMRI-to-image data pairs, training an IMRI-to-image generation model from scratch presents substantial challenges. Consequently, the objective of this phase is to harness the fMRI to extract image-related knowledge from pre-existing conditional image generation model.
Our approach extracts visual knowledge from the pretrained label-to-image LDM to generate image with the conditioning of fMRI. We incorporate the fMRI information into the LDM via cross- attention operation similar to equation (4), as proposed in (27). To further enhance the guidance provided by the conditional information, following the methodology of previous research [7] we
5

---

[8]
Noise Adding
Backward: Denoising
Gl Image
[b
VOGAN Encoder
Forward Diffusion Process
Eg
Noise Adding
21
Zt
Reas
27-1
Zy
Gl Image
Diffusion UNet
[c]
VQGAN Decoder
DG
Generated Image
2
2--1
Zy
fMRI Encoder
Ep
Cross Atteation
Figure 2: [a] Demo of the forward and backward processes of the diffusion model. [b] The forward process of the diffusion model which progressively corrupts image with Gaussian noise. [c] In the backward process, the diffusion model, conditioned on our pretrained IMRI encoder, gradually denoises white noises to generate the image.
implement both cross-aad attsettotntionn conditioning and time steps conditioning S0) During training, given an image and fMRI along with VQGAN encoder Eg and the fMRI encoder EF (which is trained in FRL Phase and 2), we freeze the LDM and finetune the fMRI encoder using the following loss:
Caimple
M ( EF(U))
Here, = = the inference phase, the process begins with standard Gaussian noise time step T. The L.DM follows the backward process sequentially to progressively denoise the hidden representation, condi- tioning on the given fMRI information. Upon reaching time step zero, the VQGAN decoder De is used to convert the hidden representations to an image. The forward and backward process of the LDM are detailed Figure 2[b,c].
VI The and or is the noise schedule of diffusion model. During
Experimental Setup
4.1
IMRI Datasets
The Human Connectome Project (HCP) originally serves as an extensive exploration into the connectivity of the human brain. It offers open-sourced database of neuroimaging and behavioral data collected from 1,200 healthy young adults within the age range of 22-35 years. Currently, stands the largest public resource of MRI data pertaining the human brain, providing excellent foundation for the pre- pre-trainnn training of brain activation pattern representations. Of the subjects involved, 1113 underwent scanning via Siemens Skyra Connectom scanner for 3T MR, while Siemens Magnetom scanner for 7T MR was utilized for the remaining 184, For the scope of this paper, we will predominantly focus on the data derived from the more populated 3T dataset.
HCP
GOD
The Generie Object Decoding (GOD) Dataset is a specialized resource developed for fMRI- based decoding. It aggregates fMRI data gathered through the presentation of images from 200 representative object categories, originating from the 2011 fall release of ImageNet. The training session incorporated 1,200 images (8 per category from 150 distinct object categories). In contrast, noteworthy that unique from those in the training session and were introduced
the test session included 50 images (one from each the 50 object categories). the categories in the test session randomized sequence across runs. On five subjects the fMRI scanning was conducted.
The BOLD5000 dataset is result of an extensive slow event-related human brain fMRI study. comprises 5,254 images, with 4,916 of them being unique. This makes one of the most comprehensive publicly available datasets in the field. The dataset's principal advantage is its high
BOLD5000

---

diversity, enabling the capture of the complexity and variability inherent in natural visual stimuli. The images BOLD5000 were selected from three popular computer vision datasets: ImageNet, COCO, and Scenes. ImageNet provided 1. 1,916 images primarily focusing on singular objects. Meanwhile, COCO contributed 2000 images featuring multiple objects, and Scenes contributed 1000 images depicting hand-crafted indoor and outdoor scenes. Four participants labeled CSI1 through CSI4, were involved this study and underwent scanning via a 3T Siemens Verio MR scanner equipped with a 32-channel phased array head coil.
4.2
Implementation Details
4.2.1 FMRI Representation Learning (FRL)
For both FRL Phase and Phase 2, the fMRI auto-encoder is the same ViT-based masked auto- encoder (MAE). We employed an asymmetric architecture for the fMRI auto-encoder, in which the decoder is considerably smaller with 8 layers than the encoder with 24 layers. We used a larger embedding-tog -ttttcchh size ratio, specifically patch size of 16 and an embedding dimension of 1024 for our model. We used random sparsification (RS) as form of data augmentation, randomly selecting and setting 20% of voxels in each fMRI to zero.
FRL Phase In Phase 1, we train the masked ViT-based fMRI auto-encoder with contrastive loss. For GOD subject 1,4,5 and BOLD5000 CSI 1,2, self-contrastive (7x) and cross-contrastive (7%) loss weights are both 1. The masking ratio is 0.5. For GOD subject 2,3 and BOLD5000 CSI 3,4, % = 1 and % 0.5, masking ratio is 0.75. We set the batch size to 250 and train for 140 epochs on one NVIDIA A100 GPU. We train with 20-epoch warming up and an initial learning rate of 2.5e-4. We optimize with AdamW and weight decay 0.05.
FRL Phase 2 In Phase 2, we tune the fMRI autoencoder jointly with an image auto-encoder, which is pre-trained ViT-based MAE released by [51] The image auto-encoder has 12-layer encoder with a 768 hidden size and a 6-layer decoder with a 512 hidden size. We set the batch size to be 16 and train for 60 epochs. We train with 2-epoch warming up. The initial learning rate is 5.3e-5. We optimize with AdamW W and weight decay 0.05. We freeze the parameters of the decoder of the image-autoencoder and only tune the encoder. The two phases in total take about 12 hours run. After the two phases, we only save the checkpoint of the fMRI encoder which has 15. 16M parameters
4.2.2 Fine-tuning LDM
In this stage, we jointly optimize the parameters of LDM cross-attention heads and the IMRI encoder, while keeping other parameters of LDM unchanged. Given fMRI-image pair, we first use the pre-trained VQ encoder to encode the image to obtain the latent representation which is further used an objective to guide the joint training of the fMRI encoder and LDM cross-attention heads. During training, the fMRI data passes through the fMRI encoder trained using FRL, producing a patchified representation. This representation is then projected into key and value representation of cross-attention modules in the UNet of LDM. Furthermore, it is added to the time embedding to conduct double conditioning. The training follows the regular training pipeline of the diffusion model, where the model is optimized to learn to predict the Gaussian noise added to the image latent representation at each time step with the guidance of the given conditioning information. Here, we use the output of the fMRI encoder as the conditioning information. We conduct training with the following parameters: the batch size of 5, diffusion steps of 1000, the AdamW optimizer, learning rate of 5.5e - 5, and an image resolution of 256 x 256 x 3.
4.3 Baseline Models and Evaluation Metric
We juxtapose our proposed model with recently published benchmarks: IC- GAN [52] Self-supervised auto-encoder AE) [53], and DC-LDM [7]. The IC-GAN is based on an CoatnnceCondiiionedd GAN, whereas the SS-AE utilizes cycle consistency and perceptual losses to reconstruct images from fMRI brain recordings. The DC-LDM employs double-conditionas LDM, demonstrating superior performance on the GOD and BOLD5000 datasets. These benchmarks reflect prevalent methodologies in visual reconstruction.
Baseline Models
Evaluation Metric Visual decoding prioritizes semantic consistency. Therefore, our evaluation metric is the n-way top-k accuracy, which aligns with the precedent set in the literature [53 7 We
7

---

employ the pre-trained ImageNet-Il IK classifier [54] as semantic correctness evaluator. During the evaluation, both the generated image and the corresponding ground truth image are fed into the classifier. Semantic correctness is then determined based on whether the top-k classification among n randomly selected classes aligns with the ground-truth classification. Further details can be referenced in the appendix.
5 Results
5.1
Reconstruction Results
In this section, we present a comparative analysis of our results with preceding studies, which include DC-LDM, IC-GAN, and SS-AE. We have evaluated model on both GOD and BOLD5000 datasets. Following the setting of previous work [7], we first display in detail the results in Figure 3 for GOD subject 3 and BOLD5000 subject CSI 1 which achieve the best performance in their respective dataset. is noteworthy that the original DC-LDM implementation utilized test set fMRI data for tuning, a setting not adopted by IC-GAN and SS-AE. To maintain fair comparison, of test set fMRI data by the DC-LDM. As shown in Figure 3[a], our method surpasses the previous models by large margin. For instance, our model achieves accuracy exceeding that of DC-LDM and CC-GAN by over 39.3 39.34% (calculated by(25.080 (55.0000-77999)/1111 (T79999/220 2200//550000 100% R 39.34%, comparing model's s accuracy 25.080 against the DC-IDM's accuracy 17.98 17.999) and 66. 66.7% respectively. The
first prohibit the use
improvement of our model over the baselines is significant. All the significant results have p-value < 0.01 with paired t-tests.
[a]
[b]
[c]
at
0.25
0.15
0.05
50--ay-44p App-1 100-way-top top-1
50-way 5
way top
OURS
DC-LDD SS-AE
IC-GAN
GT
OURS
DC-LDM
IC-GAN
SS-AE
GT
OURS
DC-LDM
Figure 3: Reconstruction results. [a] Top-1 (al) and top-5 (a2) classification accuracy of our model and other baselines on GOD subject 3. [b] Samples of reconstructed images and their ground truth from GOD subject 3's data. [c] Samples of reconstructed images and their ground truth from the BOLD5000 CSI I's data.
To further investigate the quality of the images produced by different models, we randomly select samples from the GOD test set and exhibit the generated images in Figure 3[b]. The SS-AE model can only generate general layout, while the remaining three models are cable producing images with a semantic meaning similar to that of ground truth image. Our model generates images of superior quality that exhibits a higher degree of semantic consistency with the ground truth images. In Figure 3[c], compare images generated by model and DC-LDM. (IC-GAN and SS-AE were not trained on BOLD5000 in their original paper). Our model achieves 50-way-top- accuracy of 25 CSII. Though both models can produce high-quality images, the image generated by our model bears a semantic meaning more consistent with the ground truth images.
To check if our model may reliably reconstruct brain activities on different subjects, we further evaluate against DC-LDM on all the other subjects of the GOD dataset. The bar charts in Figure 4[a] below show the results in 50-way-top-1 classification accuracy. Our model substantially outperforms the previous state-of-the-arr method (DC-LDM) on all GOD subjects. To achieve DC- LDM's 's reported performance in its original paper 77]. this method need signals from test set fMRI
8

---

data. To ensure fair evaluation, we banned DC-LDM from tuning on the test set in the comparison of the Figure 3. But show here that, our model still largely exceeds DC-LDM on four GOD subjects even after DC-LDM is tuned on the test set fMRI data.
GOD Subject
GOD Subject
GOD Subject
GOD SubjectS
14
18
16
14
12
10
10
10
8
6
2
2
[b]
SS-AB
IMRI-IC-GAN
DC-LDM
JDC-LDM-ist-tunedd
IN OURS
Gr
GT
GT
Ours
Ours
Ours
Ours
Subject
Subject
Subject
Figure 4: Reconstruction performance of our model and other baselines on GOD subjects 1, 2, 4 and 5, measured by 50-way-top-1 classification accuracy
5.2 Ablation Study Results
We train the FRL Phase with different settings of hyper-paramnnn parameters. The trained fMRI encoders then guide the LDM in generating images. We use 50-way-top- accuracy of LDM image generation to measure the influence of hyper-parameter values. Due to space limit, in this section we mainly focus on FRL Phase 2 where direct interaction between image and fMRI data takes place which may more highly influence the final reconstruction performance.
Ablation parameter
ID
AMR loss
Image loes weight
[MR mask ratio
fMR mask ratio
Dec. layer
Accuracy
1
0
0.75
0.75
6
12.52
IMRI image reconstruction
2
1
0
0.75
0.75
6
16.64
Joss weight
4
1
0.75
0.75
6
19.64
1.5
0.5
0.75
0.75
19.76
0.75
20.8
Ablation parameter
ID
[MR] rec. less weight
Image rec. loss weight
(MR mask natio
Image mask zatio
Dec. layer sum
Accuracy
7
0.5
0.75
15.52
8
0.5
6
17.72
IMRI and image mask ratio
9
0.5
0.25
6
19
19.64
10
0.75
0.5
6
21.04
Ablation parameter
ID
[MRJ Jess weight
Image Joss weight
JMRI mask ratio
Image mask ratio
Dec. layer
Accuracy
number of
11
0.75
2
17.44
image
12
0.75
0.5
4
17.76
decoder
4
1
0.75
0.5
6
19.64
layer
17.44
Best parameter
14
0.25
1.5
0.75
0.5
6
25.08
Table 2: Ablation study of FRI. Phase on GOD subject Cells with colored shades denote the hyper- parameters to be tuned in one ablation group. For example, cells with yellow shades denote that fMRI and image mask ratio are the parameters be tuned while other parameters are kept the same.

---

Reconstruction Loss Weight In the FRL Phase 2, the feature learner is tuned by optimizing the joint loss of fMRI and image reconstruction, as in equation (7). So we first focus on the influence of the fMRI and image reconstruction loss weights, namely "F and 7/- The results are reported in Table (ID1-6). experiment
and 2, we set one of the weights to 0 to evaluate the necessity of using the corresponding loss. We find that jointly optimizing the fMRI and image reconstruction loss is necessary to achieve optimal task performance. We adjust the Yf and %/ and find that setting in ID where >F 0.25 and 7/- = 1.5 yields the highest performance.
Mask Ratio In the FRL Phase 2, both the fMRI and image auto-encoders are based on MAE. The mask ratio of the imput is an important setting for these models. We demonstrate the influence of the mask ratio setting in Table 2 (ID and ID 7-10). We find that applying higher mask ratio on fMRI data and a lower mask ratio on image data generally leads to better performance. LDM conditioned by the fMRI encoder which is trained with an fMRI mask ratio of 0.75 and image mask ratio of 0.5 leads to the highest reconstruction accuracy.
Decoder Layers Following 7 51 [55), for both the IMRI and the image auto-encoder, we build asymmetric architectures where the decoder is much smaller than the encoder. In Table 2 (ID 11-13), we report the results of tuning decoder depth. We apply the same depth for the two decoders. We find that a moderate decoder depth of 6 produces optimal results. Neither a too shallow nor a too deep decoder improves reconstruction performance on GOD subject
6 Discussion
The experimental findings indicate that, with the proposed fMRI representation learning framework and pre-trained LDM, we can achieve degree of visual reconstruction of human brain activities, substantially outperforming baselines. Nonetheless, our analysis also uncovers some limitations within our model. Primarily, our model appears be slightly affected by categorical bias issue. We hypothesize that this may stem from the inherent bias present in the dataset used to train the LDM. Additionally, while our model demonstrates proficiency in capturing high-level semantics, it sometimes fails to reconstruct specific details of an image. A plausible explanation could be the concurrent imagination of multiple objects by the participants during data collection, which inevitably results in a noisy fMRI feature. Contrasting with general image generation, which is typically characterized by a focus on diversity, visual decoding underscores the importance of consistency, thereby necessitating reduction in bias during the generation process. As such, the exploration of techniques to minimize the influence of data bias, as well as methods to enhance the reconstruction of image details when generating images from fMRI data, would hold significant academic value and interest.
7 Conclusion
In this work, we propose a double-phase fMRI representation learning framework (FRL) with an LDM to reconstruct visual experiences from brain activities. The FRL denoises fMRI features by contrastive masked modeling in Phase 1. And in Phase 2 it learns to disentangle and attend to brain activation patterns most informative for visual decoding with the guidance of an image MAE. With the conditioning of the optimized fMRI feature learner, we show that an LDM generates images of high quality, largely outperforming the previous state-of-the-art. Extensive ablation studies further verify the effectiveness of each component that we propose.
8 Ethical Statements
We used preprocessed data from publicly available datasets. The fMRI data that train with have been processed and do not contain any data that can be directly linked to the participants identities. The collection procedure of the fMRI undergoes strict ethical review as stated in their original paper.
Acknowledgements
This work is funded by the CAL.UULSS project (Furopean Research Council Advanced Grant H2020-ERC-2017-ADG 788506) and the Flanders AI Impuls Programme FLAIR.
10

---

References
[1] Horikawa and Y. Kamitani, "Generic decoding of seen and imagined objects using hierarchical visual features," Nature Communications, vol. 8, 2015.
[2] K. Uğurbil, J. Xu, E. J. Auerbach, S. Moeller, A. T. Vu, J. M. Cuarte-Carvajaling C. Lenglet, X. Wu, S. Schmitter, P. Van de Moortele et al., "Pushing spatial and temporal resolution for functional and diffusion mri in the human connectome project," Neuroimage, vol. 80, pp. 80-104, 2013.
[3] E. W. Contini, S. G. Wardle, and T. A. Carlson, "Decoding the time-course of object recognition in the human brain: From visual features categorical decisions," Neuropsychologia, vol. 105,
pp. 165-176. 2017.
[4] Z. Ren, J. Li, X. Xue, X. Li, F. Yang, Z. Jiao, and X. Gao, "Reconstructing seen image from brain activity by visually-guide cognitive representation and adversarial learning." Neurolmage, vol. 228, 2021.
[5] Y. Takagi and S. Nishimoto, "High-resolution image reconstruction with latent diffusion models from human brain activity," bioRxiv, pp. 2022-11, 2022.
[6] Sun, M. Li, and M.-F. Moens, "Decoding realistic images from brain activity with contrastive self-supervision and latent diffusion," in Proceedings of the 26th European Conference on Artificial Intelligence (ECAI 2023). 2023, 2023.
[7] Z. Chen, J. Qing. T. Xiang, W. L. Yue, and J. H. Zhou, "Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding," in arXiv, November 2022. [Online]. Available: https///arxivorr/ab//221111
[8] Sun and M.-F. Moens, "Fine-tuned vs. prompt-tunce tuned supervised representations: Which better account for brain language representations?"" in Proceedings of IJCAI, Macau, China, 2023.
[9] M. Mozafari, L.. Reddy, and R. van Rullen, "Reconstructing natural scenes from fmri patterns using bigbigan," 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2020.
[10] T. T. Liu, "Noise contributions to the fmri signal: An overview," Neurolmage, vol. 143, pp. 141-151, 2016.
[11] J. C. Brooks, O. K. Faull, K. T. Pattinson, and M. Jenkinson, "Physiological noise in brainstem fmri," Frontiers in human neuroscience, vol. 7. p. 623, 2013.
[12] I. I. Groen, E. H. Silson, and C. I. Baker, "Contributions of low-and high-level properties to neural processing of visual scenes in the human brain," " Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 372, no. 1714, p. 20160102, 2017.
[13] D. J. Kravitz, K. S. Saleem, C. I. Baker, L. G. Ungerleider, and M. Mishkin, "The ventral visual pathway: an expanded neural framework for the processing of object quality." Trends in cognitive sciences, vol. 17, no. 1, pp. 26-49, 2013.
[14] T. C. Kietzmann, C. J. Spoerer, I.. K. Sörensen, R. M. Cichy, O. Hauk, and N. Kriegeskorte, "Recurrence is required to capture the representational dynamics of the human visual system," Proceedings of the National Academy of Sciences, vol. 116, no. 43, pp. 21 854-21 863, 2019.
[15] J. DiCarlo, D. Zoccolan, and N. C. Rust, "How does the brain solve visual object recognition?" Neuron, vol. 73, no. 3, pp. 415-434, 2012.
[16] J. Sun, S. Wang, J. Zhang, and C. Zong, "Neural encoding and decoding with distributed sentence representations." " IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 2, pp. 589-603- 603, 2020.
[17] K. N. Kay. T. Naselaris, R. J. Prenger, and J. L. Gallant, "Identifying natural images from human brain activity," Nature, vol. 452. pp. 352-355, 2008
[18] T. Horikawa, M. Tamaki, Y. Miyawaki, and Y. Kamitani, "Neural decoding of visual imagery during sleep," Science, vol. 340, pp. 639 642, 2013.
[19] K. Seeliger, U. Güglüü L. Ambrogioni, Y. Güęlüüürk, and M. van Gerven, "Generative adversarial networks for reconstructing natural images from brain activity." Neurolmage, vol. 181, pp. 775- 785, 2017.
11

---

[20] Y. Zhang, T. Bu, J. Zhang, S. Tang, Z. Yu, J. K. Liu, and T. Huang, "Decoding pixel-level image features from two-photon calcium signals of macaque visual cortex," Neural Computation. vol. 34, pp. 1369-1397, 2022.
[21] T. Horikawa, A. S. Cowen, D. Keltner, and Y. Kamitani, "The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions," iScience, vol. 23, 2019.
[22] A. G. Huth, T. Lee, S. Nishimoto, N. Y. Bilenko, A. T. Vu, and I.. Gallant, "Decoding the se- mantic content of natural movies from human brain activity," Frontiers in Systems Neuroscience, vol. 10, 2016.
[23] T. Fang, Y. Qi, and G. Pan, "Reconstructing perceptive images from brain activity by shape- semantic gan," ArXiv, vol. abs/2101. 1a..1111..2..,,, 12083, 2021.
[24] M. Ferrante, T. Boccato, and N. Toschi, "Semantic brain decoding: from fmri to conceptually similar image reconstruction of visual stimuli," arXiv preprint arXiv:2212.06726, 2022.
[25] Ozcelik and VanRullen, "Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion," arXiv preprint arXiv:2303.05334, 2023.
[26] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances in Neural Information Processing Systems, vol. 33, pp. 6840-6851, 2020.
[27] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image symthe- sis with latent diffusion models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6954-0695.
[28] G. Shen, K. Dwivedi, K. Majima, T. Horikawa, and Y. Kamitani, "End-to-end deep image reconstruction from human brain activity," Frontiers in computational neuroscience, vol. 13, p. 21, 2019.
[29] G. Shen, T. Horikawa, K. Majima, and Y. Kamitani, "Deep image reconstruction from human brain activity," " PLoS Computational Biology, vol. 15, 2017.
[30] X. Qian, Y. Wang, Y. Fu, X. Xue, and J. Feng, "Semantic neural decoding via cross generation," arXiv preprint M.XXiv23333147300 2023.
[31] S. Lin, T. Sprague, and A. K. Singh, "Mind reader: Reconstructing complex images from brain activities," arXiv preprint arXiv:2210.0/700 2022.
[32] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and
Y. Bengio, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp.
139-144, 2020.
[33] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics," " in International Conference on Machine Learning. PMLR, 2015, pp. 2256-226622 2265.
[34] A. Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in Interna- lional Conference on Machine Learning. PMLR, 2021, pp. 8162-8171.
[35] M. Li, T. Qu, W. Sun, and M.-F Moens, "Alleviating exposure bias in diffusion models through sampling with shifted time steps," arXiv preprint rxXv:2305.15583, 2023.
[36] Bao, C. Li, Zhu, and B. Zhang. "Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models," arXiv preprini arXiv:2201.065003 2022.
[37] P. Esser, R. Rombach, and B. Ommer, "Taming transformers for high-resolution image synthesis," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 873-17 883.
[38] A. Van Den Oord, O. Vinyals el al., "Neural discrete representation learning," Advances in neural information processing systems, vol. 30, 2017.
[39] C. Saharia, W. Chan, S. Saxena, L. J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan,
S. S. Mahdavi, R. G. Lopes et al., "Photorealistic text-to-image diffusion models with deep language understanding," arXiv preprint arXiv:2205.1 11480, 2022.
[40] G. Kim, T. Kwon, J.d C. Ye, "Diffusionclip: Text-guided diffusion models for robust image manipulation," in Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 2426-2435.
12

---

[41] N. Ruiz, Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." arXiv preprint arXiv:2208.122 2022.
[42] L. Zhang and M. Agrawala, "Adding conditional control to text-to-image diffusion models," arXiv preprint arXiv:2302.05543, 2023.
[43] C. Mou, X. Wang, L. Xie, J. Zhang, Z. Qi, Y. Shan, and X. Qie, *T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models," arXiv preprint urXiv:2302.084533 2023.
[44] T. B. Parrish, D. R. Gitelman, K. S. LaBar, and M.-M. Mesulam, "Impact of signal-to-noise on functional mri," Magnetic Resonance in Medicine: Official Journal of the International Society for Magnetic Resonance in Medicine, vol. 44, no. 6, pp. 925-932, 2000.
[45] G. K. Aguirre, R. Datta, N. C. Benson, S. Prasad. S. G. Jacobson, A. V. Cideciyan, H. Bridge, K. Watkins, O. H. Butt, A. S. Dain al., "Patterns of individual variation in visual pathway structure and function in the sighted and blind," PLoS One, vol. 11, no. 11, c0164677, 2016.
[46] J. Sun, S. Wang, J. Zhang, and C. Zong, "Towards sentence-level brain decoding with distributed representations," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 7047-7054.
[47] X. Chen, Xie, and K. He, "An empirical study of training self-supervised vision transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9640-9649.
[48] A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," " arXiv preprint arXiv: /807,03748 2018.
[49] O. Ronneberger, P. Fischer, and Brox, "U-net: Convolutional networks for biomedical image segmentation." in Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234-241.
[50] P. Dhariwal and A. Nichol, "Diffusion models beat gans image synthesis," Advances in Neural Information Processing Systems, vol. 34, pp. 7780-8794, 2021.
[51] K. He, X. Chen, Xie, Li, Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022, pp. 16 009.
[52] Ozcelik, B. Choksi, M. Mozafari, L. Reddy, and R. VanRullen, "Reconstruction of perceived images from fmri patterns and semantic brain exploration using mostance-conditionn 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022, pp. 1-8.
gans," in
[53] G. Gaziv, R. Beliy, N. Granot, A. Hoogi. F. Strappini, T. Golan, and M. Irani, "Self supervipe natural image reconstruction and large-scale semantic classification from brain activity," Neu- rolmage, vol. 254, p. 119121, 2022.
[54] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11923 2020.
[55] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, "Simmim: A simple framework for masked image modeling," " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9653-9663.
13

---

Appendix
A.1 Reconstruction Performance
We provide the performance of our model on BOLD5000 subjects 1, 2. 3, and 4 in Table A.I. Following previous work (7), all results are presented in 50-way-top-l classification accuracy.
BOL.D
CSI
OURS
18.69
16.14
18.98
Table A. Reconstruction performance of our model on BOLD5000 subject CSI 1-4, measured by 50-way-top-1 classification accuracy.
A.2 Examples of Reconstructed Images
Figures A.2 and A.3 present images generated by our model using fMRI data from GOD and BOLD5000 datasets, respectively. We generated all images at resolution of 256 x 256 x 3 using
250 PLMS steps. More samples can be generated using our code base the supplementary materials. The code will be open-sourced with the camera ready version of this paper.
14

---

A.2.1 Reconstructed Images from GOD Dataset
GOD Subject I
GOD Subject 2
GOD Subject 4
GOD Subject
Figure A.2: Randomly selected reconstructed images from GOD subject 1, 2, 4 and 5. For each subject, the upper line shows the ground truth images while the lower line shows the reconstructed images by our method.
14

---

A.2.2 Reconstructed Images from BOLD5000 Dataset
BOLDS000 CSI 1
BOLDS000 CSI 2
BOLD5000 CSI 3
BOLD5000 CSI 4
Figure A.3: Randomly selected reconstructed images from BOLD5000 CSI 1-4. For cach subject, the upper line shows the ground truth images while the lower line shows the reconstructed images by our method.
15

---

A.3 Evaluation Metrics
We use the common N-trial, n-way top-1 semantic classification as the main evaluation metries. This evaluation method is summarized in the algorithm below:
Algorithm Iterative Reasoning Module
Input:
pre-trained image classifier F, generated image ă, corresponding ground truth (GT) image Zgt Output:
success rate [0, 1] for trail = 1 to do
Ogt F Fx get the prediction of GT image
pred (()) get the output probabilities of generated image
p {p, P > .. P } generate probabilities set contains randomly selected from pred
and Ygt
Success if arg min Vgt
end for
return sr = number success N
17


Reviews:
[{"review_id": "eKZqhuyNkx", "reviewer": "Reviewer_ir1U", "summary": "The paper propose a novel two-phase fMRI representation learning method to encode the visual stimuli from neural responses. It is a significant challenging task due to noisy fMRI signals and complex intricate pattern of brain visual representation. The proposed two-phase can reconstruct image stimuli from brain activities.", "questions": "see above", "limitations": "the proposed method is too complex to follow.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The overall framework is novel and interesting. The use of double-contrastive mask auto-encoder and image-guided auto-encoder are novelty.\n2.The introduction is clear.\n3.The superiority experimental results are obtained.", "weaknesses": "1.The motivation of two phase is confused. Why two phase representation learning is needed? And what is the difference between the two phase?\n2. How the contrastive MAE works? It is not clear\n3.The model may be too big? How the efficiency?\nThe experiments are weak. How to set the mask? The model’s size? More quantification is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for all the constructive feedback. We answer your questions as follows. \n\nQ1: Why is two-phase representation learning needed? What is the difference between the two phases?\n\nA1: The two-phase design stems from the unique characteristics and challenges posed by fMRI data in the context of visual reconstruction, as stated in Section 3.1, lines 104-122.  For your convenience, we elucidate the motivation of and difference between the two phases as follows.\n\nFirst,  fMRI data is spatially redundant. Adjacent voxels tend to display similar magnitudes.  The mask and predict methodology in Phase 1 is designed to help the fMRI encoder learn the underlying structure of the input brain data, crucial for understanding brain dynamics. \n\nSecond, fMRI data is noisy and subject to individual biological variances. Masked autoencoding in Phase 1 helps suppress noise. Optimization of the contrastive loss further discerns common patterns of brain activities over individual variances. \n\nThird, the process by which a visual stimulus arouses a neural response involves multiple stages of neural processing. The resulting fMRI signal is a highly convolved representation of these distinct stages.  So we design Phase 2 with cross-modality guidance, which aims to instruct the fMRI encoder to capture the most informative signals from the convolved fMRI data for visual reconstruction. \n\nIn essence, our two-phase work synergistically: Phase 1 focuses on data denoising and structure understanding, while Phase 2 emphasizes extracting information crucial for visual reconstruction. Collectively, they ensure precise differentiation between cognitive states, which is paramount for brain decoding.\n\nQ2: How does the contrastive MAE work?\n\nA2:  Thank you for seeking clarity on the workings of the contrastive MAE. You can refer to Figure 1 and Section 3.2 Phase for details of contrastive MAE works.\nTo elucidate step by step:\n\n1) For each fMRI input sample, we generate two distinct masked versions. \n\n2) These masked versions are then processed through the fMRI autoencoder, resulting in two reconstructed samples. \n\n3) These two reconstructed samples are treated as a pair of positive samples.\n\n4) Furthermore, the association between each reconstructed sample and the original unmasked sample also forms a pair of positive samples. \n\n5) Reconstructed samples from other fMRI inputs in the same batch serve as negative samples. \n\n6) With these established positive and negative pairs, we then optimize the contrastive loss as detailed in equations (1-3). \n\nQ3: What is the size and efficiency of the model? How to set the mask? \n\nA3:  Size and Efficiency of Model: We take training on the GOD dataset as an example.\nFor the fMRI representation learning model, we train Phase 1 for 150 epochs and Phase 2 for 60 epochs on one Nvidia A100 GPU. The two phases in total take about 12 hours. After the two phases, we only save the checkpoint of the fMRI encoder which has 15.16M parameters. \nFor the diffusion model, we use the pre-trained label-to-image latent diffusion model. The model has 401.32M parameters, but during finetuning, we only tune the weights in the cross-attention layers and the fMRI encoder which has 17.4M parameters in total. We use one Nvidia-V100 GPU to finetune the model for 500 epochs and it takes around 20 hours.\n\nSetting of Mask: We have conducted experiments to study the effects of masking ratio on reconstruction performance and the results are detailed in the ablation Table 1 and 2. We also include the mask setting to achieve the best reconstruction performance on the last line of Table 2, where the fMRI mask ratio is 0.75 and the image mask ratio is 0.5. \nWe will definitely further clarify these quantifications in the revision. \n"}, {"review_id": "xjEhAKL2l3", "reviewer": "Reviewer_zH9U", "summary": "The paper proposes an approach for decoding visual stimuli from neural responses (fMRI images). The rationale behind the proposed approach lies in the difficulty of learning the complex relationship between a stimuli and the neural responses to it, and the noisy nature of fMRI images.\n\nThe authors propose a multiple phase approach comprising a contrastive learnign based method for learning denoised representations of fMRI brain activities (a new DC-MAE method is proposed), a feature learner that combines previsous representation space with image representation through  cross attention and a diffusiuon model to reconstruct image stimuli from brain activities which is conditioned on the opuput of the feature learner.    \n", "questions": "\nTo which space does v_i belongs, is is a time series ?  I don’t understand the use of 1D conclutional models that maps v_i^m_1  and v_i^m_2 into embeddings.\n\nIn Phase2 are all models (E_F, E_I, D_F, D_I) retrained or are only the cross attention layers trained ?\n\nThe way the latent diffusion model is conditionned seems new. Hwo does this compare to previous attempts of designing conditional latent diffusion models ?\n\nConcerning metrics it is said in section 4.3 « We employ the pre-trained ImageNet-1K classifier [50] as a semantic correctness evaluator. » wht does this mean ?  \n\nConcerning the top-k accuracy experiments did the authors use a statistical test to attest the significativity of the result ?   ", "limitations": "limitations are addressed ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approcah looks like a smart agregation of state of the art components that allows reaching high top-k accuracy (the chosen metric).\n\nThe paper is easy to read and the methodological choices are well motivated. \n\nThe method is validated by top k accuracy and achieves significantly better results than the baselinesit is compared to. \n\nA thourough ablation study shows the impact of the various components and of the hyperparameters of the method.\n", "weaknesses": "Few aspects of the method are not enough detailed (see questions)\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for all the constructive feedback and for appreciating our work. We answer your questions as follows. \n\nQ1: To which space does v_i belong, is it a time series? Why  1D convolutional model is used to map v_i^m_1 and v_i^m_2 into embeddings?\n\nA1: Thank you for your question regarding the nature of v_i and the usage of 1D convolutional models in our study. \nIn our work, v_i denotes the fMRI signal when the subject is viewing a picture. Importantly, it is a 1D vector, not a time series, as we have averaged the data across the time dimension. This results in a spatial pattern of fMRI signal over the visual cortex for each picture viewed by the subject. We then employ a 1D convolutional model to transform this 1D spatial pattern of the fMRI signal, v_i, into an embedding. We will add these details during revision.\n\nQ2: In Phase 2 are all models (E_F, E_I, D_F, D_I) retrained, or are only the cross attention layers trained?\n\nA2: All models (E_F, E_I, D_F, D_I) are retrained, as described in line 164.\n\nQ3: The way the latent diffusion model is conditioned seems new. How does this compare to previous attempts at designing conditional latent diffusion models?\n\nA3: Thanks for your question. We apply this type of conditioning on LDM given the noisy nature of fMRI data. We adopt both cross-attention conditioning and time-step conditioning. Time-step conditioning strengthens the overall conditioning effect. Cross-attention conditioning helps to create a stronger condition. This double conditioning is inspired by the conditioning methods in [1] and [2].\n\n[1] R. Robin, et al. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on  CVPR, pages 10684–10695.   (ref. 24 in our paper)\n\n[2] P. Dhariwal et al.“Diffusion models beat gans on image synthesis,” Advances in NeurIPS, vol. 34, pp. 8780–8794, 2021. (ref. 44 in our paper)\n\n\nQ4: What does this  “We employ the pre-trained ImageNet-1K classifier [50] as a semantic correctness evaluator”  mean?\n\nA4: Thanks for your question. We follow the previous work using n-way top-k accuracy to evaluate our model.  A detailed explanation of the computation for this metric can be found in Algorithm 1 provided in the appendix.  Regarding the statement,  “We employ the pre-trained ImageNet-1K classifier as semantic correctness evaluator”, we mean that we use the pre-trained ImageNet-1k classifier to classify both ground truth images and the generated images. These predicted classes are then utilized in computing the final metrics as explained in Algorithm 1.\n\nQ5: Concerning the top-k accuracy experiments did the authors use a statistical test to attest the significance of the result? \n\nA5: Yes, the improvement of our model over the baselines is significant, including the previous state-of-the-art DC-LDM. All the significant results have p-value < 0.01 with a paired t-test. We will further clarify them in the revision. \n"}, {"review_id": "1YklnduLK9", "reviewer": "Reviewer_BiRm", "summary": "This paper proposed to decode visual stimuli from neural responses recorded by fMRI. First, it pretrains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. Second, it tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.\n", "questions": "Please see the comments above.\n", "limitations": "The limitation is discussed in Section 6, however, since human data is used in the study, the authors can discuss some potential ethical concerns.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The method is straightforward and easy to understand.\n(2) The related work discussion seems comprehensive. \n", "weaknesses": "(1) The selection of baseline methods for comparison appears to be insufficient, thus hindering the ability to effectively demonstrate the effectiveness of the proposed method. To enhance the evaluation, it would be beneficial for the authors to include a comparison with the methods mentioned in the related work section.\n\n(2) Another crucial factor that warrants consideration is the quality of the fMRI data, as it can significantly impact the performance of the model. It would be valuable if the authors could provide some ablation studies pertaining to this aspect as well.\n\n(3) The evaluation metric solely employed is n-way top-k accuracy, which could be supplemented with additional evaluation metrics to provide a more comprehensive assessment. Metrics such as FID (Fréchet Inception Distance), SSIM (Structural Similarity Index), MSE (Mean Squared Error), among others, would be advantageous for a more qualitative evaluation.\n\n(4) Since human data is used in the study, the authors can discuss some potential ethical concerns. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for all the constructive feedback. We answer your questions as follows. (Due to the rebuttal length limit, we might summarize some of your questions.)\n\nQ1: About the selection of baselines and consideration of other models cited in related work as baselines.\n\nA1: Thanks for your advice on baseline selection. We have three points to clarify. First, we follow previous work DC-LDM’s selection for baselines to ensure a fair evaluation.  DC-LDM is the previous SOTA model which has been peer-reviewed and accepted by a top-tier conference (CVPR 2023). Compared against DC-LDM, our model shows substantial relative improvements of 39.04%, as depicted in the Results Section 5.1.  Second, we described GAN-based, Diffusion-based, and VAE-based methods in the related work. All three categories have their corresponding model in the baselines, IC-GAN for GAN, DC-LDM for diffusion, and SS-AE for VAE. Third, we did not include traditional regression-based methods, since their performances are not comparable to the latest deep learning based methods as we introduce in the related work Section 2.1. \n\n\nQ2: About the impact of fMRI data quality and ablation studies pertaining to this aspect.\n\nA2: Thank you for emphasizing the significance of fMRI data quality. We have three points to explain regarding your questions. \nFirst, our method is explicitly tailored to fMRI’s noisy nature. We employed a contrastive masked autoencoder to derive denoised fMRI representations. The effectiveness of this approach is evident in our superior task performance. \nMoreover, we indeed have some related ablation studies with masking on fMRI, as reported in Table 1 and Table 2. In masking, we randomly set part of the fMRI representation to zero which is also a kind of quality impairing. Our optimal task performance is achieved by an fMRI encoder trained with 75% masked fMRI inputs, proving our model’s robustness to fMRI quality. \nLast, it's pertinent to note that non-invasive neuroimaging, including fMRI, inevitably comes with inherent noises. Despite this, the BOLD5000 and GOD datasets, utilized in our study, have been widely acknowledged in brain decoding tasks, underscoring their quality. For example, BOLD5000 was evaluated using MRIQC [1], yielding a Signal-to-noise ratio of 5.157, attesting to its reliability. \n\n[1] Chang, Nadine, et al. \"BOLD5000, a public fMRI dataset while viewing 5000 visual images.\" Scientific data 6.1 (2019): 49. (ref. 46 in our paper)\n\nQ3:  About supplementing additional evaluation metrics  to provide a more comprehensive assessment. \n\nA3:  Thanks for your suggestions. We follow the previous SOTA DC-LDM for the evaluation setting. We further evaluate our model using MSE and SSIM based on your suggestions. We report the comparisons of our model with DC-LDM. After averaging across all subjects, our model achieves an SSIM of 52.48 and MSE of 49.59, significantly outperforming DC-LDM’s 51.85 SSIM and 51.38 MSE on the BOLD5000 dataset (p<0.01 with paired t-test for all the significant results).  We will definitely supplement these metrics in the revised manuscript.\n\nQ4:  Since human data is used in the study, the authors can discuss some potential ethical concerns.\n\nA4: Thanks very much for the suggestions. We used preprocessed data from publicly available datasets. The fMRI data that we train with have been processed and do not contain any data that can be directly linked to the participants’ identities. The collection procedure of the fMRI undergoes strict ethical review as stated in their original paper. We will definitely add a section of the ethical statement at the end of the paper. \n"}, {"review_id": "FzlswjWxhq", "reviewer": "Reviewer_zrBS", "summary": "The authors aim to decode the visual stimuli from neural responses by reverse-mapping the signals from functional MRI (fMRI) to the images the participants see while being scanned. The authors claimed to achieve this through a two-phase framework. In the first phase, they pre-train an fMRI feature learner inspired by mask auto-encoder (MAE) from unlabeled fMRI data. This phase tries to discern the common patterns and features in fMRI across participants responding the same stimuli. In the second phase, they further tune the fMRI feature learner with cross-modality correspondence from the visual stimuli, i.e., the images. This process is similar to the cross-modality guidance commonly used in vision language models. Finally, after the two phases of training, the fMRI feature learner is used to generate signals that serve as conditions for a latent diffusion model (LDM) for image generation.\n\nThe authors demonstrated that the final model is able to produce images from the participants’ fMRI signals such that the generated images well correspond — in category — to what the participants see. On the quantitative benchmarks, the proposed method outperform the baselines by a significant margin.\n\nAs much as I appreciate this work, I would like to demystify it a bit. In my opinion, it’s effectively performing two tasks sequentially: (1) fMRI signal classification into 50 or 100 classes, and (2) class-conditional image generation. The holistic framework and storytelling might have made it more impressive than it actually is — though admittedly it is still impressive.\n", "questions": "1. Do you think it is reasonable to present the intermediate fMRI representation accuracy as described in Weaknesses 2? If I understand it correctly, you have $image_{real} \\rightarrow (human) \\rightarrow fMRI \\rightarrow representation \\rightarrow image_{gen}$, and you are showing $acc(classifier(image_{gen}), label(image_{real}))$. I wonder if you can also show $acc(classifier'(representation), label(image_{real}))$ and even $acc(classifier'(representation), classifier(image_{gen}))$.", "limitations": "Nothing came to my mind.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. “Mind reading” is a very bold topic. Generating the image in a participant’s mind from the fMRI recordings is arguably a reasonable way.\n\n2. The design with three stages: contrastive pre-training, cross-modality fine-tuning and latent diffusion model is very reasonable and well conveyed.\n\n3. The ablation studies are presented neatly.", "weaknesses": "1. The experimental results, though impressive, do not seem very comprehensive. For example, in Figure 3, qualitative results are shown for two datasets while bar plots are only displayed for one.\n\n2. Also, I would recommend showing some intermediate results — for example, directly evaluating the fMRI representation accuracy by training a simple model to classify the learned fMRI representations after phase 2.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the valuable feedback and for appreciating our work.  Our responses to the weaknesses and questions are as follows. \n\nWeakness 1: In Figure 3, qualitative results are shown for two datasets while bar plots are only displayed for one.\n\nAnswer 1: Thanks very much for the advice on the presentation of the results. We put the bar plot of BOLD5000 in the appendix due to the space limit. We will definitely put it in the main text during revision.\n\nWeakness 2 and Question 1: The reviewer would recommend showing some intermediate results — for example, directly evaluating the fMRI representation accuracy by training a simple model to classify the learned fMRI representations after phase 2. Do you think it is reasonable to present the intermediate fMRI representation accuracy?\n\nAnswer 2:  Yes, it is very reasonable to evaluate the intermediate fMRI representation and we thank you for the insightful suggestion.  While assessing these representations is valuable, directly classifying them using our datasets is challenging. In BOLD5000, most image classes contain just one image. For GOD, with its 1200 images, test and training sets differ totally in class composition. Given fMRI's inherent noises and lessons from related works, training a precise classifier on these intermediate representations is very complex.\n\nHowever, prompted by your feedback, we've explored an alternative: cross-modality reconstruction in Phase 2, as illustrated in Figure 1. We believe that evaluating the masked auto-encoding results for images in Phase 2 can produce a fitting metric to assess intermediate representations. Specifically, for one image-fMRI sample pair, we mask 50% of the image.  The masked image is input into the image encoder, while the fMRI sample is directed to the fMRI encoder. The outputs from both encoders are combined and subsequently fed into the image decoder for reconstruction. Pearson’s correlation between the original and reconstructed images can also measure the quality of fMRI encoder's representations. With average correlations of 0.8971 for GOD subjects and 0.8703 for BOLD5000 subjects, we demonstrate the robustness of our intermediate fMRI representations.\n"}, {"review_id": "kLCU4jkwzk", "reviewer": "Reviewer_5Cka", "summary": "The paper focuses on decoding visual stimuli from recorded fMRI activity. Proposed method initially pre-trains an fMRI feature learner (and essentially a signal denoiser) on unlabeled data using a contrastive training scheme resembling to masked autoencoders. Subsequently this feature encoder is finetuned by using guidance from an image auto-encoder to attend to signal patterns that are informative for reconstruction. Output of this fMRI feature learner is then used to condition a latent diffusion model to be able to visually reconstruct the stimuli in the image domain with high resolution.", "questions": "- Motivation of the study from a non-invasive brain-machine interface perspective is somewhat disputable, since the neurophysiological source signal considered here is fMRI. Can the authors extensively discuss and justify to the reader why their choice of fMRI (with low temporal resolution and longer temporal scales of responses, noisy nature, and hard-to-integrate experimental recording/design) would be a better candidate than other alternatives (e.g., EEG, or (partly-)invasive methods) to tackle this problem?\n\n- Figure 2 with the \"VQGAN\" does not make sense. Starting from the last paragraph of page 5 and then in Figure 2, the paper talks about a VQGAN, whereas the LDM that the authors use should certainly be a pre-trained VQ-VAE with an encoder-decoder pair? Also based on the notation in Eq 9 and line 193, I presume the VQ-VAE encoder should be E_g rather than E_G on line 191?\n\n- Description of Phase 1 has some ambiguities. Beginning of Sec 3.2 was understandable. Then line 140 starts to talk about an unmasked original \"image\" at some point, whereas the fMRI inputs were not yet denoted as \"images\" so far? The narrative can be made more consistent. Perhaps in Section 4.1 or 4.2, authors should also briefly describe the image-like fMRI input data representation scheme in a few words.\n\n- Presentation of quantitative results are also ambigious. Figure 3 is not discussed in text, and achieved accuracies are not readable from the figure. Ablation studies from Section 5.2 are completely separated from the main quantitative result of the paper that is written in the Abstract: 39.34% in 50-way-top-1 semantic classification accuracy. In Section 5.1 or 5.2, it is not clear where this number really comes from? In general, the paper should revise its presentation of quantitative evaluations.\n\n- Methodological comparisons to DC-LDM should be a bit more elaborated. Does this method only differ in the use of a contrastive pre-training phase, or are the encoder network architectures of the 24-layer FRL also different? Can the authors clarify the differences a bit more in depth on how significant their contributions are?\n\n- Authors do not explicitly state this, but is the general idea of contrastively using a masked autoencoder scheme for denoising fMRI data a novel contribution of the paper? If not, can the authors refer to similar works and discuss that it is a powerful approach that is adopted for pre-training in their model?\n\nMinor comments: \n- It appears like in no occasion an fMRI conditioned reconstruction comes out directly identical to the ground truth, and hence it would be probably better/more realistic in Figure 2c to use a different airplane image as generated.\n- What is the \"CMA\" block indicating in Figure 1? Is it the cross-attention module?\n- In Figure 1, the dotted background pattern prevents the equation symbols/text to be easily read. Perhaps this figure text/equations can be re-generated in a better visible way.\n", "limitations": "Sufficiently addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles a very interesting problem with nice, visually illustrated results. Proposed idea of performing neural signal guidance in the latent space such that a generative latent diffusion model can be used for high-resolution image synthesis is quite a neat approach to solve this problem.", "weaknesses": "There is no higher level justification or discussion present in the paper on why one would/should use fMRI for this problem (as opposed to other alternatives) towards building a non-invasive BMI. Moreover from a technical perspective, the paper only puts together several existing methods but some technical details are appearing to be mixed up in the descriptions. These should be corrected and some clarifications are still necessary in terms of the presentation of results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for all the constructive feedback. We answer your questions and comments as follows. (Due to the rebuttal length limit, we summarize your questions.)\n\nQ1: Why fMRI is a better choice over alternatives such as EEG for this problem?\n\nA1: Primarily, fMRI offers significantly higher spatial resolution than EEG. This allows us to capture detailed activations within the visual cortex and related brain areas, a beneficial factor when reconstructing visual images from brain activity. Moreover, the presented stimuli in the dataset are all static images, making the fMRI’s limitation in temporal resolution less concerning. When working with static images, the prolonged temporal scales of fMRI responses provide ample time for the brain to process the stimuli and for the hemodynamic response to unfold. Therefore, considering both fMRI and EEG have inherent noises, fMRI's strength in spatial resolution and the nature of our stimuli make it a more fitting choice than EEG in this work. fMRI is also used in a lot of related work published in top-tier journals, such as Nature[1] and Science[2]. \n(All the following work have been cited in our paper)\n\n[1] K. N. Kay, et al, “Identifying natural images from human brain activity,” Nature, vol. 452, pp. 352–355, 2008.\n\n[2] T. Horikawa et al., “Neural decoding of visual imagery during sleep,” Science, vol. 340, pp. 639 – 642, 2013.\n\nQ2: Why does Figure 2 reference a \"VQGAN\" when the LDM used a pre-trained VQ-VAE? Based on Eq 9, should the notation for the VQ-VAE encoder be E_g instead of E_G?\n\nA2: The VQGAN, at its core, can indeed be perceived as a VQ-VAE optimized with a GAN loss. However, as corroborated by LDM’s original paper [3], the model we adopted specifically leverages a VQGAN. In section 3.2 of the LDM's original paper, it's explicitly stated in the last paragraph, “This model can be interpreted as a VQGAN...”. The open-sourced code of the LDM provides further evidence of discriminative loss implementation. \nAs for the notations, thanks for pointing this out, E_g should be used to denote the VQGAN encoder. We will correct the notation in line 193 in our revised manuscript.\n\n[3] R.Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF Conference on CVPR. 2022. (ref. 24 in our paper)\n\nQ3: About making consistent references to fMRI inputs and adding a brief description of the fMRI representation scheme.\n\nA3: Thanks for the suggestions. As a kind of neuro-imaging, one sample of fMRI recordings can be called an image. But we acknowledge that it might be a little confusing to differentiate between the image of visual stimuli. We will follow your suggestions and unify the naming of fMRI data as fMRI samples in the revision. \nAs for the image-like fMRI representations scheme, we divide the vectorized voxels into patches which will be subsequently transformed into embeddings using a 1D convolutional layer with a stride equal to the patch size (16). We will add these descriptions in Section 4.1. \n\nQ4: Is Figure 3 discussed in the text? How do the ablation studies from Section 5.2 relate to the main quantitative result mentioned in the Abstract? How is the 39.34% accuracy in the Abstract calculated?\n\nA4: Thanks for the questions.\n1. We've indeed referenced Figure 3 within the text. Specifically, Figure 3[a,b,c] is elaborated upon in lines 244-248 and 250-252 of the Results Section 5.1.\n2. Our ablation studies in Section 5.2 align with the main quantitative outcomes. Specifically, the concluding line of our ablation table (Table 2) indicates the hyperparameter setting yielding the 25.080 performance as illustrated in Figure 3[a].\n3. The 39.34% mentioned in the abstract denotes a relative improvement, not an absolute accuracy. This is derived from the formula: (25.080−17.999)/17.999×100%=39.34%, comparing our model's accuracy (25.080) against the DC-lDM's accuracy (17.999). We will directly annotate Figure 3[a] with our accuracy value in the revision for better transparency.\n \nQ5: Does the method's distinction lie solely in the contrastive pre-training phase, or are there other differences in the FRL? How significant are the authors' contributions in relation to these differences?\n\nA5: Thanks for the questions. First, our model's Phase 1 indeed differs from the simple masked auto-encoder (MAE) utilized by DC-LDM by employing a novel double-contrastive MAE. Second, exclusive to our approach is Phase 2, characterized by our Cross Modality Guidance. As elaborated in section 3.2 and illustrated in Figure 1, this phase directs the fMRI encoder towards features crucial for image reconstruction—a facet absent in DC-LDM. Our ablation study, specifically in section 5.2.1, delves into the contributions of these mechanisms to the final performance.\nWe'll ensure these distinctions are more pronounced in the revision to underscore the novelty of our approach.\n\nQ6. Is the general idea of contrastively using a masked autoencoder scheme for denoising fMRI data a novel contribution of the paper? If not, can the authors refer to similar works and discuss that it is a powerful approach that is adopted for pre-training in their model?\n\nA6: Indeed, our adoption of the contrastive masked autoencoder (MAE) scheme specifically for fMRI data denoising represents a novel contribution to neuroimaging. While the synergy of contrastive learning and MAEs has been explored in other domains, its application to fMRI is distinct, given the domain's inherent challenges like high dimensionality and pronounced noise levels. We've substantiated the efficacy of this approach in sections 5.1 (results) and 5.2 (ablation study). In light of your feedback, we'll emphasize this innovation more prominently in our revised manuscript.\n\nResponse to minor comments: Yes, we will use a different plane image in Fig. 2. And the CMA block indeed refers to the cross-attention module. We will remove the dotted background in Fig. 1 in the revision.  "}]

Author rebuttal:

Meta-review: This paper present an approach for reconstructing images from fMRI activity. The approach relies on a framework for learning representations from fMRI that conditions images generated by a diffusion model. The results appear strong and the approach is different from existing reconstruction approaches and would be useful to discuss at NeurIPS. The authors are expected to include their new materials and clarifications in their paper, along with modifying the abstract to make the improvement in performance more clear.
Decision: Accept (poster)