Title: Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural decoding

brain machine interface

mind reader

visual reconstruction

vision decoding
Abstract: Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception. However, the task is challenging due to the noisy nature of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder. The optimized fMRI feature learner then conditions a latent diffusion model to reconstruct image stimuli from brain activities. Experimental results demonstrate our model's superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-art methods by 39.34% in the 50-way-top-1 semantic classification accuracy. The code implementations is available at https://github.com/soinx0629/vis_dec_neurips/.

Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities

Abstract

Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception. However, the task is challenging due to the noisy nature of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder. The optimized fMRI feature learner then conditions a latent diffusion model to reconstruct image stimuli from brain activities. Experimental results demonstrate our model's superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-art methods by 39 . 34% in the 50-way-top-1 semantic classification accuracy. The code implementations will be available at https://github.com/soinx0629/vis_dec_neurips/ .

1 Introduction

Reconstructing visual stimuli from neural imaging data represents a promising interdisciplinary area between cognitive neuroscience and machine learning [1]. A system capable of accurately decoding neural responses to visual input can help illuminate the complex mechanisms underlying the brain's visual perception [2, 3]. Furthermore, it can elucidate the relationships between human visual systems and computational vision models [4, 5, 6]. Such a system also holds the potential to assist patients, particularly those with motor disabilities, in expressing their thoughts and intentions through brain signals.

Functional Magnetic Resonance Imaging (fMRI), as a non-invasive method to measure brain activity, has been extensively utilized to decipher perceptions from neural responses [5, 7, 8]. However,

despite its utility, achieving reliable and robust visual reconstructions from fMRI recordings remains a significant challenge [9]. Primarily, the fMRI data is inherently noisy. The recorded signals encompass not only the specific responses elicited by visual stimuli but also incorporate additional sources of noise arising from various cognitive, physiological processes, and scanner operations [10, 11]. These noises can obscure the neural activation patterns associated with the stimulus, thereby complicating the direct decoding of visual information from the fMRI data. Moreover, the process by which a visual stimulus arouses a neural response is dynamic, intricate and multifaceted. It involves multiple stages of neural processing [12, 13], from the initial perception in the retina to higher-order cognitive operations within the brain's visual and associative regions [14, 15]. The resulting fMRI signal is a highly convolved representation of these distinct processing stages, rather than a linear one [16]. It is thus non-trivial to reverse this process and disentangle the convolved representations to achieve visual decoding.

Existing methods for vision decoding tend to struggle with such challenging complexity. Previous pioneering work has relied on traditional statistical approaches, such as ridge regression, to map fMRI signals back to the corresponding stimuli [17, 18]. However, these oversimplified methods often fall short of capturing the non-linear relationship between the stimulus and the neural responses. More recently, deep learning methods such as Generative Adversarial Networks (GAN) [9, 19] and latent diffusion models (LDMs) [7, 5] have been adopted to model the non-linearity and yield better results. But the difficulty of disentangling vision-related brain activities from noises still hinders these methods from decoding images with optimal accuracy.

To navigate these challenges, we propose a double-phase fMRI representation learning framework. In Phase 1, we pre-train the fMRI feature learner on large-scale unlabeled fMRI data with a novel Double-contrastive Masked Auto-encoder (DC-MAE). The DC-MAE helps discern common patterns of brain activities shared among populations over individual noises. In Phase 2, we further tune the feature learner on the fMRI-image parallel data with an image auto-encoder. The pixel-level guidance from the image auto-encoder teaches the fMRI auto-encoder to attend to brain signals that are most informative for image reconstruction. The trained fMRI feature learner is then used to condition an LDM to reconstruct image stimuli from brain activities. Experimental results demonstrate that the proposed model generates high-resolution and semantically accurate images, outperforming the previous state-of-the-art method by 39 . 34% in 50-way-top-1 accuracy. Our research paves the way for further exploration of the potential of decoding tasks.

2 Related Works

2.1 Visual Decoding from fMRI

Driven by the substantial potential, recent years have witnessed a growing interest in reconstructing visual experiences from fMRI data. This task has been examined in various contexts, including explicitly presented visual stimuli [4, 20], perceived emotions [21] and even imagined content [1]. Though studies on this task keep emerging, the challenges presented by the low signal-to-noise ratio (SNR) of fMRI data and the high complexity of brain visual representations still exist. In the initial stage in this field, efforts to identify or reconstruct visual images from fMRI primarily utilized handcrafted features [17] and traditional regression models [18, 22]. Nonetheless, oversimplified methods can not fully account for the intricate patterns of brain visual representations, generating blurry and semantically meaningless images.

Recent research has shifted towards artificial neural network representations and deep generative models [23]. Typically, such models mapped fMRI signals to image features and fine-tuned pretrained generative models like Generative Adversarial Networks (GANs) [9, 19] or Diffusion Models [7, 24, 25, 26, 27] to generate images from the mapped features. For example, [28] decoded fMRI data to hierarchical image features extracted by a pre-trained VGG and fed the predicted features to a GAN. But [28], like other parallel work [29, 1], used fMRI directly for training and decoding. Without explicitly denoising the fMRI representations, though these works outperformed traditional regression-based models, they still fell short by generating implausible images. In contrast, our framework contains an individual phase to learn denoised fMRI representations with DC-MAE. [7] adopted a naive MAE to learn fMRI representations. But compared with [7], our framework further introduces pixel-level guidance from the image auto-encoder to help disentangle the vision-related neural activities from background noises. Our model is also not similar to other works [30, 31] which

directly map fMRI to the image feature space and inevitably face data incompatibility between the two very different modalities. We encourage the emergence of a cross-modality representation space by guiding the fMRI auto-encoder with a pre-trained image auto-encoder. Experimental results prove the superiority of our methods over these related baselines.

2.2 Diffusion Probabilistic Models

Diffusion probabilistic models have been empirically established as powerful generative models for image generation surpassing GANs [32] in terms of both diversity and fidelity. The diffusion model was first proposed in [33] and further improved by [26, 34]. The quality of generated image could also be enhanced by training-free methods [35, 36]. The initial diffusion models were primarily applied in the pixel space. They achieve impressive success in generating images of high quality, but also suffer from significant drawbacks such as prolonged inference time and substantial training costs [26]. To address these two issues, [27] introduce the latent diffusion model (LDM), also referred to as stable diffusion (SD). This approach adopts a pretrained Vector Quantized Generative Adversarial Network (VQGAN) [37] or Variational auto-encoder (VAE) [38] to construct a latent image space, within which optimization and evaluation are performed. The LDM not only generates images of high quality, but also alleviates the computational burden. Moreover, the incorporation of a cross-attention mechanism within the attention block of the diffusion UNet model permits the LDM model to offer a broad spectrum of controls in image synthesis. This includes textual controls [39, 40, 41], controls over images in various domains [42, 43] such as depth maps, sketch maps, or candy maps. Such versatility and adaptability give the LDM model a substantial advancement in the field of image synthesis.

3 Methods

3.1 Motivation and Overview

In this subsection, we provide a concise analysis of the fMRI data and brain visual representations that motivate the design of our method.

First, the fMRI recordings are inherently noisy, subject to various sources of physiological and scanner-related noises [44]. FMRI records not only responses to visual stimuli but also signals from other cognitive processes. Second, fMRI quantifies changes in the blood-oxygen-level-dependent (BOLD) signal. Adjacent voxels are often found to display similar magnitudes, suggesting the spatial redundancy of fMRI data [2]. Third, neural responses to identical stimuli can exhibit significant divergence [45, 46] across populations.

Considering these analyses altogether, we propose a double-phase fMRI representation learning framework. In Phase 1, we pre-train an MAE with a contrastive loss to learn fMRI representations from unlabeled data. The masking which sets a certain portion of the input data to zero targets the spatial redundancy of fMRI data. The calculation of recovering the original data from the remaining after masking suppresses noises. Optimization of the contrastive loss discerns common patterns of brain activities over individual variances. After pre-training in Phase 1, we tune the fMRI auto-encoder with an image auto-encoder. We expect the pixel-level guidance from the image auto-encoder to support the fMRI auto-encoder in disentangling and attending to brain signals related to vision processing. After FRL Phase 1 and Phase 2, we apply the representation learned by the fMRI auto-encoder as conditions to tune the LDM and generate the image from the brain activities.

3.2 fMRI Representation Learning (FRL)

Phase 1: Pre-training Double-Contrastive Masked Auto-Encoder (DC-MAE) We introduce a method termed the "Double-Contrastive Masked Auto-Encoder". DC-MAE has been specifically designed to pre-train fMRI representations from unlabeled data, as inspired by previous work in visual contrastive learning [47]. As shown in Figure 1, the DC-MAE consists of an encoder E F and a decoder D F . E F takes a masked version of the fMRI signal as the input, and D F is trained to predict the unmasked fMRI. The term "Double-Contrastive" implies that the model optimizes contrastive losses and engages in two separate contrasting processes during the representation learning of an fMRI example.

CA

F

CA

I

fMRI  Encoder

E

F

fMRI  Decoder

D

F

Image Encoder

E

I

Image  Decoder

D

I

FRL Phase 1

FRL Phase 2

𝑣

!

𝑢

!

𝑣

!

"

𝑢

!

"

ℎ

#

!

$

"

ℎ

#

#

%

"

𝑣

!

&

𝑢

&

predict

fMRI  Encoder

E

F

𝑣

!

𝑣

!

"

!

𝑣

!

"

"

fMRI  Encoder

E

F

𝑣

'

𝑣

#

"

!

𝑣

#

"

"

D

F

D

F

fMRI  Decoder

fMRI  Decoder

𝑣

$

%&

"

𝑣

$

%&

!

𝑣

!

𝑣

'

𝑣

$

%&

"

𝑣

$

%&

!

positive pair

negative pair

patch & mask

patch & mask

patch & mask

patch & mask

+

!

Specifically, in the first contrasting, each sample v i ( i ≤ n ) 1 within a batch of n fMRI examples v undergoes two separate instances of random masking. This procedure yields two distinct masked versions of v i , referred to as v m 1 i and v m 2 i . These masked versions establish a positive sample pair for the first contrasting. Subsequently, a 1D convolutional layer, characterized by a stride that matches the patch size, tokenizes v m 1 i and v m 2 i into embeddings. These embeddings are then fed independently into the same fMRI encoder E F . The decoder D F , in turn, receives each of the encoded latent representations as input and generates predictions v dm 1 i and v dm 2 i . The first contrastive loss, denoted as the cross-contrastive loss, is thus calculated through an InfoNCE loss [48] as follows:

̸

In the second contrasting, an unmasked original image v i ( i ≤ n ) and its corresponding masked image v i m form an inherent positive sample pair. Here, v dm i denotes the predicted image output from the decoder D F . The second contrastive loss, referred to as the self-contrastive loss, is computed through as:

̸

̸

Optimizing the self-contrastive loss L S can also achieve mask-reconstruction. For both L C and L S , the negative examples v j ( j = i ) are sourced from the same batch of instances. L C and L S are optimized jointly as follows:

In this equation, the hyper-parameters γ C and γ S regulate the weight of each loss.

Phase 2: Tuning with Cross Modality Guidance Considering the relatively low signal-tonoise ratio (SNR) and highly convolved nature of fMRI recordings , it is important for the fMRI feature learner to attend to the brain activation patterns most relevant for visual processing and most informative for reconstruction.

So as shown in Figure 1, after pre-training in Phase 1, the fMRI auto-encoder is tuned to reconstruct fMRI with the aid of an MAE for image, and vice versa in Phase 2. Specifically, denote one image from a batch of n samples as u i ( i ≤ n ) and the fMRI recorded neural responses to u i as v i . u i and v i go through patching and random masking to be u i m and v i m . u i m and v i m are respectively fed in the image encoder E I and fMRI encoder E F , getting h u i E I = E I ( u i m ) and h v i E F = E F ( v i m ) . For reconstructing the fMRI v i , h u i E I and h v i E F are merged with a cross-attention module as follows:

W and b denote the weights and biases of corresponding linear layers. √ d k is the scaling factor and d k is the dimension of the key vectors. CA is the abbreviation of cross-attention. CA F ( Q u i I , K v i F , V v i F ) is then added to h v i E F and fed into the fMRI decoder to reconstruct v i as v d i .

A similar computation is also conducted in the image auto-encoder. The output h u i E I of the image encoder E I is merged with the output of E F through a cross-attention module CA I , and then used to decode an image u i as u d i :

The fMRI and image auto-encoders are trained jointly by optimizing the following loss:

3.3 Image Generation with Latent Diffusion Model (LDM)

After the fMRI feature learner is trained through FRL Phase 1 and Phase 2, we use its encoder E F to condition a LDM to generate images from brain activities. As displayed in Figure 2, the diffusion model consists of a forward diffusion process and a reverse denoising process. The forward process gradually degrades an image to a normal Gaussian noise by incrementally introducing Gaussian noise of variable variance. This process can be mathematically formulated as q ( x t | x t -1 ) = N ( x t , √ αx t -1 , (1 -α t ) I ) , where t denotes the temporal step and α encompasses predefined noise schedule parameters. During the training phase, the diffusion model, specifically the U-Net [49] model, is optimized with the loss function below to learn the noise ϵ t added at each time step in the forward process.

In the backward process, an image is synthesized by progressively eliminating noise from a randomly initialized standard Gaussian noise. The Latent Diffusion Model (LDM) [27] performs both forward and backward processes in a low-dimensional latent space which is constructed using a pre-trained VQ-VAE model [38]. This approach significantly mitigates the computational complexity while simultaneously maintaining the generated image's quality. We leverage the LDM as the backbone of our image generation model.

Decoding fMRI to a natural image can be seen as conditional image generation task. Given the relatively low SNR inherent in fMRI, combined with the limited quantity of fMRI-to-image data pairs, training an fMRI-to-image generation model from scratch presents substantial challenges. Consequently, the objective of this phase is to harness the fMRI to extract image-related knowledge from a pre-existing conditional image generation model.

Our approach extracts visual knowledge from the pretrained label-to-image LDM to generate image with the conditioning of fMRI. We incorporate the fMRI information into the LDM via a crossattention operation similar to equation (4), as proposed in [27]. To further enhance the guidance provided by the conditional information, following the methodology of previous research [7], we

fMRI  Encoder

Cross Attention

VQGAN  Decoder

𝑧

!

𝑧

"

𝑧

#

D

G

……

……

Forward: Noise Adding

Backward:  Denoising

……

Diffusion UNet

GT Image

Generated Image

𝑧

!$"

𝑧

%

𝑧

%&"

Forward Diffusion Process

VQGAN  Encoder

E

G

𝑧

%

𝑧

!

𝑧

"

𝑧

#

𝑧

!$"

𝑧

%&"

……

……

……

Noise Adding

GT Image

𝐸

!

[a]

[b]

[c]

implement both cross-attention conditioning and time steps conditioning [50]. During training, given an image u and fMRI v , along with VQGAN encoder E G and the fMRI encoder E F (which is trained in FRL Phase 1 and 2), we freeze the LDM and finetune the fMRI encoder using the following loss:

Here, ϕ ( E g ( u )) t = √ α t E g ( u ) + √ 1 -α t ϵ and α t is the noise schedule of diffusion model. During the inference phase, the process begins with a standard Gaussian noise at time step T . The LDM follows the backward process sequentially to progressively denoise the hidden representation, conditioning on the given fMRI information. Upon reaching time step zero, the VQGAN decoder D G is used to convert the hidden representations to an image. The forward and backward process of the LDM are detailed in Figure 2[b,c].

4 Experimental Setup

4.1 fMRI Datasets

HCP The Human Connectome Project (HCP) originally serves as an extensive exploration into the connectivity of the human brain. It offers an open-sourced database of neuroimaging and behavioral data collected from 1,200 healthy young adults within the age range of 22-35 years. Currently, it stands as the largest public resource of MRI data pertaining to the human brain, providing an excellent foundation for the pre-training of brain activation pattern representations. Of the subjects involved, 1113 underwent scanning via a Siemens Skyra Connectom scanner for 3T MR, while a Siemens Magnetom scanner for 7T MR was utilized for the remaining 184. For the scope of this paper, we will predominantly focus on the data derived from the more populated 3T dataset.

GOD The Generic Object Decoding (GOD) Dataset is a specialized resource developed for fMRIbased decoding. It aggregates fMRI data gathered through the presentation of images from 200 representative object categories, originating from the 2011 fall release of ImageNet. The training session incorporated 1,200 images (8 per category from 150 distinct object categories). In contrast, the test session included 50 images (one from each of the 50 object categories). It is noteworthy that the categories in the test session were unique from those in the training session and were introduced in a randomized sequence across runs. On five subjects the fMRI scanning was conducted.

BOLD5000 The BOLD5000 dataset is a result of an extensive slow event-related human brain fMRI study. It comprises 5,254 images, with 4,916 of them being unique. This makes it one of the most comprehensive publicly available datasets in the field. The dataset's principal advantage is its high

diversity, enabling the capture of the complexity and variability inherent in natural visual stimuli. The images in BOLD5000 were selected from three popular computer vision datasets: ImageNet, COCO, and Scenes. ImageNet provided 1,916 images primarily focusing on singular objects. Meanwhile, COCO contributed 2000 images featuring multiple objects, and Scenes contributed 1000 images depicting hand-crafted indoor and outdoor scenes. Four participants labeled CSI1 through CSI4, were involved in this study and underwent scanning via a 3T Siemens Verio MR scanner equipped with a 32-channel phased array head coil.

4.2 Implementation Details

4.2.1 FMRI Representation Learning (FRL)

For both FRL Phase 1 and Phase 2, the fMRI auto-encoder is the same ViT-based masked autoencoder (MAE). We employed an asymmetric architecture for the fMRI auto-encoder, in which the decoder is considerably smaller with 8 layers than the encoder with 24 layers. We used a larger embedding-to-patch size ratio, specifically a patch size of 16 and an embedding dimension of 1024 for our model. We used random sparsification (RS) as a form of data augmentation, randomly selecting and setting 20% of voxels in each fMRI to zero.

FRL Phase 1 In Phase 1, we train the masked ViT-based fMRI auto-encoder with contrastive loss. For GOD subject 1,4,5 and BOLD5000 CSI 1,2, self-contrastive ( γ s ) and cross-contrastive ( γ c ) loss weights are both 1. The masking ratio is 0.5. For GOD subject 2,3 and BOLD5000 CSI 3,4, γ s = 1 and γ c = 0 . 5 , masking ratio is 0.75. We set the batch size to 250 and train for 140 epochs on one NVIDIA A100 GPU. We train with 20-epoch warming up and an initial learning rate of 2.5e-4. We optimize with AdamW and weight decay 0.05.

FRL Phase 2 In Phase 2, we tune the fMRI autoencoder jointly with an image auto-encoder, which is a pre-trained ViT-based MAE released by [51]. The image auto-encoder has a 12-layer encoder with a 768 hidden size and a 6-layer decoder with a 512 hidden size. We set the batch size to be 16 and train for 60 epochs. We train with 2-epoch warming up. The initial learning rate is 5.3e-5. We optimize with AdamW and weight decay 0.05. We freeze the parameters of the decoder of the image-autoencoder and only tune the encoder. The two phases in total take about 12 hours to run. After the two phases, we only save the checkpoint of the fMRI encoder which has 15.16M parameters

4.2.2 Fine-tuning LDM

In this stage, we jointly optimize the parameters of LDM cross-attention heads and the fMRI encoder, while keeping other parameters of LDM unchanged. Given an fMRI-image pair, we first use the pre-trained VQ encoder to encode the image to obtain the latent representation which is further used as an objective to guide the joint training of the fMRI encoder and LDM cross-attention heads. During training, the fMRI data passes through the fMRI encoder trained using FRL, producing a patchified representation. This representation is then projected into key and value representation of cross-attention modules in the UNet of LDM. Furthermore, it is added to the time embedding to conduct double conditioning. The training follows the regular training pipeline of the diffusion model, where the model is optimized to learn to predict the Gaussian noise added to the image latent representation at each time step with the guidance of the given conditioning information. Here, we use the output of the fMRI encoder as the conditioning information. We conduct training with the following parameters: the batch size of 5 , diffusion steps of 1000 , the AdamW optimizer, a learning rate of 5 . 5 e -5 , and an image resolution of 256 × 256 × 3 .

4.3 Baseline Models and Evaluation Metric

Baseline Models We juxtapose our proposed model with recently published benchmarks: ICGAN [52], Self-supervised auto-encoder (SS-AE) [53], and DC-LDM [7]. The IC-GAN is based on an Instance-Conditioned GAN, whereas the SS-AE utilizes cycle consistency and perceptual losses to reconstruct images from fMRI brain recordings. The DC-LDM employs a double-conditioned LDM, demonstrating superior performance on the GOD and BOLD5000 datasets. These benchmarks reflect prevalent methodologies in visual reconstruction.

Evaluation Metric Visual decoding prioritizes semantic consistency. Therefore, our evaluation metric is the n -way topk accuracy, which aligns with the precedent set in the literature [53, 7]. We

employ the pre-trained ImageNet-1K classifier [54] as a semantic correctness evaluator. During the evaluation, both the generated image and the corresponding ground truth image are fed into the classifier. Semantic correctness is then determined based on whether the topk classification among n randomly selected classes aligns with the ground-truth classification. Further details can be referenced in the appendix.

5 Results

5.1 Reconstruction Results

In this section, we present a comparative analysis of our results with preceding studies, which include DC-LDM, IC-GAN, and SS-AE. We have evaluated our model on both GOD and BOLD5000 datasets. Following the setting of previous work [7], we first display in detail the results in Figure 3 for GOD subject 3 and BOLD5000 subject CSI 1 which achieve the best performance in their respective dataset. It is noteworthy that the original DC-LDM implementation utilized test set fMRI data for tuning, a setting not adopted by IC-GAN and SS-AE. To maintain a fair comparison, we first prohibit the use of test set fMRI data by the DC-LDM. As shown in Figure 3[a], our method surpasses the previous models by a large margin. For instance, our model achieves an accuracy exceeding that of DC-LDM and IC-GAN by over 39 . 34% (calculated by (25 . 080 -17 . 999) / 25 . 080 × 100% ≈ 39 . 34% , comparing our model's accuracy 25 . 080 against the DC-lDM's accuracy 17 . 999 ) and 66 . 7% respectively. The improvement of our model over the baselines is significant. All the significant results have p-value < 0.01 with paired t-tests.

GT

IC-GAN

OURS

DC-LDM

SS-AE

GT

OURS

DC-LDM

0

0.1

0.2

0.3

0.4

0.5

0.6

50-way-top-5

100-way-top-5

OURS

DC-LDM

IC-GAN

SS-AE

0

0.05

0.1

0.15

0.2

0.25

0.3

50-way-top-1

100-way-top-1

[a]

[b]

[c]

a1

a2

To further investigate the quality of the images produced by different models, we randomly select 5 samples from the GOD test set and exhibit the generated images in Figure 3[b]. The SS-AE model can only generate a general layout, while the remaining three models are cable of producing images with a semantic meaning similar to that of ground truth image. Our model generates images of superior quality that exhibits a higher degree of semantic consistency with the ground truth images. In Figure 3[c], we compare images generated by our model and DC-LDM. (IC-GAN and SS-AE were not trained on BOLD5000 in their original paper). Our model achieves 50-way-top-1 accuracy of 25 on CSI1. Though both models can produce high-quality images, the image generated by our model bears a semantic meaning more consistent with the ground truth images.

To check if our model may reliably reconstruct brain activities on different subjects, we further evaluate it against DC-LDM on all the other 4 subjects of the GOD dataset. The bar charts in Figure 4[a] below show the results in 50-way-top-1 classification accuracy. Our model substantially outperforms the previous state-of-the-art method (DC-LDM) on all GOD subjects. To achieve DCLDM's reported performance in its original paper [7], this method need signals from test set fMRI

data. To ensure a fair evaluation, we banned DC-LDM from tuning on the test set in the comparison of the Figure 3. But we show here that, our model still largely exceeds DC-LDM on four GOD subjects even after DC-LDM is tuned on the test set fMRI data.

2

4

6

8

10

2

14

16

SS-AE

fMRI-IC-GAN

DC-LDM

DC-LDM-test-tuned

OURS

2

4

6

8

10

12

14

16

GOD Subject1

2

4

6

8

10

12

14

16

18

20

GOD Subject 2

2

4

6

8

10

12

14

16

18

GOD Subject 4

2

4

6

8

10

12

14

16

18

GOD Subject5

[a]

[b]

GOD Subject 1

GOD Subject 2

GOD Subject 4

GOD Subject 5

GT

Ours

GT

Ours

GT

Ours

GT

Ours

5.2 Ablation Study Results

We train the FRL Phase with different settings of hyper-parameters. The trained fMRI encoders then guide the LDM in generating images. We use the 50-way-top-1 accuracy of LDM image generation to measure the influence of hyper-parameter values. Due to space limit, in this section we mainly focus on FRL Phase 2 where direct interaction between the image and fMRI data takes place which may more highly influence the final reconstruction performance.

Reconstruction Loss Weight In the FRL Phase 2, the feature learner is tuned by optimizing the joint loss of fMRI and image reconstruction, as in equation (7). So we first focus on the influence of the fMRI and image reconstruction loss weights, namely γ F and γ i . The results are reported in Table 2 (ID1-6). In experiment ID 1 and 2, we set one of the weights to 0 to evaluate the necessity of using the corresponding loss. We find that jointly optimizing the fMRI and image reconstruction loss is necessary to achieve optimal task performance. We adjust the γ F and γ i and find that setting in ID 6 where γ F = 0 . 25 and γ i = 1 . 5 yields the highest performance.

Mask Ratio In the FRL Phase 2, both the fMRI and image auto-encoders are based on MAE. The mask ratio of the input is an important setting for these models. We demonstrate the influence of the mask ratio setting in Table 2 (ID 4 and ID 7-10). We find that applying a higher mask ratio on fMRI data and a lower mask ratio on image data generally leads to better performance. LDM conditioned by the fMRI encoder which is trained with an fMRI mask ratio of 0.75 and image mask ratio of 0.5 leads to the highest reconstruction accuracy.

Decoder Layers Following [7, 51, 55], for both the fMRI and the image auto-encoder, we build asymmetric architectures where the decoder is much smaller than the encoder. In Table 2 (ID 11-13), we report the results of tuning decoder depth. We apply the same depth for the two decoders. We find that a moderate decoder depth of 6 produces optimal results. Neither a too shallow nor a too deep decoder improves reconstruction performance on GOD subject 3.

6 Discussion

The experimental findings indicate that, with the proposed fMRI representation learning framework and a pre-trained LDM, we can achieve a degree of visual reconstruction of human brain activities, substantially outperforming baselines. Nonetheless, our analysis also uncovers some limitations within our model. Primarily, our model appears to be slightly affected by a categorical bias issue. We hypothesize that this may stem from the inherent bias present in the dataset used to train the LDM. Additionally, while our model demonstrates proficiency in capturing high-level semantics, it sometimes fails to reconstruct specific details of an image. A plausible explanation could be the concurrent imagination of multiple objects by the participants during data collection, which inevitably results in a noisy fMRI feature. Contrasting with general image generation, which is typically characterized by a focus on diversity, visual decoding underscores the importance of consistency, thereby necessitating a reduction in bias during the generation process. As such, the exploration of techniques to minimize the influence of data bias, as well as methods to enhance the reconstruction of image details when generating images from fMRI data, would hold significant academic value and interest.

7 Conclusion

In this work, we propose a double-phase fMRI representation learning framework (FRL) with an LDM to reconstruct visual experiences from brain activities. The FRL denoises fMRI features by contrastive masked modeling in Phase 1. And in Phase 2 it learns to disentangle and attend to brain activation patterns most informative for visual decoding with the guidance of an image MAE. With the conditioning of the optimized fMRI feature learner, we show that an LDM generates images of high quality, largely outperforming the previous state-of-the-art. Extensive ablation studies further verify the effectiveness of each component that we propose.

8 Ethical Statements

We used preprocessed data from publicly available datasets. The fMRI data that we train with have been processed and do not contain any data that can be directly linked to the participants' identities. The collection procedure of the fMRI undergoes strict ethical review as stated in their original paper.

References

T. Horikawa and Y. Kamitani, 'Generic decoding of seen and imagined objects using hierarchical visual features,' Nature Communications , vol. 8, 2015.

K. U˘ gurbil, J. Xu, E. J. Auerbach, S. Moeller, A. T. Vu, J. M. Duarte-Carvajalino, C. Lenglet, X. Wu, S. Schmitter, P. F. Van de Moortele et al. , 'Pushing spatial and temporal resolution for functional and diffusion mri in the human connectome project,' Neuroimage , vol. 80, pp. 80-104, 2013.

E. W. Contini, S. G. Wardle, and T. A. Carlson, 'Decoding the time-course of object recognition in the human brain: From visual features to categorical decisions,' Neuropsychologia , vol. 105, pp. 165-176, 2017.

Z. Ren, J. Li, X. Xue, X. Li, F. Yang, Z. Jiao, and X. Gao, 'Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning,' NeuroImage , vol. 228, 2021.

Y. Takagi and S. Nishimoto, 'High-resolution image reconstruction with latent diffusion models from human brain activity,' bioRxiv , pp. 2022-11, 2022.

J. Sun, M. Li, and M.-F. Moens, 'Decoding realistic images from brain activity with contrastive self-supervision and latent diffusion,' in Proceedings of the 26th European Conference on Artificial Intelligence (ECAI 2023), 2023 , 2023.

Z. Chen, J. Qing, T. Xiang, W. L. Yue, and J. H. Zhou, 'Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding,' in arXiv , November 2022. [Online]. Available: https://arxiv.org/abs/2211.06956

J. Sun and M.-F. Moens, 'Fine-tuned vs. prompt-tuned supervised representations: Which better account for brain language representations?' in Proceedings of IJCAI , Macau, China, 2023.

M. Mozafari, L. Reddy, and R. van Rullen, 'Reconstructing natural scenes from fmri patterns using bigbigan,' 2020 International Joint Conference on Neural Networks (IJCNN) , pp. 1-8, 2020.

T. T. Liu, 'Noise contributions to the fmri signal: An overview,' NeuroImage , vol. 143, pp. 141-151, 2016.

J. C. Brooks, O. K. Faull, K. T. Pattinson, and M. Jenkinson, 'Physiological noise in brainstem fmri,' Frontiers in human neuroscience , vol. 7, p. 623, 2013.

I. I. Groen, E. H. Silson, and C. I. Baker, 'Contributions of low-and high-level properties to neural processing of visual scenes in the human brain,' Philosophical Transactions of the Royal Society B: Biological Sciences , vol. 372, no. 1714, p. 20160102, 2017.

D. J. Kravitz, K. S. Saleem, C. I. Baker, L. G. Ungerleider, and M. Mishkin, 'The ventral visual pathway: an expanded neural framework for the processing of object quality,' Trends in cognitive sciences , vol. 17, no. 1, pp. 26-49, 2013.

T. C. Kietzmann, C. J. Spoerer, L. K. Sörensen, R. M. Cichy, O. Hauk, and N. Kriegeskorte, 'Recurrence is required to capture the representational dynamics of the human visual system,' Proceedings of the National Academy of Sciences , vol. 116, no. 43, pp. 21 854-21 863, 2019.

J. J. DiCarlo, D. Zoccolan, and N. C. Rust, 'How does the brain solve visual object recognition?' Neuron , vol. 73, no. 3, pp. 415-434, 2012.

J. Sun, S. Wang, J. Zhang, and C. Zong, 'Neural encoding and decoding with distributed sentence representations,' IEEE Transactions on Neural Networks and Learning Systems , vol. 32, no. 2, pp. 589-603, 2020.

K. N. Kay, T. Naselaris, R. J. Prenger, and J. L. Gallant, 'Identifying natural images from human brain activity,' Nature , vol. 452, pp. 352-355, 2008.

T. Horikawa, M. Tamaki, Y. Miyawaki, and Y. Kamitani, 'Neural decoding of visual imagery during sleep,' Science , vol. 340, pp. 639 - 642, 2013.

K. Seeliger, U. Güçlü, L. Ambrogioni, Y. Güçlütürk, and M. van Gerven, 'Generative adversarial networks for reconstructing natural images from brain activity,' NeuroImage , vol. 181, pp. 775785, 2017.

Y. Zhang, T. Bu, J. Zhang, S. Tang, Z. Yu, J. K. Liu, and T. Huang, 'Decoding pixel-level image features from two-photon calcium signals of macaque visual cortex,' Neural Computation , vol. 34, pp. 1369-1397, 2022.

T. Horikawa, A. S. Cowen, D. Keltner, and Y. Kamitani, 'The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions,' iScience , vol. 23, 2019.

A. G. Huth, T. Lee, S. Nishimoto, N. Y. Bilenko, A. T. Vu, and J. L. Gallant, 'Decoding the semantic content of natural movies from human brain activity,' Frontiers in Systems Neuroscience , vol. 10, 2016.

T. Fang, Y. Qi, and G. Pan, 'Reconstructing perceptive images from brain activity by shapesemantic gan,' ArXiv , vol. abs/2101.12083, 2021.

M. Ferrante, T. Boccato, and N. Toschi, 'Semantic brain decoding: from fmri to conceptually similar image reconstruction of visual stimuli,' arXiv preprint arXiv:2212.06726 , 2022.

F. Ozcelik and R. VanRullen, 'Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion,' arXiv preprint arXiv:2303.05334 , 2023.

J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in Neural Information Processing Systems , vol. 33, pp. 6840-6851, 2020.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684-10 695.

G. Shen, K. Dwivedi, K. Majima, T. Horikawa, and Y. Kamitani, 'End-to-end deep image reconstruction from human brain activity,' Frontiers in computational neuroscience , vol. 13, p. 21, 2019.

G. Shen, T. Horikawa, K. Majima, and Y. Kamitani, 'Deep image reconstruction from human brain activity,' PLoS Computational Biology , vol. 15, 2017.

X. Qian, Y. Wang, Y. Fu, X. Xue, and J. Feng, 'Semantic neural decoding via cross-modal generation,' arXiv preprint arXiv:2303.14730 , 2023.

S. Lin, T. Sprague, and A. K. Singh, 'Mind reader: Reconstructing complex images from brain activities,' arXiv preprint arXiv:2210.01769 , 2022.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial networks,' Communications of the ACM , vol. 63, no. 11, pp. 139-144, 2020.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, 'Deep unsupervised learning using nonequilibrium thermodynamics,' in International Conference on Machine Learning . PMLR, 2015, pp. 2256-2265.

A. Q. Nichol and P. Dhariwal, 'Improved denoising diffusion probabilistic models,' in International Conference on Machine Learning . PMLR, 2021, pp. 8162-8171.

M. Li, T. Qu, W. Sun, and M.-F. Moens, 'Alleviating exposure bias in diffusion models through sampling with shifted time steps,' arXiv preprint arXiv:2305.15583 , 2023.

F. Bao, C. Li, J. Zhu, and B. Zhang, 'Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models,' arXiv preprint arXiv:2201.06503 , 2022.

P. Esser, R. Rombach, and B. Ommer, 'Taming transformers for high-resolution image synthesis,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 12 873-12 883.

A. Van Den Oord, O. Vinyals et al. , 'Neural discrete representation learning,' Advances in neural information processing systems , vol. 30, 2017.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes et al. , 'Photorealistic text-to-image diffusion models with deep language understanding,' arXiv preprint arXiv:2205.11487 , 2022.

G. Kim, T. Kwon, and J. C. Ye, 'Diffusionclip: Text-guided diffusion models for robust image manipulation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 2426-2435.

N. Ruiz, Y. Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman, 'Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,' arXiv preprint arXiv:2208.12242 , 2022.

L. Zhang and M. Agrawala, 'Adding conditional control to text-to-image diffusion models,' arXiv preprint arXiv:2302.05543 , 2023.

C. Mou, X. Wang, L. Xie, J. Zhang, Z. Qi, Y. Shan, and X. Qie, 'T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models,' arXiv preprint arXiv:2302.08453 , 2023.

T. B. Parrish, D. R. Gitelman, K. S. LaBar, and M.-M. Mesulam, 'Impact of signal-to-noise on functional mri,' Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine , vol. 44, no. 6, pp. 925-932, 2000.

G. K. Aguirre, R. Datta, N. C. Benson, S. Prasad, S. G. Jacobson, A. V. Cideciyan, H. Bridge, K. E. Watkins, O. H. Butt, A. S. Dain et al. , 'Patterns of individual variation in visual pathway structure and function in the sighted and blind,' PLoS One , vol. 11, no. 11, p. e0164677, 2016.

J. Sun, S. Wang, J. Zhang, and C. Zong, 'Towards sentence-level brain decoding with distributed representations,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 7047-7054.

X. Chen, S. Xie, and K. He, 'An empirical study of training self-supervised vision transformers,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9640-9649.

A. v. d. Oord, Y . Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' arXiv preprint arXiv:1807.03748 , 2018.

O. Ronneberger, P. Fischer, and T. Brox, 'U-net: Convolutional networks for biomedical image segmentation,' in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 . Springer, 2015, pp. 234-241.

P. Dhariwal and A. Nichol, 'Diffusion models beat gans on image synthesis,' Advances in Neural Information Processing Systems , vol. 34, pp. 8780-8794, 2021.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, 'Masked autoencoders are scalable vision learners,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 000-16 009.

F. Ozcelik, B. Choksi, M. Mozafari, L. Reddy, and R. VanRullen, 'Reconstruction of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans,' in 2022 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2022, pp. 1-8.

G. Gaziv, R. Beliy, N. Granot, A. Hoogi, F. Strappini, T. Golan, and M. Irani, 'Self-supervised natural image reconstruction and large-scale semantic classification from brain activity,' NeuroImage , vol. 254, p. 119121, 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020.

Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, 'Simmim: A simple framework for masked image modeling,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 9653-9663.

Appendix

A.1 Reconstruction Performance

We provide the performance of our model on BOLD5000 subjects 1, 2, 3, and 4 in Table A.1. Following previous work [7], all results are presented in 50-way-top-1 classification accuracy.

A.2 Examples of Reconstructed Images

Figures A.2 and A.3 present images generated by our model using fMRI data from GOD and BOLD5000 datasets, respectively. We generated all images at a resolution of 256 × 256 × 3 using 250 PLMS steps. More samples can be generated using our code base in the supplementary materials. The code will be open-sourced with the camera ready version of this paper.

A.2.1 Reconstructed Images from GOD Dataset

GOD Subject 1

GOD Subject 2

GOD Subject 4

GOD Subject 5

A.2.2 Reconstructed Images from BOLD5000 Dataset

BOLD5000 CSI 1

BOLD5000 CSI 2

BOLD5000 CSI 3

BOLD5000 CSI 4

A.3 Evaluation Metrics

We use the common N-trial, n-way top-1 semantic classification as the main evaluation metrics. This evaluation method is summarized in the algorithm below:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on decoding visual stimuli from recorded fMRI activity. Proposed method initially pre-trains an fMRI feature learner (and essentially a signal denoiser) on unlabeled data using a contrastive training scheme resembling to masked autoencoders. Subsequently this feature encoder is finetuned by using guidance from an image auto-encoder to attend to signal patterns that are informative for reconstruction. Output of this fMRI feature learner is then used to condition a latent diffusion model to be able to visually reconstruct the stimuli in the image domain with high resolution.\n\nSTRENGTHS:\nThe paper tackles a very interesting problem with nice, visually illustrated results. Proposed idea of performing neural signal guidance in the latent space such that a generative latent diffusion model can be used for high-resolution image synthesis is quite a neat approach to solve this problem.\n\nWEAKNESSES:\nThere is no higher level justification or discussion present in the paper on why one would/should use fMRI for this problem (as opposed to other alternatives) towards building a non-invasive BMI. Moreover from a technical perspective, the paper only puts together several existing methods but some technical details are appearing to be mixed up in the descriptions. These should be corrected and some clarifications are still necessary in terms of the presentation of results.\n\nQUESTIONS:\n- Motivation of the study from a non-invasive brain-machine interface perspective is somewhat disputable, since the neurophysiological source signal considered here is fMRI. Can the authors extensively discuss and justify to the reader why their choice of fMRI (with low temporal resolution and longer temporal scales of responses, noisy nature, and hard-to-integrate experimental recording/design) would be a better candidate than other alternatives (e.g., EEG, or (partly-)invasive methods) to tackle this problem?\n\n- Figure 2 with the \"VQGAN\" does not make sense. Starting from the last paragraph of page 5 and then in Figure 2, the paper talks about a VQGAN, whereas the LDM that the authors use should certainly be a pre-trained VQ-VAE with an encoder-decoder pair? Also based on the notation in Eq 9 and line 193, I presume the VQ-VAE encoder should be E_g rather than E_G on line 191?\n\n- Description of Phase 1 has some ambiguities. Beginning of Sec 3.2 was understandable. Then line 140 starts to talk about an unmasked original \"image\" at some point, whereas the fMRI inputs were not yet denoted as \"images\" so far? The narrative can be made more consistent. Perhaps in Section 4.1 or 4.2, authors should also briefly describe the image-like fMRI input data representation scheme in a few words.\n\n- Presentation of quantitative results are also ambigious. Figure 3 is not discussed in text, and achieved accuracies are not readable from the figure. Ablation studies from Section 5.2 are completely separated from the main quantitative result of the paper that is written in the Abstract: 39.34% in 50-way-top-1 semantic classification accuracy. In Section 5.1 or 5.2, it is not clear where this number really comes from? In general, the paper should revise its presentation of quantitative evaluations.\n\n- Methodological comparisons to DC-LDM should be a bit more elaborated. Does this method only differ in the use of a contrastive pre-training phase, or are the encoder network architectures of the 24-layer FRL also different? Can the authors clarify the differences a bit more in depth on how significant their contributions are?\n\n- Authors do not explicitly state this, but is the general idea of contrastively using a masked autoencoder scheme for denoising fMRI data a novel contribution of the paper? If not, can the authors refer to similar works and discuss that it is a powerful approach that is adopted for pre-training in their model?\n\nMinor comments: \n- It appears like in no occasion an fMRI conditioned reconstruction comes out directly identical to the ground truth, and hence it would be probably better/more realistic in Figure 2c to use a different airplane image as generated.\n- What is the \"CMA\" block indicating in Figure 1? Is it the cross-attention module?\n- In Figure 1, the dotted background pattern prevents the equation symbols/text to be easily read. Perhaps this figure text/equations can be re-generated in a better visible way.\n\nLIMITATIONS:\nSufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to decode the visual stimuli from neural responses by reverse-mapping the signals from functional MRI (fMRI) to the images the participants see while being scanned. The authors claimed to achieve this through a two-phase framework. In the first phase, they pre-train an fMRI feature learner inspired by mask auto-encoder (MAE) from unlabeled fMRI data. This phase tries to discern the common patterns and features in fMRI across participants responding the same stimuli. In the second phase, they further tune the fMRI feature learner with cross-modality correspondence from the visual stimuli, i.e., the images. This process is similar to the cross-modality guidance commonly used in vision language models. Finally, after the two phases of training, the fMRI feature learner is used to generate signals that serve as conditions for a latent diffusion model (LDM) for image generation.\n\nThe authors demonstrated that the final model is able to produce images from the participants’ fMRI signals such that the generated images well correspond — in category — to what the participants see. On the quantitative benchmarks, the proposed method outperform the baselines by a significant margin.\n\nAs much as I appreciate this work, I would like to demystify it a bit. In my opinion, it’s effectively performing two tasks sequentially: (1) fMRI signal classification into 50 or 100 classes, and (2) class-conditional image generation. The holistic framework and storytelling might have made it more impressive than it actually is — though admittedly it is still impressive.\n\nSTRENGTHS:\n1. “Mind reading” is a very bold topic. Generating the image in a participant’s mind from the fMRI recordings is arguably a reasonable way.\n\n2. The design with three stages: contrastive pre-training, cross-modality fine-tuning and latent diffusion model is very reasonable and well conveyed.\n\n3. The ablation studies are presented neatly.\n\nWEAKNESSES:\n1. The experimental results, though impressive, do not seem very comprehensive. For example, in Figure 3, qualitative results are shown for two datasets while bar plots are only displayed for one.\n\n2. Also, I would recommend showing some intermediate results — for example, directly evaluating the fMRI representation accuracy by training a simple model to classify the learned fMRI representations after phase 2.\n\nQUESTIONS:\n1. Do you think it is reasonable to present the intermediate fMRI representation accuracy as described in Weaknesses 2? If I understand it correctly, you have $image_{real} \\rightarrow (human) \\rightarrow fMRI \\rightarrow representation \\rightarrow image_{gen}$, and you are showing $acc(classifier(image_{gen}), label(image_{real}))$. I wonder if you can also show $acc(classifier'(representation), label(image_{real}))$ and even $acc(classifier'(representation), classifier(image_{gen}))$.\n\nLIMITATIONS:\nNothing came to my mind.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposed to decode visual stimuli from neural responses recorded by fMRI. First, it pretrains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. Second, it tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder.

STRENGTHS:
(1) The method is straightforward and easy to understand.
(2) The related work discussion seems comprehensive.

WEAKNESSES:
(1) The selection of baseline methods for comparison appears to be insufficient, thus hindering the ability to effectively demonstrate the effectiveness of the proposed method. To enhance the evaluation, it would be beneficial for the authors to include a comparison with the methods mentioned in the related work section.

(2) Another crucial factor that warrants consideration is the quality of the fMRI data, as it can significantly impact the performance of the model. It would be valuable if the authors could provide some ablation studies pertaining to this aspect as well.

(3) The evaluation metric solely employed is n-way top-k accuracy, which could be supplemented with additional evaluation metrics to provide a more comprehensive assessment. Metrics such as FID (Fréchet Inception Distance), SSIM (Structural Similarity Index), MSE (Mean Squared Error), among others, would be advantageous for a more qualitative evaluation.

(4) Since human data is used in the study, the authors can discuss some potential ethical concerns.

QUESTIONS:
Please see the comments above.

LIMITATIONS:
The limitation is discussed in Section 6, however, since human data is used in the study, the authors can discuss some potential ethical concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for decoding visual stimuli from neural responses (fMRI images). The rationale behind the proposed approach lies in the difficulty of learning the complex relationship between a stimuli and the neural responses to it, and the noisy nature of fMRI images.\n\nThe authors propose a multiple phase approach comprising a contrastive learnign based method for learning denoised representations of fMRI brain activities (a new DC-MAE method is proposed), a feature learner that combines previsous representation space with image representation through  cross attention and a diffusiuon model to reconstruct image stimuli from brain activities which is conditioned on the opuput of the feature learner.\n\nSTRENGTHS:\nThe approcah looks like a smart agregation of state of the art components that allows reaching high top-k accuracy (the chosen metric).\n\nThe paper is easy to read and the methodological choices are well motivated. \n\nThe method is validated by top k accuracy and achieves significantly better results than the baselinesit is compared to. \n\nA thourough ablation study shows the impact of the various components and of the hyperparameters of the method.\n\nWEAKNESSES:\nFew aspects of the method are not enough detailed (see questions)\n\nQUESTIONS:\nTo which space does v_i belongs, is is a time series ?  I don’t understand the use of 1D conclutional models that maps v_i^m_1  and v_i^m_2 into embeddings.\n\nIn Phase2 are all models (E_F, E_I, D_F, D_I) retrained or are only the cross attention layers trained ?\n\nThe way the latent diffusion model is conditionned seems new. Hwo does this compare to previous attempts of designing conditional latent diffusion models ?\n\nConcerning metrics it is said in section 4.3 « We employ the pre-trained ImageNet-1K classifier [50] as a semantic correctness evaluator. » wht does this mean ?  \n\nConcerning the top-k accuracy experiments did the authors use a statistical test to attest the significativity of the result ?\n\nLIMITATIONS:\nlimitations are addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

comments: SUMMARY:
The paper propose a novel two-phase fMRI representation learning method to encode the visual stimuli from neural responses. It is a significant challenging task due to noisy fMRI signals and complex intricate pattern of brain visual representation. The proposed two-phase can reconstruct image stimuli from brain activities.

STRENGTHS:
1.The overall framework is novel and interesting. The use of double-contrastive mask auto-encoder and image-guided auto-encoder are novelty.
2.The introduction is clear.
3.The superiority experimental results are obtained.

WEAKNESSES:
1.The motivation of two phase is confused. Why two phase representation learning is needed? And what is the difference between the two phase?
2. How the contrastive MAE works? It is not clear
3.The model may be too big? How the efficiency?
The experiments are weak. How to set the mask? The model’s size? More quantification is needed.

QUESTIONS:
see above

LIMITATIONS:
the proposed method is too complex to follow.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25