LEA: Learning Latent Embedding Alignment Model for fMRI Decoding and Encoding
Authors: Xuelin Qian
Yikai Wang
Xinwei Sun
Yanwei Fu
Xiangyang Xue
Jianfeng Feng
Venue: Accepted by TMLR ()

Abstract: The connection between brain activity and visual stimuli is crucial to understanding the human brain. Although deep generative models have shown advances in recovering brain recordings by generating images conditioned on fMRI signals, it is still challenging to generate consistent semantics. Moreover, predicting fMRI signals from visual stimuli remains a hard problem. In this paper, we introduce a unified framework that addresses both fMRI decoding and encoding. We train two latent spaces to represent and reconstruct fMRI signals and visual images, respectively. By aligning these two latent spaces, we seamlessly transform between the fMRI signal and visual stimuli. Our model, called Latent Embedding Alignment (LEA), can recover visual stimuli from fMRI signals and predict brain activity from images. LEA outperforms existing methods on multiple fMRI decoding and encoding benchmarks. It offers a comprehensive solution for modeling the relationship between fMRI signals and visual stimuli. The codes are available at \url{https://github.com/naiq/LEA}.

Published in Transactions on Machine Learning Research (12/2024)
LEA: Learning Latent Embedding Alignment Model for fMRI Decoding and Encoding
Xuelin Qian
aueling02@gmail.cmm com
School of Data Science Fudan Universily
Yikai Wang"
yi-kai. Canaalaaaagoolaaa Soutlook.coooo com
School Data Science
Pudon University
Xinwei Sun
School of Data Science Pudan University
Yanwei Fu'
yorweifuu@fudan.ddu. edu.cc
School of Data Science Fudan University
Xiangyang Xue
zyguu Witdeen edu.crr
School of Computer Science Pudan University
Jianfeng Feng
jionfeng64 @gmaail com
Institute of Science and Technology for Bruin-inspired Intelligence
Pudan University
Reviewed on OpenReview: https: // openreview. net/ forum? id= 89QT2DsKy)
Abstract
The connection between brain activity and visual stimuli is crucial understanding the human brain. Although deep generative models have shown advances in recovering brain recordings by generating images conditioned on fMRI signals, it is still challenging to gen- erate consistent semantics. Moreover, predicting fMRI signals from visual stimuli remains a hard problem. In this paper, we introduce a unified framework that addresses both fMRI decoding and encoding. We train two latent spaces to represent and reconstruct fMRI signals and visual images, respectively. By aligning these two latent spaces, we seamlessly transform between the fMRI signal and visual stimuli. Our model, called Latent Embedding Alignment (LEA), can recover visual stimuli from fMRI signals and predict brain activity from images. LEA outperforms existing methods multiple fMRI decoding and encoding benchmarks. Tt offers a comprehensive solution for modeling the relationship between (MRT signals and visual stimuli. The codes are available at https: httts:///tthheii //tg::/bgttttttt cotiuu//.coo/aii//AA....
1 Introduction
The human brain reacts dynamically to visual stimulus from the eyes (Teng Kravitz 2019). We can indirectly measure these responses using functional Magnetic Resonance Imaging (IMRT). Identifying and categorizing distinct patterns of brain activity in response to visual stimuli is crucial step in understanding
*:Euull contribution. f Corresponding author.

---

Published in Transactions on Machine Learning Research (12/2024)
2017 Horikawa & aamitani 2017); reconstructing the visual stimulus from the [MRT signal. Since the brain may not capture all the details in an image, an exact reconstruction is not necessary. Researchers are mainly focused on docoding the semantics of images (Horikawa & Kamitani 2017 Shen et al.) 2019 Beliy et al. 2019 Gaziv
the human brain. One method of achieving this is inverse modeling (Parthasarathy et al.
eet al. 2022). On the other hand, IMRI encoding tries to predict the [MRI signal from visual stimulus. It is important to note that semantic information in fMRI data is spread out and closely linked between nearby elements (Chang et al. 2019). In real-world situations, there are not always many pairs of images and signals available. This makes it difficult to use deep learning methods effectively, as they often require large datasets to understand IMRT data.
Recent researches (Zzellik et al. 2022
Ferrante et al. 2022 Chen et al. 2022 Liu et al. 2023 Du et al. 2023
Ozcelik & VanRullen, 2023) decode fMRI signals based on pre-trained generative models such as Instance-
Conditional GAN (Casanova et al.
2021 diffusion models (Ho et al.] 2020), masked autoencoders (He
2021). J, to name a few. Despite achieving impressive results in high- fidelity generation, they lace several challenges: (1) Despite their ability to generate images of exceptional quality, pre-trained generative models struggle to ensure semantic consistency with fMRI signals, presenting an ongoing challenge. (2) These models have demonstrated the capability to produce high-quality images even when provided with random noise disguised as fake fMRI signals (see Sec. for details). This raises concerns about their reliability, particularly when dealing with a wide range of visual stimuli, including those not seen during training (out-of-distribution) This reliability issue is particularly relevant to fMRI decoding. Thus when utilizing large-scale pre-trained generative models for fMRI decoding, distinguishing between genuine [MRT signal decoding and unconditional image generation becomes critical concern.
ct al.) 2022), CLIP (Radford et al.,
This paper proposes novel approach to address the challenges associated with fMRI signals by simultane- ously tackling the tasks of fMRI decoding and encoding. Our approach designs encoder-decoder architectures specifically for both fMRI signals and images, avoiding the need for paired image-fMRI data. These encoder- decoder structures effectively Iearn dense and compact latent representation spaces. As a result, our model achieves the best performance on IMRI zero-shot classification where the calegories of [MRI signals are unseen during training. Moreover, these encoder-decoder architectures can be trained via self-supervised learning techniques (Vincent et al., 2008 He ct al. 2020; 2022). climinating the need for additional supervi- sion. Particularly, we introduce the ROI embedding layer to tackle the individual mismatch of fMRI signals, enabling the pre-train on multi-subjects without manual alignment of different subjects. Furthermore, this learned representation serves to reduce the feature dimension of both modalities, thus simplifying the task of transformation between them.
Technically, to align the two learned representation spaces effectively, we propose a lightweight, alignment module that can be trained with a relatively small set of fMRI-image pairs. Aligmment across diverse types of signals has been extensively explored in various vision-related tasks, such as image-text (Radford et al.,
2021 Jia et al., 2021 Mahajan et al. 2018) and image-audio (Arandjelovic & Zisserman 2017 Morgado Owens & Efros| 2018 Patrick et al. 2020). These studics have revealed that well-encoded signals from different modalities can be effectively aligned within a shared representation space. However, given the absence of a large-scale fMRI-image paired dataset to support end-to-end training of such an alignment model, inspired by the linear relationships observed between text- and vision-only models (Merullo et al. 2023), we employ a lincar regression model to learn the mapping between the two latent spaces. Furthermore, this lightweight linear model can be trained with small number of IMRI-imaee pairs, eliminating the need for large-scale paired dataset and enabling the creation of individuul-speccfee models. Within the framework of LEA, the alignment module serves a crucial role in assessing the reliability of both fMRI encoding and decoding models through and su subsequently reconstructing the visual stimuli, the ability to generate semantically consistent images serves as a reliability indicator.
ct al.
reconstruction. By initially estimating fMRI signals from images
Consequently, we have devised the Latent Embedding Alignment (LEA) framework for fMRI decoding and encoding. LEA encompasses two distinct encoder-decoder architectures for handling fMRI signals and images, respectively. We introduce an alignment module to facilitate the transformation between the latent representation spaces of fMRI signals and images. As illustrated in Figure [ when reconstructing visual stimuli from fMRI signals, we encode the fMRI signal, convert it into a latent image embedding, and
2

---

Published in Transactions on Machine Learning Research (12/2024)
Visual Stimail
Visual Stimali
Visual Stiaall
Excale
Encode
Excode
Enoole
MMn
Decade
Deeele
Predicted IMRI
IMRI Signals
Reconstruct Image
IMR Signals
Reconstruct Image
Prodicted IMRI
(a) fMRI Encoding
(b) fMRI Decoding
(c) Image-fMRI-lmage Reconstruction
Our proposed LEA efficiently enables (a) [MRT encoding to estimate the brain activity from visual stimuli; (b) fMRI decoding to recover the visual stimuli from recorded fMRI signals, and (c) image- [MRI-image reconstruction to ensure the reliability of [MRI decoding and encoding.
Figure 1:
subsequently decode it to generate images. Conversely, when predicting brain activity, we encode the images, transform them into latent [MRI embeddings, and then decode them to produce IMRI signals. An important feature of LEA is its support for image-fMRE fMRL-image reconstruction, which serves as an indicator for the reliability of both fMRI encoding and decoding models. Through extensive experimentation BOLD5000 and GOD beuchmarks, we demonstrate that LEA exhibits both efficiency and effectiveness in [MRI decoding and encoding.
Contribution. Our key contributions can be summarized as follows:
Encoder-Decoder Architectures: We introduce specialized encoder-decoder architectures that learn latent representations for fMRI signals and images without the need for extensive paired training data. ROI Embedding Layer: We devise an innovative ROI embedding layer to extract [MRI signals, ensuring geometric consistency in feature extraction. Additionally,
propose the use of global representations for IMRI signal reconstruction, promoting efficient feature compression.
Latent Space Alignment: We establish a linear connection between the latent spaces of fMRI and images, enabling seamless fMRI decoding and encoding within a unified framework called LEA. Superior Performance: Through rigorous testing on multiple benchmark datasets, we demonstrate the exceptional performance of LEA, underscoring its effectiveness and significant advancements in the field of [MRI decoding and encoding.
2
Related Work
MMRI Encoding and Decoding. (MRT encoding involves learning the mapping from visual stimuli
[MRI signals. Early studies (Yamins et al. 2013 2014 Güçlü & van Gerven 2015 Yamins & DiCarlo 2016) primarily employed convolutional neural networks to extract semantic information from visual stimuli. However, with the resurgence of neural language models, recent research has successfully utilized models
such as BERT (Devlin et al. 2018), Transformer (Vaswani et al. 2017), and GPT-2 (Radford et al. 2019 to
predict fMRI responses from stimuli, including images and even /ondds/ennteces. Conversely, fMR decoding
focuses on reverse mapping, from brain activity to stimuli. Previous studies (Mozafari et al. 2020 Ozcelik
et al. 2022 Chen et al. 2015: 2016 [Zhang et al.] (016)) developed regression models specifically designed extract valuable information from IMRI signals. Ozcelik et al. (2022) employed pre-trained Instance- Conditional GAN model (Casanova et al. 2021 to reconstruct images by decoding latent variables from fMRI data. Ferrante al. (2022) utilized pre-trained latent diffusion models (Ho et al. 2020 to generate stimulus images by mapping EMRI signals to visual features. [Beliy et al. (2019) adopted self-supervised learning on testing data to enable adaptation to testing statistics. Chen el. al. (2022) introduced a self- supervised sparse masked modeling strategy to encode fMRI data into latent embeddings and fine-tuned latent diffusion models with double conditioning, Huo et al. (2024) further ensures that the decoded images retain both the semantics and stucture. In this paper, we present a unified framework that efficiently realizes both [MRI encoding and decoding.
3

---

Published in Transactions on Machine Learning Research (12/2024)
fMRI Reconstruction
Rol Embhed
Rol Predict
Transformer Encoder Layer
Transformer Decoder Layer
1PA
[MR Signals
Token Embedding
Tokens
Reconstruct Signals
Image Reconstruction
Inference
fMRI Decoding
Image Feature
Encoder
Decoder
Image Encoder (CLIP)
Image Decoder
stimuli
Regression
Encoder
Decoder
Stimuli Image
Reconstruct Image
[MRI Encoding
Figure 2: The overview of proposed LEA framework. We learn latent representation space via the encoder-decoder architectures designed specifically for fMRI signals and images. Then lightweight align- ment module is proposed to link the two latent spaces, enabling both fMRI encoding and decoding.
Multi-Modalitty Alignment. It aims to establish connections across different modalities, such as image-
text alignment (Radford et al.] 2021 Jia et al., 2021 Mahajan et al. 2018) and image-audio align-
ment (Arandjelovic Zisserman, 2017 Morgado et al. 2021 Owens & Ffros 2018 Patrick et al. 2020).
Girdhar al. (2023) endeavors to align multiple modalities within unified latent representation space.
The majority of efforts in this direction are centered around the CLIP model (Radford et al. 2021 ), which
is trained extensive image-text pairs to learn shared latent space minimizing the distances between correlated image-text pairs. For instance, Liu et al. (2023) bridged the modality gap by leveraging CLIP's cross-modal generalization ability to address the limitation of limited IMRI-image paired data. Similarly, Du et al. 2023 employed multi-modal deep generative models to capture the relationships between brain, visual, and linguistic features. Ozcelik & VanRullen (2023) predicted multi-modal features from fMRI signals and generated reconstructed images using latent diffusion model (Rombach et al., 2022)
3 Methodology
3.1
Problem Setup
Given a IMRT signal f L recorded from brain activity and the corresponding visual stimuli Hx the purpose of our work is learn unified framework that can perform both fMRI decoding task recover the observed image from [MRI signal and fMRI encoding lask to predict the brain activity from the image.
For the decoding task, the idcal scenario involves the reconstructed images matching the real ones preciscly. However, the unique nal ure of biological mechanisms suggests that individual memory or attention can influence an individual's brain activity response to the same image, and thus variations may occur. In light of this, we adhere to the approach introduced in (Mozafari et al. 2020 Ozcelik al., (022). requiring the recovered images to exhibit semantic consistency with Consequently, our primary focus on the semantic, JMRT decoding task.

---

Published in Transactions on Machine Learning Research (12/2024)
For the encoding task, it is challenging to make precise predictions of element-wise fMRI signals due to their inherent noise and redundancy. Therefore, in this paper, our focus shifts towards predicting the overall trends in fMRI signals. More specifically, we aim to generate estimated fMRI signals that exhibit a correlation with the actual fMRI signals. This correlation serves as an indicator of whether a Region-Of-Intere (ROI) region is activated or not. To achieve this goal, our objective is to produce estimated [MRT signals based on the visual stimulus, that is highly correlated with the actual fMRI signals.
3.2 Latent Space Construction for fMRI
The fMRI signals indirectly record neural activity in the visual cortex of the brain by measuring the blood- oxygen-level-deeender (BOLD) signals, which are usually represented as data of 3D voxels, covering visual regions, like early visual cortex (EVC), lateral occipital complex (LOC), occipital place area (OPA), parahip- pocampal place area (PPA), and retrosplenial complex (RSC) (Horikawa & Kamitani] 2015). Biological principles suggest that adjacent voxels in the brain's 's visual cortex often exhibit similar intensitics in their stimulls-responsee (Ugurbil et al. 2013 resulting in redundancy within IMRI signals.
To investigate potential response patterns to visual stimuli and establish a reverse mapping for semantic encoding, it is essential to focus on the relationships among different regions while simultancously capturing activity changes across long-range voxels. Inspired by the advantages of long-range receptive fields found transformer layers, we follow Chen et al. (2022) by adopting masked autoencoder (He al.) 2022) as the encoder-decoder model for IMRI signals.
ROI embedding layer. Unlike Chen et al. (2022) which equally divides the [MRT signals into patches of equal size, we maintain the voxel structure with ROI regions. Concretely, we partition the fMRI signal vector into distinct ROI regions, each potentially having different dimensions. To address the dimension inconsistency, we propose the Region of Interest (ROI) Embedding layer. Given a specific ROI region i, denote the corresponding fMRI signal is of size N, x 1, indicating N, voxels and 1 dimension of BOLD response, this MMRI signal region is first passed through a convolution layer with 32 kernels to extract multi- head features, resulting in dimensions of N; x 32. Subsequently, we employ fully-connected layer to project the fMRI signal, which may have varying lengths, into a unified dimension of 1024. We then concatenate the embeddings from all ROI regions, yielding the final input of size 32Mx1024, where M indicates the number of ROTs Finally, we append a learnable [CLS] Loken to represent the "class" of the input.
Global feature extraction. In our encoder-decoder architecture training, we employ a masked auto- encoder (MAE) (He al., (002).. The original MAE employs patch-based reconstruction, where a portion of the patches is masked, and the task to predict these masked patches. While this approach effective for learning local representations, it presents challenges when aim to acquire global latent code that represents the entire [MRI signal. To address this, only preserve the global [CLS] token of the encoded fMR signals, mask all the patch-based based latent code, and let the decoder learn to reconstruct fMRI signals from the learned [CLS) token. This strategy proves advantageous for learning a dense fMRI representation within the [CLS] token, and the trained decoder can subsoquently be used to generate fMRI signals basod on the estimated latent IMRI code.
Finally, the decoder's output undergoes post-processioe through the ROI Project layer. This layer mirrors the structure of the ROI Embed layer, effectively reversing the pre-processing applied to the fMRI signals, ensuring that the length matches that of the original [MRI signals. The overall model is trained by minimizing the L2 reconstruction loss of the fMRI signals.
Limri
[f. f.lle.
where f, and f. indicate the ground-truth and reconstructed fMRI signals, individually; K means the totally amount of training [MRI data.
5

---

Published in Transactions on Machine Learning Research (12/2024)
3.3 Latent Space Construction for Image
The image-based latent space construction is well-established domain of research. To recover semantic content from fMRI signals, we leverage the pre-trained CLIP visual encoder as our foundation for image latent space. CLIP benefits from extensive training on large-scale image-text paired data, making its latent space rich in semantics and benefits the semantic consistency in our task.
Note that CLIP operates as an encoder-only model, and we cannot directly generate images from the image latent space. To address this issue, we adopt MaskGIT (Chang al. 2022) as our decoder to reconstruct images from the image latent space. In this approach, we fix the CLIP visual encoder and fine-tune the class-conditional MaskGIT as latent-conditiona generative decoder. Further details regarding the objective functions can be found in CCaag et al. 2022).
3.4 Latent Embedding Alignment
The alignment of fMRI and image latent spaces represents critical step in enabling the simultancous execution of IMRI decoding and encoding lasks. Unlike conventional methods (Du et al.) 2023: Liu et al., 2023) that rely on intermediary modalities to bridge the gap between fMRI and image data, our approach directly connects the latent feature representations of both domains through a linear model. This strategy leverages the expressive power of these well-defined latent spaces, climinating the need for additional data modalities.
To address the inherent variability in fMRI signals and the potential for individual-speccfic patterns in latent fMRI embeddings, we em- ploy a personalized modeling approach. Individuul-speecffic linear models are trained to map fMRI embeddings to image embeddings and vice versa. L2 regularization, implemented via ridge regression, is used to stabilize the learning process and constrain the solution space. With the aligned latent spaces and the embedding align- ment module in place, we integrate IMRI decoding and encoding tasks into a unified framework.
Specifically, the latent embedding of the fMRI is the trained [CLS] token of our IMRI encoder output, denoted as eMRRI- The latent embedding of the image is the global visual CLIP feature, denoted as Eimage- For a specific individual, we gather all training data form two embedding matrices, EMRI and Emmage- Then we learn two linear mappings via the ridge regression such that:
Figure 3: LEA aligns the latent space of [MRI signals and image features, enabling the bi-directional transformation.
Cimage 3
eIRII Cinage Bimage-IMRI IMRI-
(2)
The solutions are fiMRI-siaaee E FMR EMRRI + AT) E Emmage and Aimage-AMMRI (Einage Emmage XI) - Emlngge EIMRI respectively. We use the same regularization parameter À for both regression models.
In the decoding task, the [MRT signal is first encoded into the latent [MRI space. This latent representa- tion is then projected into the latent visual feature space using the individual-speeific linear model. The projected visual feature is decoded by a fine-tuned MaskGIT model to reconstruct the original visual stim- uli. Conversely, in the encoding task, CLIP visual features are encoded and projected into the latent fMRI space through another linear model. The resulting latent fMRT representation is subsequently decoded to approximate the original fMRI signals.
4 Experiments
4.1 Experiment Setup
Datasets. We validate our LEA on Brain, Object, Landscape Dataset (BOLD5000) (Chang et al. 2019)
and Generic Object Decoding Dataset (GOD) (Horikawa & Kamitani 2017 (1) BOLD55000 encom passes
6

---

Published in Transactions on Machine Learning Research (12/2024)
substantial fMRI-image dataset with total of 5254 fMRI-image stimulus trials, involving four subjects. This dataset includes 1000, 2000, and 1916 images sourced from the Scene, COCO, and ImageNet datasets, respectively. Following the protocol in (Chen et al. 2022). we choose 4, 803 images presented on a single trial for training, and the remaining 113 images for testing. We use the pre-defined five ROI regions (e.g., EV, LOC, OPA, PPA, RSC) for our ROT Embed layer. (2) The GOD comprises data from subjects, each viewing 1,250 images spanning 200 categories, resulting a total of 1,250 IMRI-image pairs. We split the training set with 1200 pairs from 150 categories, and the other non-overlapping 50 classes are used for testing. The visual cortical ROI regions include V1, V2, V3, V4, FFA, PPA, LOC, and HVC.
Metrics. For the IMRI decoding task, we utilize the following metries for comprehensive comparisons: N-way Classification Accuracy (Acc) (Gaziv et al. 2022) to assess the semantic correctness of generated samples, Fréchet Inception Distance (FID) (Heusel et al.) 2017) to evaluate the quality of generated images, and CLIP correlation and distance to gauge the semantic consistency of generated images. For Acc we calculate the average top-1 accuracy over 1,000 trials. For FID we use 64-dimensional features extracted by the Inception-v3 model to measure their similarity to ground-truth testing images. The CLIP distance means the cosine similarity of two images and we average the similarity among all fMRI-image pairs. Meanwhile, CLIP correlation calculates the correspondence of positive fMRI-image pairs compared to others, i.e., 2-way comparison of CLIP features via pair-wise correlation. For the fMRI encoding task, we employ Pearson's 's metric to assess the similarity between estimated fMRT signals and the ground truth. We calculate the Pearson correlation (x10²) on each vertex of the N x L matrix, where N represents the number of fMRI signals and L is the dimension, and then average these correlations across different vertices.
Setting. We follow the general principle of not using any testing data in training. We denote our algorithm in this setting as LEA. Additionally, for a fair comparison with competitors, we follow the previous work (Chen ct al. 2022 Beliy et al.) 2019) and use the fMRI test set without revealing the label information. Specifically, the fMRI testing data is only trained for the reconstruction task in self-supervised manner. When learning the latent embodding alignment module, we do not access any paired fMRI-image testing data, and the parameters of both [MRI and image reconstruction modules are frozen. These experiments are distinguished by adding the suffix of "relax" to the model name, as utilized in SSIMRI2IM (Beliy et al. 2019) and MinD-V -ivs (Chen et al. 2022).
Implementation. LEA is implemented using Py The IMRI reconstruction model is initialized with pre-trained weights from HCP datasets |Chen et al. 2022), but we randomly initialize parameters that do not match the dimensions of those from pre-trained weights (e.g., ROI Embed layer). We further fine-tune it for each individual. The encoder and decoder have a depth of 24/8 with dimensions of 1024/512, and there are 16 multi-heads. For fine-tuning both models, we use the AdamW optimizer with hyper-parameters $1= = 0.9, 81 = 0.95, and batch size of 8. The initial learning rate is set to 5e-5 with a weight decay of 0.01. We apply a linear learning rate schedule, gradually reducing the learning rate until it reaches the minimum value. The total number of training iterations is 100K and 300K for the fMRI and image reconstruction models, respectively.
4.2 Quantitative Results
Analysis of zero-shot classification. As we inherit the latent space from the original CLIP model, LEA enables open-vocabulary recognition of fMRI signals. To assess this capability, we perform experiments on zero-shot IMRT signal classification tasks to identify the most similar category for each IMRT signal. Specifically, we create a set of candidate class names using various text templates. These class names are then encoded using the CLIP text encoder. Subsequently, we calculate the similarity between the class embeddings and the fMRI latent embeddings learned by LEA.
We conduct experiments on GOD dataset, which contains 50 visual stimuli from 50 categorics that are unseen during training. We report the top-1 classification accuracy (with the chance levels of 2%) of several competitors in Table [ Results of competitors are from Liu et al. (002)). We divide the competitors into two groups basod on their supervision modality, where "V" indicates visual supervision only and "V&T" indicates both visual and textual supervision. Our LEA trained with visual supervision beats all competitors, indicating the superiority of our bi-directional transformation.
7

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Top-1 accuracy of zero-shot classification on the GOD dataset.
METHON
MODALITY
SBJ-1
SBJ-2
SBJ-3
SBJ-4
SBJ-5
AVERAGE
CADA- VAE [Schonfeld et
2019
V&T
6.31
6.45
17.74
12.17
7.45
10.02
MVAE Wu & Goodman
2018
V&T
5.77
5.40
17.11
14.02
7.89
10.04
MMVAE (Shi el. al. 2019
V&T
6.63
6.60
22.11
14.54
8.53
11.68
MoPoE-VAE (uttter et al.,
2021
V&T
8.54
8.34
22.68
14.57
10.45
12.92
BrainCLIP-Lineal (iu et al.
2023
V&T
10.00
12.00
18.00
12.00
12.00
12.80
CADA-VAE (Schonfeld
2019
5.66
6.01
16.51
9.17
6.01
8.67
MVAE Wu & Goodman 2018
5.30
5.21
14.13
8.03
5.44
7.62
MMVAE (Shi ct al. 2019
V
5.41
5.39
13.76
10.62
5.02
8.04
MoPoE-VAl VAE (Sutter et al.
2021
V
5.20
7.42
14.05
9.25
6.37
8.46
BIaVL. (Du et al._ 2023)
V
8.91
8.51
18.17
14.20
11.02
12.16
BrainCLIP Liu et al. 2023
6.00
16.00
16.00
14.00
14.00
13.20
LEA
V
10.00
18.00
12.00
16.00
12.00
13.60
Table 2: The comparison of (MRT encoding with other competitors. All methods pre-trained CLIP RRad-
ford et al. 2021 to extract image features. The metric is the Pearson correlation coefficient (x10²).
GOD
BOLD5000
METHOD
Sbj-1
Sbj-2
Sbj-3 Sbj-4 Sbj-5
Avg. CSI-1 CS1-2 CS1-3 CS1-4 Avg.
MinD-Vis Vis
Chen et al. 2022
5.21
12.05
15.47
10.63
10.55
10.78
30.58
19.78
16.48
19.96
21.70
Regressor Gifford et al. 2023
6.65
16.22
18.58
12.82
14.38
13.73
32.07
20.58
20.46
21.18
23.57
LEA (relax)
8.43
15.51
21.20
19.08
15.07
15.86
34.32
26.74
27.74
24.44
28.19
LEA
8.49
17.01
20.58
17.82
16.86
16.15
34.78
25.91
27.86
24.37
28.80
Analysis of fMRI encoding. With two latent spaces for fMRI signals and images, LEA not only recover the visual stimuul but also predict the fMRI signal by the visual image. As in Table 2 we conduct experiments to validate the estimation capability of our LEA. We construct two competitors to show the effectiveness of LEA. For the "Regressor", we apply liner regression to directly generate fMRI signals based on the image feature from the CLIP model. With the absence of learned fMRI embedding in LEA, the "Regressor" adopts the pipeline of 'image features - IMRT signals' instead of 'image features - [MRI features - fMRI signals'. However, such method causes misalignment of features during reconstruction without the fMRI latent space, validated in the empirical results. The "MinD-Vis" utilizes the transformer decoder to predict fMRI signals from CLIP image features. Such solution is too rough and not conducive to fine-grained feature alignment, and generation. Our LEA achieves the best Pearson's 's score, which indicates that the alignment between latent embedding spaces is more effective for fMRI prediction.
Analysis of fMRI decoding. We present a comprehensive comparison of fMRI decoding performance on the GOD and BOL.50000 datasets in Table and Table 4 respectively. To ensure a fair assessment, we reproduce the results of competitors using their officially released models. It is worth noting that due to the availability of the official model of CIS-1,4 the BOLD5000 dataset in MinD-Vis, the reproduced CLIP scores for these two individuals. Our novel LEA consistently demonstrates superior performance across various perspectives, including image quality, semantic correctness, and correlation, on both the GOD and BOLD5000 datasets. Notably, LEA excels in generating images of higher quality than its competitors on the GOD dataset. While MinD- Vis may outperform LEA on specific subjects, it significantly lags behind LEA on others, such as SUB-1,2,4 in the GOD and CSI-2,4 in BOLD5000, resulting in an overall inferior average performance. In summary, LEA not only outperforms current competitors in fMRI decoding but also possesses the capability for (MRI encoding, making comprehensive solution in the field.
exclusively report
8

---

Published in Transactions on Machine Learning Research (12/2024)
Table 3: fMRI decoding performance comparison on the GOD dataset.
METRICS
МETHOоS
SiJ-1
SBJ-2
SBJ-3
SBJ-4
SBJ-5
AVERAGE
Gaziv et al. (2022)
7.67
2.67
2.51
8.93
2.77
4.91
Ozcelik et al. (002
5.22
3.72
3.65
4.42
3.69
4.14
Vodrahalli
2018
3.61
2.77
2.34
2.95
2.55
2.84
FID
MinD-Vis CChn al.
2022
1.97
1.63
1.68
1.77
2.33
1.88
LEA (relaz)
1.45
1.43
1.52
1.48
1.25
1.23
LEA
2.57
1.51
1.67
1.71
1.34
1.76
Gaziv al. 2022
58.57
62.00
67.59
62.89
61.14
62.44
Ozeclik al 2022
74.02
80.83
77.65
75.44
76.46
76.88
Vodrahalli et al.
2018
69.55
76.38
79.57
75.34
75.48
75.26
CLIP Corr. f
MinD-Vis Chen et al.
2022)
66.69
77.02
83.78
74.78
80.04
76.46
LEA (relax)
73.96
78.20
80.94
78.73
70.86
76.54
LEA
73.79
79.63
79.96
76.69
76.32
77.28
Gaziv et al. (2022
0.31
0.34
0.34
0.32
0.32
0.33
Ozcelik et al (2022)
0.39
0.40
0.42
0.39
0.41
0.40
Vodrahalli et al.
(2088
0.35
0.39
0.41
0.39
0.40
0.39
CLIP Dist. f
MinD-Vis is Chen et. al.
2022
0.33
0.35
0.43
0.38
0.39
0.38
LEA (relaz)
0.38
0.40
0.41
0.40
0.37
0.39
LEA
0.38
0.41
0.40
0.39
0.41
0.40
Gaziv et al. (022
1.88
3.79
12.99
8.45
6.26
6.67
Ozcelik et al. (2022)
10.71
15.95
18.79
14.52
12.31
14.46
Vodrahalli ot al.
2018
9.51
16.48
18.79
13.35
13.28
14.28
Accuracy f
MinD-Vis CCen al.
2022
9.10
15.91
27.44
15.81
14.28
16.51
LEA (relaz) LEA
11.18
18.62
20.45
20.04
13.15
16.69
11.35
18.55
18.92
17.52
14.45
16.16
Table 4: IMRI decoding performance comparison on BOLD5000 dataset.
METRICS
METHODS
CSI-1
CST-2
CST-3
CSI-4
AVERAGE
MinD-Vis (Chen et al.
2022
1.20
1.90
1.40
1.32
1.46
FID
LEA (relax)
1.31
1.53
1.48
1.19
1.38
MinD- Vis (Chen et al.
2022
88.29
83.09
CLIP Corr. t
LEA (velax)
86.60
83.56
83.82
84.77
84.69
MinD-Vis (Chen et al. LEA (relar)
2022
0.41
0.37
CLIP Dist.
0.41
0.38
0.37
0.39
0.39
MinD-Vis (Chen et al.
2022
29.94
18.50
21.00
20.37
22.45
Accuracy
LEA (relar)
28.48
19.74
19.65
22.48
22.59
4.3 Qualitative Results
To intuitively illustrate the efficacy of our method, show some generated images in Figure and Figure 5 for the GOD and BOLD5000 datasets, respectively. We compare with MinD-Vis (Chen et al. 2022 and Gaziv ct al. 2022) on the GOD and MinD-Vis on BOLD5000.
More concretely, our generated images consistently maintain the same semantic information as the ground truth, encompassing humans, objects, animals, architecture, and landscapes. contrast, competitors may falter in some instances, failing to preserve semantic consistency. For instance, Figure 4 LEA excels producing semantic-consistenn and high-fidelity images, whereas competitors struggle to preserve semantics. In the last on the left side of Figure MinD-Vis only generate person while neglecting the water,

---

Published in Transactions on Machine Learning Research (12/2024)
GT
Ours
Chen (2023) Ozcelik, (2022) Gaziv (2022)
Ours
(2023) Ozcelik (2022) Gaziv (2021)
Figure 4: fMRI decoding performance comparison the GOD dataset.
GT
MinD-Vis
Ours
GT
MinD-Vis
Ours
RA
Figure 5: fMRI decoding performance comparison on BOLD5000 dataset.
whereas LEA consistently generates both the person and the water. These results highlight our ability to not only maintain the correctness of objects but also capture finer-grained semantics. Moreover, our approach achieves image fidelity in numerous cases, contributing to a more comprehensive and effective solution.
4.4 Ablation Studies
Analysis of ROI Embedding Layer. Human brains are known to be spatially segregated, and different ROI regions react to different visual stimuli. Preserving the spatial structure beneficial to help understand the IMRT signals. Therefore, we design ROT embedding layer (<ROTEmbed" for short) to separately encode signals from different ROI regions. Differently, previous mothods (Chen et al., (022) simply divide fMRI signals into n 16 equal parts to obtain sequence features of appropriate length. Zero padding is added if the original length of fMRI signals is not multiple of 16. We denote such method as "PatchEmbed"" To evaluate the effectiveness of our design, we conduct experiments by predicting fMRI signals from visual images and generating stimuli images from fMRI signals. In Table 5 and Table 6 our proposed ROIEmbed achieves better performance than PatchEmbed, which demonstrates its efficacy.
For the ROIEmbed layer, we extract multi-head features for each ROI region. Here, the number of multi- head is a hyper-paranee parameter. Intuitively, if the number of heads too small (i.e., 1), may lead to poor representation capacity; By contrast, larger number of heads may result in information redundancy. To
10

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Ablation study of varying the number of heads in the ROI embedding layer for fMRI decoding task. We report the classification accuracy of CSI-1 BOLD5000 dataset. For PatchEmbed, 16 is the best choice reported in (Chen et al. 2022
SIzE
METHOD
8
16
32
64
96
PatchEmbed CChen et al. 2022 ROLEmbed (Ours)
26.38
24.75
24.35
26.69
28.48
23.96
26.08
Table 6: Ablation study of varying the number of heads in the ROI embedding layer for fMRI encoding task. Here, we report Pearson correlations between the ground truth and the predicted [MRI signals CSI-1 on BOLD5000 dataset.
SIzE
METHOD
8
16
24
32
64
96
PatchEmbed (Chen et al. 2022 ROLEmbed (Ours)
32.88
30.72
30.44
31.82
34.32
29.67
31.04
further investigate the effect of this parameter, we report results with different values of heads in the second row of Table 5 and Table 6 As expected, it shows trend of first rising and then falling. The best result is achieved when the number of heads is 32. Note that do not tune it for each individual but set it to 32 consistently to show the efficacy and generalizability of our proposed layer.
Analysis of Reliable fMRI decoding. Ensuring the accuracy of fMRI decoding models is paramount, as they should decode brain activity rather than merely generate images without meaningful context. However, current approaches often rely on large pre-trained generative models capable of high-fidelity image generation, even when conditioned on random noise. This raises concerns about whether such high-quality images truly reflect brain activity, especially in zero-shot scenarios where the IMRT signals were not part of the training data. To assess the reliability of fMRI decoding models, we conduct two experiments:
Zero-shot Smage-JMRI-imagg reconstruction: In this task, we use a novel image to estimate the IMRT signals, and then use the estimated fMRI signals reconstruct the image. Successful Maag--MMR--maae reconstruction implies better semantic consistency for encoding and decoding. fMRI-image with random noise fake fMRI signals: In this task, randomly generate fMRI signals from a Gaussian distribution, which significantly deviates from the real fMRI signal distribution. We then decode these fake MMR signals to generate images. The reconstructed image should contain non-semantic information, indicating non-informative [MRI signals.
It is important to note that these tasks are not intended to demonstrate the superiority of model new tasks but rather to assess the reliability of methods designed for visual decoding. As shown in Figure D LEA excels in the image-iMRllimage reconstruction task while generating unrealistic and non-informative images when conditioned on fake fMRI signals that deviate from the distribution of real fMRI signals. contrast, MinD-Vis still generate high-fidelity and informative images when conditioned on random noise. For in-distribution sampling, both methods perform well bocause fMRI signals with similar distribution have already been seen in the training set.. Nevertheless, for out-of-distribution sampling, is unclear whether the noise in brain signals is really so-called noise unknown perceptions.
Under this circumstance, methods equipped with powerful image generators still produce high-quality and meaningful images, which we think is unreliable. One possible reason is that a large-scale pre-trained image generator can handle the situation of ditt---iistiitttion (or zero-shot) inputs. However, they are hard to evaluate because we don't know if the input fMRI signal really cxists or what the ground-truth image On the other hand, our generated images, though scemingly meaningless, are not complete noise. The model hasn't 't seen these distributions but appears to combine some known semantics, like sky, lawn, and
11

---

Published in Transactions on Machine Learning Research (12/2024)
Minh-
Qurs
BCC J0T 1EC BOCT
(a) Zero-shot maag--MRR-lmmag reconstruction on GOD dataset.
In-boorbatinn Samepingg
MinB-V
Duael
2-2000 I)
7-
Ours
(b) Neural decoding from randomly sampled nolses as fake fMRI signals.
Figure 6: Reliable fMRI decoding analysis.
house, These findings underscore that existing models have not thoroughly explored the reliability of fMRI decoding models. comparison, both the fMRI decoding and encoding components LEA demonstrate greater reliability than current algorithms.
5 Broader Impact
Our proposed LEA model provides a unified framework, to efficiently recover visual stimuli from fMRI signals and predicts brain activity from images. could serve as tool to analyze human perception and uncover underlying cognitive mechanisms. We hope it can inspire more valuable and innovative studies in the future, and promote the development of neuroscience. However, the potential negative impact lies in that the usage of this method may cause privacy breaches, since people don't want others to pry into their thoughts. Besides, this technology shouldn't be used illegally for commercial or cognitive strategies, As a result, governments and officials may take action to govern the use of relevant datasets and technologies, and researchers should avoid using datasets that may raise ethical concerns.
6 Conclusions
In this paper, we address the challenge of jointly performing fMRT decoding and encoding within a single, unified framework. Our proposed Latent Embedding Alignment (LEA) not only constructs latent spaces for fMRI signals and images but also aligns them, enabling bidirectional transformation. By mitigating issues related to redundancy, instability, and data scarcity in MMRI datasets, T.F.A excels at producing high-fidelity, semantically consistent fMRI decoding results. Moreover, LEA demonstrates the capability to accurately estimate human brain activity from visual stimuli through [MRI encoding. Our experimental results on two benchmark datasets validate the effectiveness of LEA, highlighting its significant contribution in neuroimaging.
7 Acknowledgement
The computations in this research were performed using the CFFF platform of Fudan University. Yanwei Fu is with the School of Data Science, FudanISTBI ZJNU Algorithm Centre for Srain-inspired Intelligence, Fudan University, Shanghai Key Lab of Intelligent Information Processing, and Technology Innovation Center of Calligraphy and Painting Digital Generation, Ministry of Culture and Tourism, China. Yanwei Fu is the corresponding author.
12

---

Published in Transactions on Machine Learning Research (12/2024)
References
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617, 2017.
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pix- els and back: Sell-supervision nalural-image reconstruction from finri. Advances Neural Information Processing Systems, 32, 2019.
Aranixa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano. Instance- conditioned gan. Advances in Neurol Information Processing Systems, 34:27517 27529, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New TEFF, 2022.
Orleans, LA, USA, June 18-24, 2022, pp.
Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. Bold5000, a public fmri datasct while viewing 5000 visual images. Scientific data, 6(1):49, 2019.
Po-Hsuan Chen, Xia Zhu, Hejia Zhang, Javier Turek, Janice Chen, Theodore L Willke, Uri Hasson, and Peter Ramadge. A convolutional autoencoder for multi-subject fmri data aggregation. arXie preprinl arXiv:1608.0448 2016.
Po-Hsuan Cameron Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge. A reduced-dimennion fini shared response model. Advances in neural information processing systems, 28, 2015.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Condi- tional diffusion model with sparse masked modeling for vision decoding. Xie preprinl or 0rxix:2211.00 1..09956, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint 1xXi::8800.04800 2018.
Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding visual neural representations by mul- timodal learning of brain-visualllnnguisti features. IEEF Transactions Pattern Analysis and Machine Intelligence, 2023.
Matteo Ferrante, Tommaso Boccato, and Nicola Toschi. Semantic brain decoding: from fmri to concepl.ully similar image reconstruction visual stimuli. CoRR, abs/2212.0677 2022.
Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Trani. Self-supervised natural image reconstruction and large-scale semantic classification from brain activity, Newrolmage, 254:119121, 2022.
Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascolles, Aude Oliva, Kendrick Kay, Gemmna Roig, and Radoslaw M Cichy. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. arXie preprint orXiv:2301.08198, 2023.
Rohit Girdhar, Alaaelin El-Nouby, Zhuang Lin, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra, Imagebind: One embedding space to bind them all. arXiv preprint 3rXiv:2305.0566 2023.
Unut Güçlü and Marcel AJ van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations ions across the ventral stream. Journal of Neuroscience, 35057):10005-1000 10014, 2015.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances neural infor- mation processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierar- chical visual features. CoRR, abs/1510.06479, 2015.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding seen and imagined objects using hicrar- chical visual features. Nature communications, 8(1):15037, 2017.
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi-level modulation. In European Conference on Computer Vision, pp. 56 73. Springer, 2024.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual- linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXio preprint arXiv:2302.1297. 2023.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview www.eeeeeeeneeeeeeeeeeeeeee (ttte/oorrrmmdddssttRoooobos
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modas agreement. In Proceedings of the TEEE/CVF Conference Compuler Vision and Pallern Recognition, pp. 12475 12486, 2021.
Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstructing natural scenes from fmrr patterns using bigbigan. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, pp. 1 8.8. IEEE, 2020.
Andrew Owens and Alexci A Efros. Audio-visual scene analysis with self-supervised multisensory features.
Tn Proceedings of the European Conference on Compuler Vision (ECCV) pp. 631-648, 2018.
Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXio:2303.055 05334, 2023.
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of perceived images from furr patterns and semantic brain exploration using instanco-conditioond gans. In International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022 pp. 1-8. IEFF, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Nikhil Parthasarathy, Eleanor Batty, William Falcon, Thomas Rutten, Mohit Rajpal, EJ Chichilnisky, and Liam Paninski. Neural networks for efficient bayesian decoding of natural images from retinal neurons. Advances in Neural Information Processing Systems, 30, 2017.
Mandela Patrick, Yuki Asano, Polina Kuznetsova, Ruth Fong, Joao F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. openreview.net 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 188):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition,, pp. 10684-10695, 2022.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8247 8255, 2019.
Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in computational neuroscience, 13:21, 2019.
Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems, 32, 2019.
Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Generalized multimodal clbo. arXie preprint NXiv:2205.02270, 2021.
Chunyue Teng and Dwight J Kravitz. Visual working memory directly alters perception. Nature human behaviour, 3(8):827-88) 2019.
Kåmil Ugurbil, Junqian Xu, Edward J. Auerbach, Steen Moeller, An T. Vu, Julio Martin Cuarte-Carvajalinn Christophe Lenglet, Xiaoping Wu, Sebastian Schmitter, "ierreFFrançol van de Moortele, John P. Strupp, Guillermo Sapiro, Federico De Martino, Dingxin Wang, Noam Harel, Michael Garwood, Liyong Chen, David A. Feinberg, Stephen M. Smith, Karla L. Miller, Stamatios N. Sotiropoulos, Saåd Jbabdi, Jesper L. R. Andersson, Timothy Edward John Behrens, Matthew F. Glasser, David C. Van Essen, and Essa Ya- coub. Pushing spatial and temporal resolution for functional and diffusion MRI in the human connectome project. NeuroImage, 80:80-104, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-5999999 00088600,,2000 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoin Manzagol. Extracting and compos- ing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1108, 2008
Kiran Vodrahalli, Po-Hsuan Chen, Yingyu Liang, Christopher Baldassano, Janice Chen, Esther Yong, Christopher Honey, Uri Hasson, Peter Ramadge, Kenneth A Norman, et al. Mapping between fmri responses to movies and their natural language annotations. Neurolmage, 180:223-228 2211, 2018
Mike Wu and Noah Goodman. Multinodal generative models for scalable weakly-supervised learning. Ad- vonces in neural information processing systems, 31, 2018.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Daniel L Yamins, Ha Hong, Charles Cadieu, and James J DiCarlo. Hicrarchical modular optimization of convolutional networks achieves representations similar to macaque it and human ventral stream. Advances in neural information processing systems, 26, 2013.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-36) 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optininn hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619 8024,86244444 2014.
Hejia Zhang, Po-Hsuan Chen, Janice Chen, Xia Zhu, Javier S Turek, Theodore I. Willke, Uri Hasson, and Peter J Ramadge. A scarchlight factor model approach for locating shared information in multi-subject funi analysis. Xiv preprint ar Xiv: 1009 09432, 2016.
A Appendix
A.1. Model Architectures
Our proposed Latent Embedding Aligument (LEA) framework aims to build the bridge between images and IMRI signals. To this end, we incorporate two distinct encoder-decoder architectures to learn latent representations for these two modalities by performing the self-supervised reconstruction task. Subsequently, we introduce a linear model to efficiently facilitate the transformation between the latent representation spaces of images and fMRT signals.
[MRI Reconstruction. We adopt the architecture of masked autoencoder (He et al. 2022) the encoder- decoder model for fMRI signals. More concretely, the fMRI encoder consists of 24 transformer layers, each of which has 1024 feature dimensions with 16 heads. We append [CL.) token to the input fMRT signals before foeding them into the fMRI encoder. After self-attention processing, the [CLS] token is expected to capture the global representation of the entire input IMRI signals. Next, we mask all other latent tokens and encourage the decoder to reconstruct fMRI signals from the learned [CLS] token. Similarly, the decoder includes 8 transformer layers, with 512 feature dimensions and 16 heads. Different from (Chen al., 2022) that equally splits fMRI signals into patches, we instead maintain the voxel structure with ROI regions by additionally adding ROT-dependent processing layers (i.e., ROT Embed and ROT Predict) before and after the encoder and decoder, respectively. Both ROI-dependent layers are two-layer modules with symmetric structure and not shared among diffcrent ROI regions. Taking the ROI Embed layer as an example, it first adopts a convolution layer with 32 kernels to extract multi-head features of the input IMRI signal and then uses fully-connected layer to project the different lengths of fMRI signal into a unified dimension of 1024, leading to the final fMRI embeddings for one ROI as 32 1024.
Image Reconstruction. Image-based latent space construction has been widely investigated. As our target is to decode the semantic content of IMRI signals, we adopt the pre-trained CLIP (VTT-H/14[aas Jas the image encoder and the yielding CLIP latent space as our image latent space. To generate images from the image latent space, we utilize MaskGITH as the image decoder in this paper, but other generative models are also acceptable. Specifically, the decoder is a decoder-only transformer layer, with a depth of 24, a feature dimension of 768, and 16 heads. By conditioned on the image latent features, the generation procodure starts from all [MASK] tokens and autoregressively predicts logits of tokens with higher confidence scores. It is non-sequential generative procedure, following cosine mask scheduling function.
A.2. Data Preprocessing and Augmentation
Preprocessing. We use fMR1 data originally collected and pre-processed from BOLD5000 (Chang et al. 2019 and GOD (Hrikaaw & Kamitani 2017 datasets to conduct experimental analyscs. Briefly, Chang
httpe://gghhub..co//fftendattttt clip
https://github. htw.hhhh..cccoooooooeeeeeeraraaa
16

---

Published in Transactions on Machine Learning Research (12/2024)
al. (2019) follow the BOLD- prouD--poesssing workflow of FMRIPREI and extract data features via the general
linear model (GLM). Horikawa & Kamitani (2017) preprocess data including 3D motion correction by SPM co-registeration to anatomical images and interpolation by 3 x 3 x 3 mm voxels. Please refer to the "fMRI Data Analysis" section in (Chang et al., 2019 and the "MRI data preprocessing" section in (Horikawa &
Kamitani 2017 for more details.
Augmentation. We utilize data augmentation to enlarge training data for both fMRI and image recon- struction and discard them all during the inference the fitting of linear models. For fMRI signals, only use one augmentation method by randomly adding Gaussian noise with the probability of 0.25. For image data, we apply random cropping, random horizontal flipping and finally resize images to 256 x 256.
A.3. Training and Testing Procedures
All experiments are implemented with the PyTorrc toolkit. The encoder-decoder model for fMRI recon- struction is initialized with pre-trained weights from HCP datasets (Chen et al. 2022). We finetune it with both fMRI training and testing sets for each individual. Note that using fMRI testing data is a common practice as in (Chen et al.) 2022 Beliy et al. 2019), and we do not accoss any paired fMRI-image testing data when learning linear models. For image reconstruction, we use a frozen CLIP encoder and utilize the image decoder of MaskGIT initialized by ImageNet (Deng et al. 2009h, and further finetune it with images on both the GOD and BOLD5000 datasets. Once trained, it is shared for all subjects
from the training and experiments. AdamW optimizer is applied with 81 0.9, 81 0.95 and batch size 8 finetune both models. The initial learning rale is defined as 5e-5 with 0.01 weight decay. We linearly decrease the learning rate until it reaches the minimal rate. The total number of training iterations for fMRI reconstruction and image reconstruction is 100k and 300k. For fitting linear models, we adopt ridge regression from the Scipy library. The coefficient of L2 regularization term a is set to 500 for both image-to-fMBR and IMRI-to-image regressions. During the inference of [MRT decoding, the number of steps for MaskGTT as default, and we generate samples for all evaluations.
A.4. More Qualitative Results
In this section, we show more visualizations of (MRI decoding (Figure and Figure S and Braage-MRRlimagee reconstruction (Figure
http:://imsserrrrrrraaaaeeeaesss http://www.ftl. wtw. uk/spp
17

---

Published in Transactions on Machine Learning Research (12/2024)
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 7: Samples of fMRI decoding on CSI-1 of BOLD5000 dataset.
18

---

Published in Transactions on Machine Learning Research (12/2024)
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Sepes
Figure 8: Samples of IMRI decoding Sbj-3 of the GOD dataset.
Prodined
Produced PIRI
Diaeode
Desiie
Immgr
heag
Prediced
Prwdieed PHiE
Predined
Emege
Figure 9: Visualizations of Image-MMRL-imayye reconstruction on CSI-1 of BOLD5000 dataset.
19


Reviews:
[{"review_id": "obT9cfVoBv", "reviewer": "Reviewer_PUR8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem motivation is of interest in neuroscience - understanding the relationship between visual stimuli and brain signals. \n- The proposed approach, which first learns the reconstruction models then aligns the latent embeddings with a linear model, allows for training the individual domain auto encoding models from larger amounts of data followed by learning from smaller amount of paired data to link the two latent domains.\n- Multiple experiments with different evaluations are performed to assess the proposed method.\n- The approach nominally performs as well or better than many of the compared baselines.\n\nWeaknesses: \n- Many method details are unclear or missing, which are key to understanding the approach \n- Some experimental methods details are unclear or missing, which are key to evaluating the results \n- Some additional experiments would help provide more evidence for some model choices\n- The results of the paper, while showing some overall improvement, are based on a small number of subjects (5 from one dataset, 4 from another). I recognize this is in part due to the availability of such a dataset, but it is hard then to say whether the results show true improvement over baselines, and further, whether the results would be generalizable.\n- The paper needs additional proofreading and editing for language and typos.\n\nPlease see next section for specific details/questions regarding the weaknesses", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach for predicting the stimulus image presenting from fMRI data (fMRI decoding) and predicting the fMRI signal given the visual stimuli (fMRI encoding). The approach first trains separate auto encoding reconstruction models for fMRI signal and images separately, then learns a linear transformation to map the latent embeddings of paired fMRI and image data. The method is tested and compared against baselines on multiple fMRI-visual stimulus datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is unclear what exactly the input is the fMRI data f. While image I is given dimensions, f is not defined. Reading the ROI embedding section, it seems that the fMRI may be the BOLD image data from a single time point, but the imaging is dynamic, so how is the frame chosen? For pre-training this is less a concern, in that each frame could be used for learning the reconstruction (although the dynamic information is then lost), but for decoding the fmri signal -> image stimuli and vice versus, how was the correspondence between fMRI frame and visual stimulus determined? \n\n2. The paper mentions \"standard pre-processing\" for fMRI, but there is no real \"standard\". At least a brief description of the type of fMRI preprocessing performed is needed.\n\n3. The variable M is used to represent 2 quantities (number of ROIs and total amount of training data) - different variable names should be used to avoid confusion.\n\n4. The paper states that the image encoder comes from a CLIP pretrained model, but there are no details regarding what dataset is used to train the CLIP encoder. Please clarify.\n\n5. To learn the latent fMRI embeddings, the authors state they take an \"individual-specific approach\", where the embedding is learned per subject. I am confused then how generalizable a model would be, both in terms of machine learning standpoint of applying to new subjects and new data, but also in a practical sense, since if the model needs to be tuned for each subject, you would always need to have some paired data available for each new subject, which is not necessarily feasible. Could the authors please discuss and address these concerns.\n\n6. The metrics used to evaluate the results need more explanation. The authors use what they call \"CLIP correlation\" and \"CLIP distance\". I am still not sure what the CLIP correlation is based on the description in the paper, and the CLIP distance seems to actually be a similarity measure (where higher is better) which is confusing, and also I don't know how exactly the similarity is measured. Please edit and clarify.\n\n7. The authors note that the fMRI testing data is used for learning the reconstruction task in self-supervised manner. While it is true that no \"label\" is used, this still means that the learned embedding space will be based on the testing data, and thus is not really an independent test dataset. Please provide some comment / explanation if this is the case.\n\n8. Some of the methods have \"relax\" as part of the model name. Although the authors note that some setting corresponds to relax, I don't see in contrast what the corresponding method without relax refers to. Could the authors please clarify both what it means to have relax and not have relax appended to the model name.\n\n9. In Fig. 4, qualitative examples are shown from Ozcelik et al., but I do not see any corresponding quantitative results that would be expected to be shown in Table 3. The results from this comparison should be included.\n\n10. The compared PatchEmbed method is only tested on one size (n = 16), which actually outperforms many RoIEmbed settings that tested other dimensions for the proposed approach. It would be good to see the effect of different size patches for PatchEmbed to show that RoIEmbed really gives an advantage.\n\n11. Overall, the paper still needs a lot of editing for polishing language for clarity.", "broader_impact_concerns": "A broader impact statement section is included, and while it discusses some possible concerns (though many are still far-reaching considering the current state of the results), I think it could be helpful to also discuss the potential effects of using the approach when the results are inaccurate/incorrect (eg the fMRI decoding prediction is wrong)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RErZkwJf9j", "reviewer": "Reviewer_rbVG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the authors propose a method that outperforms the baselines both for encoding and for decoding. The method for embedding and decoding images, fMRI signals and for linking the 2 latent spaces is well described.\n\nWeaknesses: some aspects of the experimental setting could be clarified (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a method for fMRI encoding (predicting fMRI activations given a visual stimulus) and decoding (reconstructing the stimulus image given the fMRI activations). \nTo do so, they use 2 separate encoder-decoder pairs, one for (stimulus) images and one for fMRI signals.\nThen they train linear models to learn the mappings from the latent space used to embed stimuli to the latent space for fMRI signals and vice-versa.\nTraining the encoder-decoder architectures separately for fMRI and natural images, and using a simple linear model to link the 2, allows exploiting available fMRI-only or image-only data, and requiring only a small amount of aligned (fMRI, image) pairs which are scarce.\n\nThe authors assess the encoding and decoding capabilities of their approach on the BOLD and GOD datasets, which contain an important amount of fMRI data on a small number of subjects.\nThey compare their method to several baselines from the literature, which the new method outperforms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors provide more details on the \"fMRI signal f\" -- its dimensions, how it was obtained from the BOLD timeseries, etc. In the N x L matrix, IIUC N is the number of images and L the number of voxels that are in one of the considered ROIs. What are the values of N and L? It might be interesting to display a brain map of the correlations, so that we can see which regions' activity is well predicted by the encoding model. \n\nI am not sure I understand exactly which testing data is used and for which task in the LEA(\"relax\") setting. Is the encoding model allowed to see the test stimuli (but not the corresponding fMRI data) during training?\nWhy is LEA(relax) present on table 3 but not table 1; and does \"ours\" refer to LEA or LEA(relax)?\n\n\"Empirical findings suggest that a similar linear relationship exists between fMRI- and vision-only models\" -- could the authors provide a reference?\n\n\"on various benchmark datasets\" -- here IIUC \"various\" means 2 (BOLD5000 and GOD)?\n\nDo the authors intend to share their code and trained models online after publication?", "broader_impact_concerns": "The \"broader impact\" statement feels overly cautious -- there is a gap from decoding the broad category of image someone is looking at with low accuracy after hours of fMRI recording and \"reading people's mind\".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5EY8BfvLjO", "reviewer": "Reviewer_KKrJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*:\nThe methods are evaluated on several datasets and metrics and generally come out ahead compared to the competitors. The paper also highlights issues with previous methods, which should generate nonsense when provided with a noise fMRI signal, but which instead generate sensible high-fidelity images, and qualitatively starts to demonstrate that their model may have better properties on this metric (generate nonsense when provided with nonsense).\n\n*Weaknesses*:\nMy main hesitations about this paper derive from insufficient benchmark method comparisons and the lack of citations of earlier, highly related work. Currently, the paper lacks comparisons to earlier work in aligning multi-subject fMRI and learning alignment maps between shared fMRI representation spaces and other embedding spaces - it would be ideal to understand the improvement of the neural method proposed in this work over the simpler linear Shared Response Model (SRM) approach to learning a latent embedding space for fMRI signals across subjects (https://proceedings.neurips.cc/paper/2015/hash/b3967a0e938dc2a6340e258630febd5a-Abstract.html), as well as the resulting linear alignment maps between fMRI space and image space (https://www.sciencedirect.com/science/article/pii/S1053811917305128, where the paired representation is text). The ROI embedding layer may also benefit from a comparison to the earlier works (https://arxiv.org/abs/1608.04846, https://arxiv.org/abs/1609.09432). These works (and the first two mentioned) also contain similar considerations to the ROI embedding layer, but are a simple linear approach and another convolutional autoencoder approach, and thus natural baselines and comparisons.\n\nAnother question I had was about Figure 6 - I’m not sure I completely agree that the images generated on noise from LEA were particularly more uninformative and unrealistic (perhaps just the house in the sky?) It would be nice if there were a more thorough study of this component of the paper, since it seems like quite an important test to better understand whether the models are vacuously generating images or not. Likewise for the zero-shot task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method (LEA) for learning and aligning two latent spaces -- that of an fMRI signal, and that of corresponding visual images. The proposed method predicts bidirectionally, and can recover either fMRI signal from image, or image from fMRI signal. Of particular note is that this approach does not require (too much) paired fMRI-image data, which is a constraint in previous methods which attempt to directly apply seq2seq style approaches to learn mappings (though certainly previous methods also solve this problem in similar ways). The approach is to use CLIP embeddings for the vision latent space and to introduce the ROI embedding layer, a masked autoencoder applied to fMRI data segmented by ROI region. Once the latter model is trained from a lot of (but non-paired) fMRI data, linear maps between the latent spaces are learned on a small set of paired data, which then fill in the gap for the whole decoding/translation pipeline. The methodology is evaluated on several relevant metrics and datasets. They also add an important additional test, where they present fake noise fMRI signals to determine whether the image generation capabilities appropriately degrade.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add comparisons to the methods mentioned above, as they are quite similar to various parts of the pipeline and are simpler. It would be good to understand if we can replace more complex methodology with simpler methodology, and what the delta is when resorting to more complex methods. It would also be good to provide better attribution for previous works (mentioned above) which outline very similar ideas to the ones presented in this paper.\n\n2. It would be great to add a more thorough analysis of the experiments performed in Fig. 6 - perhaps some time spent coming up with a qualitative measure of this property could be fruitful, as well as being highly relevant for future works.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The consensus is that the paper is OK in general, but does not include enough explanation on the fMRI preprocessing procedures. These explanation shave to be included.

Besides, I found several typos. I list a few below, but the text obviously needs help from a grammar checker / speller.

* Abstract: "it is still challenge" ->  "it is still challenging" 
* "Our approach starts by avoiding the need" does not sound right
* " the issue of signal redundancy present in fMRI data" -> does not make sense. redundacy brings robustness for encoding/decoding...
* "assessing the reliability of both fMRI encoding and decoding models through the image-fMRI-image reconstruction assessment." assessing/assessment
* p.2 "when predicting brain activities" -> "when predicting brain activity"
* p.3 bottom, misplaced comma
* "the trained decoder can subsequently used" -> "the trained decoder can subsequently be used"
* RoI or ROI ?
* section 3.4 is too informal, and should be rewritten more thoroughly
* "the fMRI testing data is only trained for the reconstruction task via a self-supervised manner." -> "the fMRI testing data is only trained for the reconstruction task in a self-supervised manner."
* table 2 presents percents I guess ?
* "With the absent of" -> "With the absence of"
* "Human brains are known to spatially separated, that different RoI regions..." -> "Human brains are known to spatially segregated, that different RoIs..."
* "number of head" -> "number of heads"
* "On the contrast" -> "by contrast"
* "to efficiently recovers" -> "to efficiently recover"
* "read people’s mind." is too colloquial and too simplistic. Please rephrase
* "its significant contributions in the field." -> "its significant contribution in neuroimaging."
* "decnoder" -> "decoder"
Decision: Accept (poster)