Alleviating the Semantic Gap for Generalized fMRI-to-Image Reconstruction
Authors: Tao Fang
Qian Zheng
Gang Pan
Venue: NeurIPS 2023 spotlight (2023)
Keywords: fMRI
image reconstruction
brain decoding

Abstract: Although existing fMRI-to-image reconstruction methods could predict high-quality images, they do not explicitly consider the semantic gap between training and testing data, resulting in reconstruction with unstable and uncertain semantics. This paper addresses the problem of generalized fMRI-to-image reconstruction by explicitly alleviates the semantic gap. Specifically, we leverage the pre-trained CLIP model to map the training data to a compact feature representation, which essentially extends the sparse semantics of training data to dense ones, thus alleviating the semantic gap of the instances nearby known concepts (i.e., inside the training super-classes). Inspired by the robust low-level representation in fMRI data, which could help alleviate the semantic gap for instances that far from the known concepts (i.e., outside the training super-classes), we leverage structural information as a general cue to guide image reconstruction. Further, we quantify the semantic uncertainty based on probability density estimation and achieve Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) within a diffusion process. Experimental results demonstrate that the proposed GESS model outperforms state-of-the-art methods, and we propose a generalized scenario split strategy to evaluate the advantage of GESS in closing the semantic gap.

Alleviating the Semantic Gap for Generalized IMRI-to-Image Reconstruction
Tao Fang!? Qian Zheng";; and Gang Pan
'College of Computer Science and Technology. Zhejiang University, Hangzhou, China The State Key Lab of Brain-Machine Intelligence, Zhejiang University. Hangzhou, China {duolafang, qianzheng, gpan) @zju.edu.cn
Abstract
Although existing iMRIIto-image reconstruction methods could predict high- quality images, they do not explicitly consider the semantic gap between training and testing data, resulting in reconstruction with unstable and uncertain semantics. This paper addresses the problem of generalized fMRI-to-iman toiiiagigeeeeeeee reconstruction by explicitly alleviates the semantic gap. Specifically, we leverage the pre-trained CLIP model to map the training data to a compact feature representation, which essentially extends the sparse semantics training data to dense ones, thus alle- viating the semantic gap of the instances nearby known concepts (i.e., inside the training super-classes). Inspired by the robust low-level representation fMRI data, which could help alleviate the semantic gap for instances that far from the known concepts (i.e., outside the training super-classes), we leverage structural information general cue guide image reconstruction. Further, we quan- tify the semantic uncertainty based on probability density estimation and achieve Generalized fMRI-to- -i--gee reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) within a diffusion process. Experi- mental results demonstrate that the proposed GESS model outperforms state-of- the-art methods, and we propose a generalized scenario split strategy to evaluate the advantage of GESS in closing the semantic gap. Our codes are available https:/ggihbb.commmdddaaaa//SSS
Introduction
Functional magnetic resonance imaging (fMRI) a powerful tool for studying the human brain and visual system, as provides a non-invasive way to measure neural activity. Image reconstruction from fMRI data is important for studying visual representation in the cortex and for developing the vivid "reading the mind" brain-computer interface (BCI) technology
High-quality IMRI-to-image reconstruction is a typical cross-modality problem 14 28
and suffers from severe ill-posedness (2). Existing state-of-the of--h---rtttttt methods leverage the data-driven scheme to address such ill-posed problem by learning data prior from training data. However, training data are often collected a limited number of instances [9), and real-world images distributed wide, broad semantic space with long-tail distribution 12]. This brings about the problem of the semantic gap, the semantics testing instances may be unknown in training stage.
Addressing the semantic gap between the training data collected from the laboratory and the testing instances in the real world helps develop reconstructions for generalized fMRI-to AMRI-to-iman foil-to-mmaee scenarios and significantly promote the application of BCI. However, previous methods put too much attention on improving the image quality while less focus the semantic accuracy of the reconstructed images, which brings two problems. Unstable semantics: The limited samples in each super-class fails to
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

form compact feature space, which may result in incorrect decision boundaries and inability to estimate robust semanties even located within the known concepts (Fig.Ic). This is what we called the inside-space gap (ISG). Uncertain semantics: The concepts covered by the training set is not enough, resulting uncertainty the prediction of test samples with unknown concepts (more like a zero-shot problem), which is called the outside-space gap (OSG). Traditional methods assume that the training set covers all the semantics in the test set (Fig. lb), and ignore the semantic gap caused by unknown samples in reality.
To this end, this paper addresses the generalized fMRI-to- -imge reconstruction problem by explicitly alleviating the semantic gap. To deal with the instances within known semantic space (ISG problem), we map the fMRI signals to compact semantic space via pre-trained Contrastive Language- Image Pre- rre-Trainng (CLIP) model[3]. To deal with the instances within unknown semantic space (OSG problem), we propose to use the structure information as a transferable cue to guide the reconstruction, which is inspired by the robust and redundant low-level representation in visual cortex [9]. As is difficult to find hard boundary to define ISG and OSG cases for given instance, quantify its semantic confidence by probability density estimation on the training semantics and adopt the likelihood as the contribution indicator. Finally, we achieve Generalized IMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) in a diffusion process.
Our contributions in this paper could be summarized as:
We explicitly address the generalized fMRI-to-image reconstruction problem and formulate its solution as alleviating the semantic gap within known and unknown semantic subspaces.
We propose CLIP based method to expand the fMRI features to compact semantic space alleviate the inside-space gap, and structural information guided diffusion model to alleviate the outside-space space gap.
We construct a confidence indicator by quantifying the semantic similarity between a given instance and the training data, based on which we propose GESS to achieve generalized IMRI-to-image reconstruction by adaptively weighting the semantic and structural informa- tion.
Our experimental results demonstrate that the proposed GESS model outperforms the classical and state-of-the-art methods. Additionally, we propose a dataset split method to construct generalized IMRI-to-image scenario, which allows us further evaluate the model's generalization ability.
2 Related works
2.1 Image reconstruction from IMRI
Decoding visual signals from the visual cortex has been investigated for long time, ranging from early manual image feature extraction and linear mapping for prediction 20 [1] to later combining
deep learning models (DNNs, GANs, DMs) 25 15 27 10 and continuously improve the effect of generating images. Here we roughly divide the existing decoding methods into three categories:
(1) Reconstructing from the low-level features, such 220 designs contrast patches image stimuli and adopts linear mapping to predict from fMRI signals, [21] adopts Gabor features in reconstruction, and 2 directly establishes bidirectional mapping between fMRI and images based on CycleGAN without explicitly modeling the semantics. Directly reconstructing the structural features comparatively simple and efficient while suffers from the low quality results.
(2) Explicitly extracting high level conditions like categories for reconstructions. [6] refer to Masked Autoencoders (MAE) [16] to extract high-level features in self-supervised manner, and [3] explicitly predicts category information by a classification task and uses it as conditions for a diffusion model. Such researches achieve vivid reconstructions due the well- well-pretrained image generator, but have not considered the great semantic gap in reality. Obtaining incorrect semantics result in unrelated reconstructions that may look natural but meaningless.
(3) Hybrid methods that extract multiple-level features and jointly use them for reconstruction. 10] extracts high-level semantics and low-level shapes from different visual areas and adopt GAN for

---

(a)
Generalized fMRi-to-image Scenario
(c)
what's this?
Texting Semantica
Training Semartics
have seen more calogories in reality...
Outside-space
Inside
Koewn
spece
Text
Dage, caté and..
Model
[b]
Subject
Trafflonll Assunction
???
?!?
"cat"
"Eigr"
In experiments
"Felines" (super-class)
Figure I: Illustration of the semantic gap in generalized IMRI-to-image reconstruction: (a) Humans have excellent generalization ability for long-tail distributed categories in the real world because they have seen a lot, but for models that have only seen limited fMRI data, difficult to generalize to more complex scenarios. (b) Traditional methods implicitly assume that the training set contains the semantics present in the test set. (c) We consider that a generalized scenario, the training set forms semantic space called the known space, and there is intersection between the semantics in the testing set and the known space. In the inside-space scenario, although there may be unstable semanties, if mapped correctly, similar semantics (e.g., felines) still be found in the neighborhood. However, in the outside-space scenario, the semantic divergence of the testing samples is too large, leading to uncertain or even irrelevant estimated semantics.
reconstruction. 27 also extracts high-level and low-level features conditions for diffusion model. [1] injects category information within a GAN reconstruction phase. The mentioned hybrid methods still do not achieve satisfactory results due to incorrect semantics caused by the semantic gap. We propose hybrid method that not only explicitly extracts semantic and structural information, but also adaptively integrate the features with based on the semantic uncertainty, alleviating the semantic gap and achieving general and vivid reconstructions.
2.2 Large-scale prefrained models
prove that models with large enough number of parameters trained with large-scale datasets show more generality and practicality for many downstream tasks like zero-shot identification. CLIP 30 is one pretrained model which predicts a wide range of visual concepts using natural language supervision. proposes that learning directly from raw text about images provides broader source of supervision, and scalably learn state- of-the-art image representations from scratch on a dataset of 400 million. We obtain an image-text shared feature space through CLIP, where the point-wise distance indicates the potential semantic relationship, and implicitly provides an extending manifold for the limited training semantics and a potential interpolation ability in the prediction phase. Diffussin models model that decompose the image formation process into a sequential denoising steps. The diffusion process destroies the image structure by gradually adding gaussian noise, and simulate the image generation as an inverse process using a UNet model [17]. Latent Diffusion Models (LDMs) 24 better handle high-resolution images with lower computational complexity in latent diffusion process. We select LDM for image reconstruction due to the high-resolution and high-quality generation ability,
With the development of large-scale models, 24
23
one type of generative
3 Methods
3.1 Problem Definition and Method Overview
Dividing the brain-image pair data as training X yrr and testing set X with the yte are unknown, we aim to decode accurate and natural visual stimulus ÿtte from the testing fMRI data X and make te te, The visual features can be divided into two components: semantic information

---

Semantic Module M,
et
Uncertain
Blur but Stable
eler
Insect
Tiger
CLIP (image encoder)
Regression
Stimulus
Reprojaction
Gating Function M
Prior
Structual Module M.
Reconstructed
IMRI X
VOGAN (decoder)
B2
Regression
Zt
Z--1
Features
Training samples
Fixed models
Test samples
Zt
R--1
Zo
Augmented samples Known space
Latent Diffusion Model Mjm
Figure 2: Framework of GESS. The semantic module M. maps fMRI data semantic space using CLIP and then reprojects it onto the dense manifold to obtain faithful semantics ctc_ Structural module M, extracts low-frequency features structural information S Gating function Mg adaptively assigns To for e based on the semantic uncertainty, and assigns = Te for st". LDM Mam reconstructs images conditioned on the two types of features weighted by me. Ns.
add and structural information s nxd, that shared between fMRI and images (where n denotes the number of samples, de, d, denotes the feature dimension). Here e derived by semantic module M. using CLIP (with parameter Φ), and s is derived by a structural module M, using VQGAN (with parameter 7)) To describe the generalized EMRI-to-Image scenario, letting Rd be the universal set of semantic space with d dimension we get c E R' as the subspace of the training semantic features (that we call known space) and c Rt the subspace the testing semantic features and RRx C Rd. Rs Rd. As in Fig.Ic, lig.lc, in the generalized fMRI-to-lmage scenario, there is overlapping that Rdr n Re * and Re g RRr (different from traditional assumption that R C R")). To avoid confusion, from now when referring to space, it will be in the context of the training semantic space. Due to the different uncertainties of c E RR and c & R' across samples caused by the generalized scenario, introduce gating function Tc(c")) and (s*)) in mixture of experts (MOE), to adaptively assign confidence for semantic and structural information. With the aforementioned components and their estimated weights, we obtain reconstructed images yte from conditioned latent diffusion model, fo " s*e Te. ).
In this paper propose two-stage decoding model: (1) To alleviate the semantic gap, implement a semantic module (M, in Fig.2) that maps fMRI data onto a dense and continuous manifold constructed by CL.IP and structural module (M,) to extract low-frequency features as a transferable clue. (2) To determine the ISG/OSG situation, we propose Mg to estimate the inside-space likelihood to implicitly assess the situation and combine the two types of features using a weighted assignment approach in a diffusion process (Mam).
3.2 Semantic gap alleviation
The visual concepts are long-tail distributed in the real world so that the semantics that collected in a limited BCI experiment could not cover enough real world concepts. To achieve reconstruction in the generalized
scenario, we further divided the semantic gap into two cases, which have been addressed by modules Mc, Mg respectively.

---

3.2.1 Semantic module Mc: inside-space case
Even in the known space RT there may not be enough samples continuously represent the space due to the limited number IMRI samples. Gaps in this subspace without instance filling may result in classification failure, even if a sample has similar concepts (super-classes) in the training set. This is referred to as the inside-space gap (ISG). As ISG is still a problem within the same domain, propose that the fundamental reason lies in the discontinuous manifold directly constructed from the limited number of fMRI samples. As mentioned before, the CLIP feature space shows a locally continuous and semantically reasonable manifold (i.e., the cosine similarity reveals semantic relationships). By appropriately projecting the fMRI samples onto this manifold, becomes possible to implicitly extend the discontinuous fMRI features into manifold that covers semantics of both of the collected and unseen fMRI samples.
Concretely we use linear model (ridge regression [8] to predict from with large regularization Ae to limit the impact of noise on predicting semantics. We get ct fo (xx") and fit the coefficients 3e of ridge regression with & of CLIP fixed. Although linear models and regularization may reduce the diversity of semantics and make it difficult to capture non-linearity, the simple relation assumption shrinks the coefficient variance and make the model less prone to over-fitting to the noise.
However, the mean squared error (MSE) and ridge regression cannot guarantee that the estimated cte" are mapped onto the dimeneessional manifold in the nigh-dimensional space. Therefore, we propose two methods to alleviate this problem.
Momentum alignment We propose to align the statistics of the predictions and the image prior distributions to reduce semantic domain shift from distributional perspective. To get the statistics of the manifold, we first sample Ct from the manifold by feeding the training images into CLIP models. However, the scarcity of training samples makes difficult to accurately describe the manifold. Here we use augmentation C = fo (X Xaus)) to help the manifold sampling (let Call = C U Caus) and to satisfy the local linearity to some extend. Previously we get Ctte" by ridge regression Be, and here we adjust its statistics by using a simple domain adaptation method named first-order momentum alignment like [13], which involves a whitening and an alignment process to get Cite' which has the same mean and variance as Call (supplementary materials 6.2).
Linear re-projection Here we directly projects Cte' onto the manifold. We assume that in small local area of the manifold. there are enough feature points such that manifold point can be approximated as a linearly weighted sum of its K-nearest neighbor feature points (Fig. 2). The projection principle is that we re-project the object point ctv' to ce using its K-nearest points ch C K. on the manifold (KC C'lll) by a linear combination model: ctc = Ex wet cbh where w the coefficients of each mutiii-mmnniood points oi Here we solve constrained least-squares problem:
arg min wit
||ce w 2h 112 s.t. w
to get the cth that located in manifold for latter reconstruction (supplementary materials 6.1). This approach [5] essentially considers that the information that causes the components to deviate from the manifold is not important and could be removed.
3.2.2 Structural module M,: outside-space case
Although previous work has also considered extracting semantics (27. it has not addressed the extreme scenario where the image semantics are distributed outside the known space, which is hard to get trustworthy predictions. We call it outside-space gap (OSG). For generalized scenario, propose to use structural information Ste to compensate for the insufficient generalization caused by solely depending on semantics. That is because structural information is a category-independee attribute, and is robustly and redundantly represented in multiple cortex areas 19] We use the low quality S to supplement the uncertain semanties in reconstructions.
To extract the fow-freyuency components as S, before fitting we conduct Gaussian filtering on visual stimuli Gaussian (Y"t, r) with large kernel = 15 to prevent the model from trying to extract the high-frequency components from fMRL. Besides this can seen as to suppress the variance of the predictions and to some extent, suppress the noise. In this task simple ridge regression could
5

---

already achieve acceptable sh which is fast and easy to implement. Concretely, we use a pretrained VQGAN to extract the ground truth representation s f- (v²") of each image. Then, with a ridge regression Ss. we can establish the relationship between fMRI and the latent variables by minimizing (o,(()))) Str + A,a). where Š has been flattened here. Another approach is to use CycleGAN (2) that trained on y to generate 5 which shows higher quality. Since a large number of epochs are needed to fit the model, it is computationally more expensive, so that the above two methods represent a trade-off between quality and efficiency.
3.3 Adaptive Integration with LDM
3.3.1 Gating function Mg for Integration
Although in the first part, we have explicitly divided the problem into the ISG and OSG sub-problems, and construct the faithful manifold combined with structural information to alleviate the semantic gap, but in reality it is difficult to determine the boundary between the ISG and OSG cases. As the boundary, probably determined by a threshold, would introduce an extra inaccessible for the testset. Considering that different features have their own strengths and inspired by mixture of experts (MOE (3), we regard Mx(x) and Mx(x) as two experts, and use a gating function (r(r), T (x) to indicate each expert's own regions within X We assume that there exists an integrated intermediate information h which could be marginalized as a weighted sum of:
which is
p(1 (x}*) = To( )p(c))/()) + n. T,(x" p(() at S.L.T. + n. = 1,
where i = (1,.....)) n) and Tee are gated functions of the semantic and structural modules which works as adaptive weights for each component, and (c["[[]]])) p, (s(" (=][[[[]]] describes the intermediate features derived from the two modules respectively.
Here we propose estimate the gating function in the form of likelihood of inside known space as Te(2!") ((c [C*t)) and compute т. (I²"") 1 (x("")) This could be interpreted as samples that are far away from the manifold that the training semantics covered are more uncertain. Concretely, we propose a non- non--oaaamae parametric density estimation method based on kernel density estimation (KDE 3 with bandwith=1.5, 55, and then normalized to [0, 1J), which uses training samples to estimate the underlying probability distribution to approximately estimate p(c{") C (supplementary materials 6.3).
3.3.2 Diffusion model Mam for generation
Considering the powerful generality of LDM, we propose to implement MOE within the diffusion process. But directly taking the conditions in the form of cross attention imputs is not appropriate as it difficult to realize feature weighting and structural guidance containing spatial information. For MOE model, we need common space to integrate the different modalities of t and g We first consider weighting in the gradient space and introduce gradient guided method (GG for short), and inspired by 8 we get:
(2., 22,,41, h.) = A po(2i,122,,44 (h.l2,
(3)
where A is the normalization term, 0, 17 denote the parameters of the UNet and the guidance module (like a classifier), 2i, means the latent features of sample i at time step in the LDM. For the sake of readability, we omit the 'i' and 'te" tags in the following. As mentioned before we have estimated the confidence म. and n. from x; and get (hlzz) = (c(2)) + T&Ps (s|zz) with the confidence fixed during the diffusion process. By Jensen's 's inequality we obtain the lower bound of log likelihood:
= log(pe + log Σ Pm (ml|ze
(4)
2 log(po
+ Σ Tm log Pm (fmml (2)), m E (c,
(5)
maximizing which across the diffusion process is equivalent to maximizing the log likelihood, in this way we realize the instance- wise adaptive weighting of different image features in the diffusion process. Further with Taylor expansion as in [8]. we get an approximation of the likelihood:
log Pm (hm l2x) (st Me)gm + B.m {c,s),
(6)

---

wherein the gradient information 9, comes from a loss term in the guidance module that measure accuracy of certain decoded feature, and B is a constant term. In this case the gime comes from guidance of CLIP features C and fims comes from St with Lise guidance. Known that the latents means a perceptual compression
9 of raw image, still retains certain image properties that performing image manipulations such as kernel smoothing in the z-space makes sense. Here we apply low-pass filter F(z) to extract structural information from 2, and get Lmse =
F(z)) F(s) where z/ are estimated from [26].. One advantage of gradient guidance lies in its flexibility, like
combination of image variance loss and other optimizations to improve the quality of the generated
images.
However, GG method requires computing the gradients each timestep. which greatly increases computational complexity. Inspired by ILVR [7]. which can directly replace different frequency components the image-space, we propose an efficient component substitution method (CS for short). Firstly we propose the component segmentation method:
2 = Σ Fm(4)) E {c,s),
where Fm means the filtering functions for different levels of information (e.g., high or low frequency information). In this work we have two-level features: semantics F. and structure F,. (zx) can be obtained through low-pass filtering function and get F (z)) = - F, (2().. 2). We compute each frequency component of 24 as weighted combination of various experts: F (z,) = n. J, (2,,s)
F.F (2t,),, where 2t žt,s means the latent features derived from the c, conditions respectively. However, for the structural guidance Fe(2,.) does not contain accurate semantics that the F.FF term does not make sense. Thus we only focus on the low-frequency component here. Referring to
we get 2t-1 from Z₁:
2t-1,c =~ po 120(21-1 (2t- 112,, c); 2x-1 Ft-1,c c (=(= + m. F.FFa
(8) (9)
where we compute 2t-1,a by gradually adding noise in that has been estimated by M,. Since the generation process does not require gradient estimation, it faster and requires less computational resources compared to the previous method. However, the generated images sometimes become blurry (possibly due to the discontinuous low-level substitution that may disrupt the image structure). Therefore, the two methods are a trade-off between efficiency and image quality.
3.4 Training details
In our model only the ridge regression modules of Mg, M. need be trained, while the other modules are pretrained on targe-scale data with fixed parameters. The overall process of our method is described in Algorithm
Semantic information. To extract semantic features, we use the pretrained VIT-L/14 clip model on 1200 training images and about 4000 augmentation images. We ridge regression trained with A, = 1000. We slightly smooth the images with a Gaussian kernel (r 5) before extracting semantic features.
Structural information. To extract structural features, images are pre-smoothed with a Gaussian kernel (r 15). In experiments
CycleGAN with the same setting as (2). We do not use augmentation as (2 because high-quality details are not necessary here.
Generating. We adopt the pretrained LDM in 4 (knn2img model). During integration, both of the two methods mentioned section 3.3 are implemented and compared section 4.4, and we finally select CS method considering its efficiency and acceptable generation. A temporally scaling strategy used in diffusion process reduce the influence of 2t,s and results in clearer generated images. We DDIM [26] acceleration during sampling with 50 time steps.
7

---

Algorithm The pseudo code of GESS.
Paired fMRI and Image Y Dataset: D {(x)", y "!" and De {( Output: Reconstructed stimulus images 1² from fMRI x Training:
Input:
Initialization: Constructing CLIP 23 VQGAN 99] and latent diffusion model 24 by pretrained
parameters Φ, y and 0.
Training parameter Be of semantic module Mc:
1. Extracting semantic features 6," from y/" by CLIP: 4" = ")
2. Training ridge regression parameters 3e by à th and xF:: Ac (CTŌ+AI) + -0T
where C = |6i tr , ..... CN "] and X = [ 22 xx]
Training parameter 3, of structural model Mg:
1. Extracting structural features 6," from " by VQGAN: 8, (vt").
2. Flatting s/", and training ridge regression parameters 8, by s," and at.: Be (STS
1 5 x where S = [ S 5N
Inference:
1. Semantic module Predicting semantics from fMRI: ce" = Beat Using momentum aligmment to get cae (section 3.2.1). Using linear re-projection to get ce (section 3.2.1).
2. Predicting structure from fMRI: be ; = 3,
3. MOE Estimating weighting parameters n. by KDE and म, = 1 - ne (section 3.3.
4. Latent diffusion process by CS strategy (section 3.3.2):
(a) Initializing 2a by Gaussian.
(b) For 1 1,,,...,?!
i. Conditioned by cross attention: 2t-1,c ( po(2)-1|2,, c00)
ii Conditioned by CS strategy: žt-1 2t-1,0 - 7el. a (21- + a (2t-11)))
(c) y = [T(T))
4 Results
4.1
Dataset
We evaluated the performance of our model using two datasets: the General Object Decoding (GOD) dataset [19] and the Natural Scenes Dataset (NSD) The GOD dataset contains 1200 images from 150 categories for training and 50 images from 50 categories for testing. NSD uses images from the COCO dataset and roughly 10,000 fMRI-image pairs for one subject. During a visual experiment, researchers recorded subjects' brain activities by fMRI scanners while they are watching images screen. In GOD, there was no overlapping between the training and testing classes which matches the generalized scenario (supplementary materials 7.1).
However, the random split method used in 27 for NSD causes too much overlap between the training and testing semantics (Fig.4a 4a left), so that simple baseline model (Fig. 4a left) by nearest searching can achieve comparable performance. To address this issue, we proposed generalized split strategy for NSD in Section 4.4. We showed that while the simple baseline model perform worse under this strategy, our model is still able to achieve stable reconstruction in this situation.
4.2 Evaluation of image similarity
Comparing and estimating image similarity is challenging task in computer vision as several factors like object dislocation, different viewpoints greatly increases pixel-wise distance like L2. To measure the similarity between the reconstructed and ground truth visual stimulus, we propose a combined comparison criteria that builds upon previous works with slight modifications. Specifically, we use: (1) a 2-way comparison task as in [19] with three different metrics, including RMSE, Spectral Angle Mapper(SAMP) 29), a physically-baie based metric), and perceptual similarity metric based

---

pretrained CLIP (VIT-L//44) to make an overall assessment of reconstruction accuracy, and (2) visual comparison of the reconstructed images from different methods. Besides to remove the randomness caused by pairwise comparison, we repeat and averaged 10 trials for each sample.
We compare method with several existing methods GOID: GAN-based methods include 121. which use CycleGAN to directly decode images from fMRI data with augmentation and [22] use ICGAN as backbones. Diffusion-basse methods include [27] use MAE extract features unsupervisedly as conditions and inputs into a latent diffusion model, and [ from fMRI and use diffusion model reconstruct the image conditioned on the category information. Most previous methods approach LDM as purely generative model without exercising tight control over the diffusion process.
predict image labels
4.3 Performance comparison
G.T.
90
Ours
80
Belly
Accuracy(N)
70
60
Ozcalkk
Takagi et.l
Cana
Farranta et al.
Yakigg at
Sub1
Sub2
Subb
AVG
Figure 3: Performance comparison. Left: visual comparison of reconstructed images from different methods (more examples could be found supplementary materials). Right: quantitative comparison of different methods by perceptual similarity.
We performed a quantitative and visual comparison between our model and several candidate models using the GOD dataset on three subjects and testing samples, respectively. Each sample has been averaged between 5 and 24 samples to improve signal-to-noise ratio. We also randomly sampled around 4000 images from ImageNet for augmentation. The final results are shown visually and quantitatively in Fig.3. We outperform the recent published methods [22 27 by a large margin (3.8%, 13.8% by perceptual similarity, 7.2%, 5.3% by SAM, 1.1% 1.1%, 16.3% by RMSE). As can be seen in Fig.3 3 left we generate high-quality and high-resolution (768 x 768) images that are more realistic in terms their content than others. By explicitly taking the generalized scenario into account, method outperforms the other methods visually and quantitatively.
3). We used 1200 and 50 samples as the training
4.4 Ablation study
Table 1: Abalation study on GOD dataset (Perceptual similarity).
Subjects
cLLLDMM
"= **+$*+MM+++G
Subl
74.2%
69.2%
78.0 %
76.2%
Sub2
77.6%
75.2%
84.8 %
76.6%
Sub3 79.6%
77.2%
80.4%
82.6 %
AVG 77.1%
73.9%
81.1 %
78.4%
To evaluate the usefulness of the components our approach, we conducted an ablation study on the GOD dataset. We compared the performance of different combinations of our components, mainly to evaluate the influence of semantic and structural information on reconstruction, and the importance of MOE. As shown in Table 1, equal- weighted feature integration without considering the semantic uncertainty may even lead to negative optimization. When the structural information is weighted as in columns and 5, it can improve reconstruction accuracy by 4% and 1.3% on average compared to using only semantics, respectively. can also observed that the CS method achieves higher accuracy than the GG method, even though the latter produces more natural-looking images. In the

---

supplementary materials we provide detailed evaluations of both the individual effectiveness of each component and the associated computational costs (in Table 4 and Table 5).
4.5
A generalized split strategy
G.T.
Base
Ours
TSNE randomly spited scenario
of generalized
spited scenario
Method
SAM
RMSE
Perceptual
Tarning Tostng
basc+RS
60.8%
62.3%
81.0%
G.T.
Base
Ours
basc+GS
50.8%
48.5%
52.0%
ours+RS
66.8%
63.5%
88.8%
ours+GS
60.5%
64.5%
63.3%
Figure 4: Left: visualize the difference of scenario semantic space. Our model achieves stable reconstructions across scenarios visually. Right: quantitative results of baseline model and model on different scenarios. RS:random Seenario:GS:generaliiee scenario.
We emphasize that the generalized fMRI-to- Imate setting is crucial in forcing the model to learn the visual mechanism the brain, instead of relying on a pattern matching strategy that simply retrieves from the database. Here we propose simple generalized scenario split strategy encode the images into semantic space and then cluster the images into several clusters (in our case 20 clusters) by K-means to obtain pseudo labels. Then construct graph with the cluster centers as nodes and their cosine similarity as edges, and apply a minimum cut algorithm to split the graph into two parts with the minimum number of disconnected edges. As visualized in TSNE in Fig.4 middle, a comparatively larger concept gap exists between the training and testing sets in the semantic space.
NSD. We first
As in Fig.4, we propose a simple retrieval model based on k-nearest- k-nearest-neighhor (KNN) on NSD as baseline model (supplementary materials 7.2), which can generate relatively accurate reconstructions based on st neighbor on the dataset that has been randomly split as [27] (Fig.4 4 left). Its perfor- mance deteriorates in generalized experimental setting (Fig.4 right). Here we demonstrate that in such generalized scenario, our method still perform more stable reconstructions, even though the reconstruction accuracy has decreased by 28.4% compared to the baseline model (37.0%).
4.6 Disscussion
In this paper, we explicitly define the generalized fMRI-to-image scenarios and the great semantic gaps that they bring. To explicitly address the semantic gap, decompose into inside-space and outside-space cases and propose a model called GESS (Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information) that adaptively integrates semantic and structural information within a diffusion process. Our experimental results demonstrate that GESS achieves state-of-the-a oh-tte-art performance.
Limitations. Our model still suffers from several limitations including: (1) The significant noise exists in fMRI signals, which may cause variability in the quality of generated images. This is common challenge in fMRI-based visual reconstruction and requires further research to improve robustness. (2) Sampling one image over multiple time steps in LDM still has greater computational cost compared to GANs. (3) The current model needs retraining for each specific subject and cannot handle cross-subject data. collecting experimental recordings expensive, designing model capable of cross-subject prediction could reduce the difficulty of applying it subjects.
10

---

Acknowledgments
This work was supported by STI 2030 Major Projects (2021ZD0200400), Natural Science Foundation of China (No. 61925603, U1909202), the Key Research and Development Program of Zhejiang Province in China (2020C03004).
References
[1] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, al. massive fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[2] Roman Beliy. Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. In Advances in Neural Information Processing Systems, pages 6514-6524, 2019.
[3] Christopher Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[4] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi- parametric neural image synthesis. In Advances in Neural Information Processing Systems, 2022.
[5] Shu-Yo Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. Deepfacedrawing: Deep generation of face images from sketches. ACM Transactions on Graphics (TOG), 39(4):72-1, 2020.
[6] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint 2tt:::22111005555 1(2)4, 2022.
[7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2/08.02938 2021.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-88 8794, 2021.
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[10] Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape-semantic gan. Advances in Neural Information Processing Systems, 33:1 1038--30848, 2020.
[11] Tao Fang, Qian Zheng, Yu Qi, and Gang Pan. Extracting semantic-dynamic features for long- term stable brain computer interface. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5965-5973, 2023.
[12] Anja Feldmann and Ward Whitt. Fitting mixtures of exponentials to long-tail distributions to analyze network performance models. Performance evaluation, 31(3-4):245227 1998.
[13] Matteo Ferrante, Tommaso Boccato, and Nicola Toschi. Semantic brain decoding: from fmri to conceptually similar image reconstruction of visual stimuli, arXiv preprint arXiv:22/2.06726, 2022.
[14] Prosper Aghesi Fiave, Saloni Sharma, Jan Jastorff, and Koen Nelissen. Investigating common coding observed and executed actions in the monkey brain using cross-modal multi-varian fmri classification. Neuroimage, 178:306-317, 2018.

---

[15] Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. Self-supervised natural image reconstruction and large-scale semantic classification from brain activity. Neurolmage, 254: 19121, 2022.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[18] Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1))55-67, 1970.
[19] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1 311):15037, 2017.
[20] Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using combination of multiscale local image decoders. Neuron, 6065)995 2008.
[21] Thomas Naselaris, Ryan J Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant, Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6)990--915 2009.
[22] Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Recon- struction of perceived images from fmri patterns and semantic brain exploration using instance- conditioned gans. In 2022 International Joint Conference Neural Networks (IJCNN), pages 1-8. IEFE, 2022.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763 PMLR, 2021.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[25] Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, 13, 2019.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint w:Xi:22010002502. 2020.
[27] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pages 2022-11, 2022.
[28) Haitao Yu and Quanfa Zhao. Brain-inspired multisensory integration neural network for cross-modal recognition through spatiotemporal dynamics and deep learning. Cognitive Neuro- dynamics, pages 1-14, 2023.
[29] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop. 1992.
[30] Liang Zhang, Ludan Ruan, Anwen Hu, and Qin Jin. Multimodal pretraining from monolingual to multilingual. Machine Intelligence Research, 20(2):200-2 222, 2023.
12


Reviews:
[{"review_id": "BFn9J4LccA", "reviewer": "Reviewer_ciXJ", "summary": "This paper's objective is to enhance the generalization performance of the fMRI-to-image reconstruction task through dense representation learning. To achieve this, a pre-trained CLIP is utilized to establish a semantic space, thereby bridging the gap between the training and test sets. Specifically, the paper presents an adaptive method for integrating semantic and structural information. A latent diffusion model is developed to align the semantic and structural data using the proposed gradient-guided method. Finally, the method's effectiveness is evaluated using both GOD and NSD datasets.   ", "questions": "While I appreciate the use of data augmentation for regularization in Momentum alignment, could you clarify how this might be viewed as a novelty of the paper in addressing the semantic domain shift? \n\nCould you provide more insight into the assumption of a linearly weighted sum of neighboring elements mentioned in line 169? Why is this assumption necessary? \n\nRegarding line 240, the authors examine the computational complexity of GG. It seems that ablation studies both in terms of computational and generalization performance would be beneficial. \n\nThe discussion on the outside-space gap is interesting, yet I feel some experimental evaluations demonstrating this problem in real-world cases would strengthen the argument. \n\nIn line 282, the paper considers the performance of other methods using the paper's new split strategy. Why should a new training and test split cause a decrease in performance? What would the method's accuracy be under the previous split? I believe a more comprehensive comparison is needed for a fair assessment. \n\nAnd finally, what about between-subject accuracy? Do you think that the current method can manage the domain shift and perform well in between-subject contexts?\n", "limitations": "Indeed, the paper discusses the issue of significant variance in generation quality when dealing with noisy data. Could you spell please check “Allievating” in the title.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of the paper looks novel and easy to read and the proposed method looks novel. \n", "weaknesses": "While I recognize that the paper presents a straightforward and novel extension, I am not completely persuaded by its benefits. \n\nIn its current state, the manuscript lacks clarity regarding the distinct advantages of each section. \n\nFurthermore, I believe additional comparisons with other image reconstruction methods would be beneficial to fully substantiate the paper's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer ciXJ\n### Q1. About the novelty of data augmentation in momentum alignment\nDifferent from previous works [b][d], we explicitly define the semantic gap and reduce it through momentum alignment. The alignment requires accurate descriptions of the data distribution while estimating statistics accurately from the limited fMRI data that deviates from real-world distributions remains a challenge. Therefore, the data augmentation used in momentum alignment is important for our novel contribution 1 (Section 1), as it helps learn well-aligned CLIP features. Ablation studies in Table R2 demonstrate its effectiveness (75.7% v.s. 81.1% of full method). Kindly note that our contribution 1 involves several components (momentum alignment, linear reprojection, etc.).\n\n\n### Q2. About the importance of linear weighting assumption.\n\n$c_{te}$ is reprojected from $c_{te}^*$ as a linear combination of its nearest neighbors. Practically, we found that optimizing based on the MSE loss in Section 3.2.1 does not ensure that $c_{te}^*$ lies on the manifold, and this deviation leads to performance degradation (Table R2, 71.8% v.s. 81.1% of full method). To project the vector onto this unknown manifold, referring to [a], we make the assumption that after the aforementioned augmentation, the feature space $C_a$ is locally continuous such that a linear combination of its vectors approximately lies on the manifold. With this non-parametric assumption, we do not need to explicitly define the manifold or fit an additional model to approximate it, making our approach more efficient and effective.\n\n### Q3. The computational complexity of GG and CS methods.\nTo compare the computational costs of the two conditioning strategies, we measured the inference time costs of the GG and CS methods when generating 50 test images of size 768 x 768 pixels using 50 DDIM steps on an NVIDIA RTX 4090 GPU, AMD Ryzen 5950x CPU, and 64GB RAM. As shown in Table R1, the CS method requires approximately 3 times less time than the GG method, demonstrating significantly better computational efficiency.\n\n### Q4. More experiments and examples of outside-space cases.\nIn our model, the outside-space problem is addressed by the MOE strategy (Section 3.3.1). As shown in Table R2, when outside-space examples are not explicitly handled (only constant weighting assumption), performance degrades significantly (73.9% vs 81.1% of the full method). Concretely, in the third row of plots of Figure 4, when the building photos' concept is absent from the training set, the baseline model predicts according to prior experience and generates an irrelevant reconstruction (fruit), while GESS, which considers the zero-shot learning (ZSL) scenario, works comparatively well. \n\n### Q5. More discussions and comprehensive comparison of split strategy. \nThe generalized split in our paper differs from the random split in that our split strategy considers the zero-shot learning (ZSL) scenario, where the training and test sets come from different categories. Such split follows [b] and aligns with reality: in experiments, collecting brain signals is expensive and time-consuming, which leads to a limited sampling scope.\n\nWhen a random split is considered (the training and test sets have high semantic overlap), both our model and the baseline model perform well (Figure 4, 81% vs 88.8% accuracy by perceptual similarity).\n\nRegarding the performance degradation (81% v.s. 52% of baseline model, in Figure 4), some methods that are trained on a limited number of samples tend to be overfitted without understanding the underlying visual mechanisms. As a result, they fail to generalize to real-world images under the more challenging ZSL split. \n\n\n### Q6. About applying GESS in the cross-subject case.\nGeneralizing GESS to the cross-subject scenario remains challenging. The brain signals are highly individualised, with different signal dimension and visual area locations across subjects [b][c]. So that it is difficult for a shared network to process them directly. Addressing the lack of cross-subject generalization is complex and beyond the scope of the current work. We leave exploring methods to achieve cross-subject decoding for future work.\n\n### Table R1: Time-consuming of each module.\n| Method | Time-consuming |\n|-|:-:|\n| Gradient Guided strategy | 679.87 s |  \n| Component Substitution strategy | 215.45 s |\n| Fitting Semantic Module $M_c$| 0.158 s |\n| Fitting Structural Module $M_s$ | 2.459 s |\n| CLIP embedding (pre-processing) | 47.553 s |\n| VQVAE embedding (pre-processing) | 72.366 s |\n\n### Table R2: Effectiveness of different modules by perceptual similarity (CLIP).\n\n| Subject | Sub1 (%) | Sub2 (%) | Sub3 (%) | AVG (%) |\n|---------|---------:|---------:|---------:|--------:|\n| Full Method | 78.0±0.7 | 84.8±0.8 | 80.4±0.6 | 81.1±0.7 |\n| w.o.MOE | 69.2±1.0 | 75.2±3.0 | 77.2±1.0 | 73.9±1.7 |\n| w.o. momentum alignment | 63.6±0.1 | 60.2±2.7 | 68.2±0.4 | 64.0±1.1 |\n| w.o. data augmentation | 72.4±3.0 | 76.4±0.6 | 78.2±2.4 | 75.7±2.0 |\n| w.o. CycleGAN feat. | 75.6±1.8 | 78.0±0.8 | 78.0±0.3 | 77.2±1.0 |\n| w.o. linear reprojection | 68.4±1.4 | 70.8±2.7 | 76.2±1.0 | 71.8±1.7 |\n\n\n## References\n[a] Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. Deepfacedrawing: Deep generation of face images from sketches. ACM Transactions on Graphics (TOG), 39(4):72–1, 2020\n\n[b] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.\n\n[c] Rieck, Bastian, et al. \"Uncovering the topology of time-varying fMRI data using cubical persistence.\" Advances in neural information processing systems 33 (2020): 6900-6912.\n\n[d] Takagi, Yu, and Shinji Nishimoto. \"High-resolution image reconstruction with latent diffusion models from human brain activity.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023."}, {"review_id": "8ADn0UvXHi", "reviewer": "Reviewer_NuyD", "summary": "This paper proposes a GESS model to solve the semantic gap between the training and the testing data in the generalized fMRI-to-image reconstruction task. A CLIP based method is used to alleviate semantic gap for instances with known semantic space, and a structural information guided diffusion model is used to alleviate semantic gap for instances with unknown semantic space. In addition, this paper quantifies the semantic similarity between a given instance and training data.", "questions": "It is confusing between \"VQGAN to extract the latent representation\" on line 190 and the decoding part of VQ-GANd in Figure 2. As far as I understand, this part of the figure uses VQ-GAN to encode features, and the encoding part should be used. \nIs it possible to provide a computational efficiency comparison of the models?", "limitations": "It's better to include the discussions about limitations on experiment validation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The design of Generalized fMRI-to-image reconstruction task is interesting. It considers both the known and the unknown subspace, and proposes corresponding solutions.\nClarity：The motivation of the method is clearly addressed.\nSignificance: The proposed method not only explicitly extracts semantic and structural information, but also adaptively integrate the features based on the semantic uncertainty, alleviating the semantic gap and achieving general and vivid reconstructions.", "weaknesses": "1. The comparison is insufficient which maybe partially due to the specificity of the task. \n2. The quantitative results are insufficient, and the ablation experiment is incomplete, which cannot well explain the quality of the proposed method. For example, \"Momentum alignment\" and \"Linear reprojection\" described in 3.2.1, and \"VQ-GAN\" and \"CycleGAN\" in 3.2.2, etc. \n3. Quantified Semantic Confidence is not given in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer NuyD\n### W1. About the limited comparison experiments.\nWhen we submitted the paper, we strived to find limited open-source methods ([a], [b], [c], etc.), among which [c] is the state-of-the-art of CVPR 2023. We will continue searching and include more up-to-date methods for comparison.\n\n### W2. More quantitative experiments of ablation studies.\nDue to space constraints, we did not include many ablation studies. To demonstrate the effectiveness of the proposed modules in Sections 3.2 and 3.3, we add more quantitative results (including the mean and standard deviation of the comparison metrics) and ablation studies (the effectiveness of momentum alignment, linear reprojection, CycleGAN features, data augmentation, MOE strategy, etc.) in Figure R1, R2 and Table R2.\n\n### W3. About the quantified semantic confidence.\n\nIn our approach, the semantic confidence is implicitly measured by kernel density estimation (KDE) in the MOE component. To quantify the semantic confidence, we perform a 100-class classification task and use the maximum of the estimated posterior probabilities as the instance confidence. As shown in Table R3, the semantics estimated by our model is comparatively confident across different subjects (93.6% on average).\n\nTo further demonstrate the effectiveness of our estimated semantic confidence, we compared the performance of our model reconstructed with MOE-allocated weights (confidence) to those with a constant weight in Table R2. Results (73.9% vs 81.1% with MOE) show the benefit of our estimated semantic confidence.\n\n### Q1. Details of VQGAN module.\nIn our approach, we treat the VQGAN model as a perceptual compression method and use its encoder to extract compressed image features similar to the CNNs. The subsequent diffusion process and other calculations are all performed in the compressed image feature space. The VQGAN decoder is responsible for ultimately decoding the compressed features into the image space for reconstruction. We will add the above details to the paper to clarify our usage of VQGAN's encoder and decoder.\n\n### Q2. About the computational efficiency comparison.\nWe provide a list of the computational costs of the individual modules in our method and provide a computational cost comparison between our proposed GG and CS strategies in Table R1. The results demonstrate that the CS achieves approximately 3 times faster performance compared to GG. Further analysis indicates that the most computationally demanding components of the model are the inference and embedding stages. We will add the above results to the Supplementary Material.\n### Limitations. More about the experiments.\nRegarding the experimental results, the current model limits its ability to generalize across subjects. As great difference exists across different subjects' signals, we cannot validate whether it can accurately decode semantic representations for new subjects.\n\nAnd following [b], we had to average the repeated fMRI recordings to improve their signal-to-noise ratio, which is an inefficient use of the available data. We will expand the discussion of these limitations and avenues for future improvement in the paper.\n\n### Table R1: Time-consuming of each module.\n| Method | Time-consuming |\n|-|:-:|\n| Gradient Guided strategy | 679.87 s |  \n| Component Substitution strategy | 215.45 s |\n| Fitting Semantic Module $M_c$| 0.158 s |\n| Fitting Structural Module $M_s$ | 2.459 s |\n| CLIP embedding (pre-processing) | 47.553 s |\n| VQVAE embedding (pre-processing) | 72.366 s |\n\n\n### Table R2: Effectiveness of different modules by perceptual similarity (CLIP).\n| Subject | Sub1 (%) | Sub2 (%) | Sub3 (%) | AVG (%) |\n|---------|---------:|---------:|---------:|--------:|\n| Full Method | 78.0±0.7 | 84.8±0.8 | 80.4±0.6 | 81.1±0.7 |\n| w.o.MOE | 69.2±1.0 | 75.2±3.0 | 77.2±1.0 | 73.9±1.7 |\n| w.o. momentum alignment | 63.6±0.1 | 60.2±2.7 | 68.2±0.4 | 64.0±1.1 |\n| w.o. data augmentation | 72.4±3.0 | 76.4±0.6 | 78.2±2.4 | 75.7±2.0 |\n| w.o. CycleGAN feat. | 75.6±1.8 | 78.0±0.8 | 78.0±0.3 | 77.2±1.0 |\n| w.o. linear reprojection | 68.4±1.4 | 70.8±2.7 | 76.2±1.0 | 71.8±1.7 |\n\n\n### Table R3: Averaged confidence across subjects on testset.\n| Subject | Confidence (%) |\n|-|:-:|\n| Subject1 | 93.2±14.2  |  \n| Subject2 | 94.4±10.1 |\n| Subject3 | 93.3±12.3 |\n| Average | 93.6±12.2 |\n\n## References\n[a] Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. In Advances in Neural Information Processing Systems, pages 6514–6524, 2019.\n\n[b] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.\n\n[c] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06956, 1(2):4, 2022.\n"}, {"review_id": "V0Hf7qhEj5", "reviewer": "Reviewer_qNjJ", "summary": "This paper addresses the problems of semantic gap between training and testing fMRI neural responses and generalization of fMRI-to-image reconstruction models. A pre-trained CLIP model is leveraged to map the training data to a latent feature space in which sparse semantics are extended into dense semantics, thereby alleviating the semantic gap within known semantic spaces.  Overall, it is an interesting paper, and the empirical studies show some improvement.", "questions": "The followings are the major concerns and minor comments:\n\n\n1) In this paper, the notations are confusing. In regular papers, scalers are denoted by small letters, vectors are defined with small letters (highlighted in bold), and matrices are denoted by capital letters using bold. In this paper, there are a lot of conflicts. It is so hard to trace what is a set, a matrix, or even a distribution. \n\n\n2) The proposed method can be summarized in the form of an algorithm or pseudocode.\n\n\n3) This paper is hard to follow. To explain the technical details of the proposed method clearly, some sections should be revised and reorganized. \n\n\n4) Some abbreviations are presented before their definitions – e.g., CLIP in the abstract.\n\n\n5) There are some minor linguistic and typo problems in this paper. E.g. “Alleviating” not, “Allievating” in the title of the paper.", "limitations": "Please refer to the question section", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please refer to the question section", "weaknesses": "Please refer to the question section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer qNjJ\n### Q1. About the confusing Notations.\nWe acknowledge the potential confusion caused by the notations employed in the paper and recognize the need for adopting clearer conventions. Accordingly, we will make revisions to define vectors with bold lowercase letters and represent matrices with uppercase letters using bold font, to achieve overall harmony in the final version. Additionally, for the sake of clarity (Algorithm R1), we have modified the subscript notation in the original text to superscript. Furthermore, we will ensure that the paper adheres to standard conventions.\n\nConcretely, (1) we will add subscripts to denote the conditional variable $h$ (in line 207) for clarity and provide more descriptions for each variable to make their meanings intuitive. (2) Some variables are noted with the subscript 'te' while others are not (like in line 235), depending on the context, which can be confusing. To standardize the notation, we will use superscripts uniformly across all variables. (3) We notice that in our paper, functions, modules, and matrices are all denoted using uppercase letters (e.g. lines 139, 171, 236, etc.), which is confusing. To follow standard conventions, we will revise the notations.\n\nIn summary, we will make revisions to make the final version easier to follow for readers. \n\n### Q2. To summarize as a Pseudo code description.\nWe realize that presenting the proposed method in the form of pseudo code could aid clarity and we provide it in Algorithm R1. Kindly note that we have modified the subscript notation in the original text to superscript for clarity. We will include this description in the revised paper.\n\n### Q3 About technical details for reproducibility.\n\nDue to page limits, some details were omitted or put in the supplementary materials like the detailed parametric settings (Section 3.4). To improve reproducibility and readability, we will add more details:\n\n(1) In Section 3.2.1, we will provide more context for the momentum alignment and linear reprojection methods. This includes the motivations, assumptions and parameter values.\n\n(2) The main text lacked details of the structural information extraction module, such as the transformer size [d] and codebook size. Section 3.2.2 will include more details on the feature extraction using CycleGAN [a].\n\n(3) Section 3.3 omitted some details on the normalization, kernel density estimation and mixture of experts methods, as well as parameters of our proposed conditioning strategy. We will expand this discussion in the main text and supplementary materials. \n\n(4) Section 4.1 lacked details on the dataset and preprocessing steps, which can be found in references [b][c]. We will add the necessary information in the supplementary materials. \n\nLater, we will provide intermediate features and additional results to enable reproducibility. We will also **release the code** to facilitate understanding of the work and enhance reproducibility.\n### Q4. Q5. Some correctness.\nWe acknowledge that some sections could be improved by correcting all the spelling errors in the revised version. We will also revise the case where abbreviations are presented before their definitions (line 45 for CLIP, line 117 for VQGAN, and some others in line 71, etc.). Additionally, we will correct spelling errors such as \"alleviating\", and address inaccuracies in word choices, such as revising \"sub-class.\" These revisions will be made in the final version.\n\n\nThank you for your advice and reminders. We will add more details and revise the paper accordingly to improve the reading experience.\n\n## Algorithm R1\nImage reconstruction from fMRI using GESS (component constitution strategy).\n\n**Input**:\nPaired fMRI $X$ and Image $Y$ Dataset: $D^{tr} = {(x_i^{tr}, y_i^{tr})}^N_{i=1}$ and $D^{te} = {(x_i^{te})}_{i=1}^N$, N is the number of samples.\n\n**Output**: Reconstructed images $\\hat{y}^{te}$ from fMRI $x^{te}$.\n\n\n**Training**:\n\nInitialization: Constructing CLIP [a], VQGAN [b] and LDM [c] by pretrained parameters $\\phi$, $\\gamma$ and $\\theta$.\n\nTraining $\\beta_c$ of **semantic module $M_c$**:\n1. Extracting semantic features $c_i^{tr}$ from $y_i^{tr}$ by CLIP: $c_i^{tr}=f_\n{\\phi}(y_i^{tr})  $.\n2. Training ridge regression parameters $\\beta_c$ by {$c_i^{tr}, x_i^{tr}$.}\n\nTraining $\\beta_s$ of **structural model $M_s$**:\n1. Extracting structural features $s_i^{tr}$ from $x_i^{tr}$ by VQGAN: $s_i^{tr}=f_\n{\\gamma}(y_i^{tr})  $.\n2. Flatting $s^{tr}_i$, and training ridge regression parameters $\\beta_s$ by {$s_i^{tr}, x_i^{tr}$}.\n\n\n**Inference**:\n1. **Semantic module.** Predicting semantics from fMRI: $\\hat{c_i}^{te} = \\beta_cx_i^{te}$. Using momentumn alignment and linear reprojection to get $\\hat{c}^{te}_{i,r}$ (Section 3.2.1).\n2. **Structural module.** Predicting structure from fMRI: $\\hat{s}^{te}_i = \\beta_sx_i^{te}$.\n3. **MOE.** Estimating weighting parameters $\\pi_c$ by KDE and $\\pi_s = 1-\\pi_c$ (Section 3.3.1).\n4. **LDM.** Reconstructing by CS strategy (Section 3.3.2):\n    1. Initializing $z_0$ by Gaussian.\n    2. For $t = 1, 2, ..., T$:\n        1. Conditioned by cross attention: $z_{t-1}' \\sim p_{\\theta}(z_{t-1}'|z_t,\\hat{c}^{te}_{i,r})$.\n        2. Conditioned by CS strategy: $z_{t-1}=z_{t-1}'-\\pi_sF_s(z_{t-1}')+\\pi_s F_s(\\hat{s}^{te}_{i})$\n\n    3. $\\hat{y_i}^{te} = f_{\\gamma}(z_0)$.\n\n[a] Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. \n\n[b] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. \n\n[c] Takagi, Yu, and Shinji Nishimoto. \"High-resolution image reconstruction with latent diffusion models from human brain activity.\" \n\n[d] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis."}, {"review_id": "fjNQFXRtAH", "reviewer": "Reviewer_AK4m", "summary": "This paper presents a new approach to generalized fMRI-to-image reconstruction with a focus in incorporating mage semantics and addressing semantic gaps during the reconstruction. To address inside-space semantic gap, a CLIP based feature space is utilized. To address outside-space semantic gap, a structural information guided diffusion model to transfer semantics. An adaptive strategy to integrate the semantic and structural information is also used. Experiments are conducted on the GOD and NSD dataset with several existing baselines. \n", "questions": "Please respond to the questions raised above.", "limitations": "The authors discussed briefly some limitations associated with the current methodology, without discussing potential future directions to address such limitations or their impact. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well motivated and the methodology components clearly described. The focus on incorporating semantics and addressing semantic gaps is interesting, and the adaptive integration with LDM is novel. \n\nThe experiments are relatively comprehensive, and the margins of improvements especially on GOD appears to be significant. \n\nThe ablation study provides clear evidence on each module of the methodology, especially their adaptive integration. \n", "weaknesses": "It is not clear why did the authors consider only one baseline for comparison on NSD? The performance metrics only lack sufficient statistics on this dataset.\n\nIt is not clear if the training and testing is done separately per subject, or for all three subjects simultaneously but without held-out subjects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer AK4m\n### W1. About only considering one baseline model for comparison on NSD.\nGiven that in Section 4.5, we want to illustrate that following the dataset split method in [a], even a very simple baseline model (k-nearest-neighbor) can achieve a comparatively good decoding result. It could indicate that such a random split cannot ensure that the model has learned the visual representation mechanism and maintain generality. Therefore, we propose a more realistic zero-shot learning (ZSL) division method. The aforementioned baseline methods fail to generalize (Figure 4, 52% in ZSL split), but GESS can generalize well in this complex scenario.\n\nFor sufficient statistics, we provide variance across repeated experiments (10 trials with different seeds) in Table R2 and will provide more details of the experiments in the supplemental materials.\n\n### W2. Details of the training and testing split.\n\nFollowing [b], we train a model on one subject's training trials (1200 samples from 150 categories) and test it on the test trials (50 samples from 50 categories) of the same subject. The training and testing images come from different categories to construct a semantic gap (or ZSL scenario). We repeat our experiments on three subjects to demonstrate the generality and average the performance for the final results.\n\nCross-subject generalization remains challenging. Because brain signals are highly personalized with different signal dimensions and visual area locations across subjects [c], effectively aligning data from multiple subjects into a shared space is difficult. Addressing cross-subject decoding is thus important but complex, beyond the scope of the current work. We leave exploring methods to achieve cross-subject prediction for future work.\n\n\n### Limitations. Discussions of future directions to address the limitations.\n\n(1) About low SNR in fMRI, combining EEG [d] and fMRI [b] may capture better brain signals and provide complementary information, yielding more robust and accurate models for decoding semantic representations. Integrating multi-modal neural data has the potential to significantly advance performance.\n\n(2) Cross-subject generalization is an important next step. Collecting experimental recordings is expensive, and the current models [a] require larger datasets, so that training a model on several subjects' neural signals could provide further gains in accuracy. \n\n(3) More efficient inference strategy. The current model samples one image by multiple time steps which has greater computational cost compared to GANs. With the development of LDM, a more efficient solver is expected.\n\nWe will add more future direction descriptions in the paper. All of the details mentioned above will be added to the paper or supplementary material.\n\n\n### Table R2: Effectiveness of different modules by perceptual similarity (CLIP).\n| Subject | Sub1 (%) | Sub2 (%) | Sub3 (%) | AVG (%) |\n|---------|---------:|---------:|---------:|--------:|\n| Full Method | 78.0±0.7 | 84.8±0.8 | 80.4±0.6 | 81.1±0.7 |\n| w.o.MOE | 69.2±1.0 | 75.2±3.0 | 77.2±1.0 | 73.9±1.7 |\n| w.o. momentum alignment | 63.6±0.1 | 60.2±2.7 | 68.2±0.4 | 64.0±1.1 |\n| w.o. data augmentation | 72.4±3.0 | 76.4±0.6 | 78.2±2.4 | 75.7±2.0 |\n| w.o. CycleGAN feat. | 75.6±1.8 | 78.0±0.8 | 78.0±0.3 | 77.2±1.0 |\n| w.o. linear reprojection | 68.4±1.4 | 70.8±2.7 | 76.2±1.0 | 71.8±1.7 |\n\n## References\n[a] Takagi, Yu, and Shinji Nishimoto. \"High-resolution image reconstruction with latent diffusion models from human brain activity.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[b] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.\n\n[c] Rieck, Bastian, et al. \"Uncovering the topology of time-varying fMRI data using cubical persistence.\" Advances in neural information processing systems 33 (2020): 6900-6912.\n\n[d] Bai, Yunpeng, et al. \"DreamDiffusion: Generating High-Quality Images from Brain EEG Signals.\" arXiv preprint arXiv:2306.16934 (2023).\n"}]

Author rebuttal:
## To Reviewers
We sincerely appreciate all reviewers devoting time for our paper and provide valuable comments. We also feel encouraging that all reviewers agree with our contributions in addressing the semantic gaps, introducing the adaptive confidence-weighted approach, and presenting the specially designed condition strategy for the diffusion process.

We have taken meticulous care in addressing each of the concerns raised by the reviewers. Our responses have been crafted to effectively address the questions and provide comprehensive explanations.

Furthermore, we have included a PDF file that contains additional results (Figure R1 - R2) to substantiate our responses, particularly for aspects of additional ablation studies.


We will merge the details and figures from our responses into both the main text and the supplementary materials. Once again, we extend our gratitude for your valuable feedback, and we firmly believe that these refinements will significantly enhance the quality and impact of our work. 

Meta-review: All reviewers unanimously agree that there are significant merits to this paper. The paper presents an original and well-motivated design for a Generalized fMRI-to-image reconstruction task that takes into account both known and unknown subspaces, offering innovative solutions. The clarity of the paper is evident, as the motivation and methodology components are clearly described. Its significance lies in its unique approach to extracting semantic and structural information while adaptively integrating features to bridge the semantic gap, resulting in vivid reconstructions. This focus on semantics and adaptive integration is particularly novel. The experiments conducted are comprehensive, showing significant improvements.
Decision: Accept (spotlight)