SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \log \left(p _ {\theta} \left(z _ {t} \mid z _ {t + 1}\right) p _ {\phi} (h \mid z _ {t})\right) = \log \left(p _ {\theta} \left(z _ {t} \mid z _ {t + 1}\right)\right) + \log \sum_ {m} \pi_ {m} p _ {m} (m \mid z _ {t}) (4) \\ \geq \log \left(p _ {\theta} \left(z _ {t} \mid z _ {t + 1}\right)\right) + \sum_ {m} \pi_ {m} \log p _ {m} \left(h _ {m} \mid z _ {t}\right), m \in \{c, s \}, (5) \\ \end{array}
$$
maximizing which across the diffusion process is equivalent to maximizing the log likelihood, in this way we realize the instance-wise adaptive weighting of different image features in the diffusion process. Further with Taylor expansion as in [8], we get an approximation of the likelihood:
$$
\log p _ {m} \left(h _ {m} \mid z _ {t}\right) \approx \left(z _ {t} - \mu_ {t}\right) g _ {m} + \mathrm {B}, m \in \{c, s \}, \tag {6}
$$
wherein the gradient information  $g_{i}$  comes from a loss term in the guidance module  $\eta$  that measure accuracy of a certain decoded feature, and  $\mathrm{B}$  is a constant term. In this case the  $g_{i=c}$  comes from guidance of CLIP features  $C^{\mathrm{te}}$  and  $g_{i=s}$  comes from  $S_{t}$  with a  $\mathcal{L}_{\mathrm{mse}}$  guidance. Known that the latents  $z$  means a perceptual compression[24, 9] of raw image, it still retains certain image properties that performing image manipulations such as kernel smoothing in the  $z$ -space makes sense. Here we apply low-pass filter  $\mathrm{F}(z)$  to extract structural information from  $z_{t}$  and get  $\mathcal{L}_{\mathrm{mse}} = ||\mathrm{F}(z_0') - \mathrm{F}(s)||_2^2$ , where  $z_0'$  are estimated from  $z_{t}$  [26]. One advantage of gradient guidance lies in its flexibility, like combination of image variance loss and other optimizations to improve the quality of the generated images.
However, GG method requires computing the gradients at each timestep, which greatly increases computational complexity. Inspired by ILVR [7], which can directly replace different frequency components in the image-space, we propose an efficient component substitution method (CS for short). Firstly we propose the component segmentation method:
$$
z _ {t} = \sum_ {m} \mathrm {F} _ {m} \left(z _ {t}\right), m \in \{c, s \}, \tag {7}
$$
where  $\mathrm{F}_m$  means the filtering functions for different levels of information (e.g., high or low frequency information). In this work we have two-level features: semantics  $\mathrm{F}_c$  and structure  $\mathrm{F}_s$ .  $\mathrm{F}_s(z_t)$  can be obtained through low-pass filtering function and get  $\mathrm{F}_c(z_t) = z_t - \mathrm{F}_s(z_t)$ . We compute each frequency component of  $z_t$  as a weighted combination of various experts:  $\mathrm{F}_m(z_t) = \pi_s\mathrm{F}_m(z_{t,s}) + \pi_c\mathrm{F}_m(z_{t,c})$ , where  $z_{t,c},z_{t,s}$  means the latent features derived from the  $c,s$  conditions respectively. However, for the structural guidance  $\mathrm{F}_c(z_{t,s})$  does not contain accurate semantics that the  $\pi_c\mathrm{F}_c(z_{t,s})$  term does not make sense. Thus we only focus on the low-frequency component here. Referring to [7] we get  $z_{t-1}$  from  $z_t$ :
$$
z _ {t - 1, c} = \sim p _ {\theta} \left(z _ {t - 1} \mid z _ {t}, c\right), \tag {8}
$$
$$
z _ {t - 1} = z _ {t - 1, c} - \pi_ {s} \mathrm {F} _ {s} \left(z _ {t - 1, c}\right) + \pi_ {s} \mathrm {F} _ {s} \left(z _ {t - 1, s}\right), \tag {9}
$$
where we compute  $z_{t - 1,s}$  by gradually adding noise in  $s$  that has been estimated by  $\mathrm{M_s}$ . Since the generation process does not require gradient estimation, it is faster and requires less computational resources compared to the previous method. However, the generated images sometimes become blurry (possibly due to the discontinuous low-level substitution that may disrupt the image structure). Therefore, the two methods are a trade-off between efficiency and image quality.
3.4 Training details
In our model only the ridge regression modules of  $\mathrm{M_s}$ ,  $\mathrm{M_c}$  need to be trained, while the other modules are pretrained on large-scale data with fixed parameters. The overall process of our method is described in Algorithm 1.
Semantic information. To extract semantic features, we use the pretrained VIT-L/14 clip model on 1200 training images and about 4000 augmentation images. We use ridge regression trained with  $\lambda_{c} = 1000$ . We slightly smooth the images with a Gaussian kernel ( $r = 5$ ) before extracting semantic features.
Structural information. To extract structural features, images are pre-smoothed with a Gaussian kernel  $(r = 15)$ . In experiments we use CycleGAN with the same setting as [2]. We do not use augmentation as [2] because high-quality details are not necessary here.
Generating. We adopt the pretrained LDM in [4](knn2img model). During integration, both of the two methods mentioned in section 3.3 are implemented and compared in section 4.4, and we finally select CS method considering its efficiency and acceptable generation. A temporally scaling strategy is used in diffusion process to reduce the influence of  $z_{t,s}$  and results in clearer generated images. We use DDIM [26] acceleration during sampling with 50 time steps.
Algorithm 1 The pseudo code of GESS.
Input: Paired fMRI  $X$  and Image  $Y$  Dataset:  $\mathcal{D}^{\mathrm{tr}} = \{(x_i^{\mathrm{tr}},y_i^{\mathrm{tr}})\}_{i = 1}^n$  and  $\mathcal{D}^{\mathrm{te}} = \{(x_i^{\mathrm{te}})\}_{i = 1}^n$ .
Output: Reconstructed stimulus images  $\hat{y}^{\mathrm{te}}$  from fMRI  $x^{\mathrm{te}}$ .
Initialization: Constructing CLIP [23], VQGAN [9] and latent diffusion model [24] by pretrained parameters  $\phi$ ,  $\gamma$  and  $\theta$ .
Training parameter  $\beta_{c}$  of semantic module  $\mathrm{M_c}$ :
1. Extracting semantic features  $\hat{c}_i^{\mathrm{tr}}$  from  $y_{i}^{\mathrm{tr}}$  by CLIP:  $\hat{c}_i^{\mathrm{tr}} = f_\phi (y_i^{\mathrm{tr}})$
2. Training ridge regression parameters  $\beta_{c}$  by  $\hat{c}_i^{\mathrm{tr}}$  and  $x_{i}^{\mathrm{tr}}$ :  $\beta_{c} = (\hat{C}^{T}\hat{C} +\lambda I)^{-1}\hat{C}^{T}X^{\mathrm{tr}}$  where  $\hat{C} = [\hat{c}_1^{\mathrm{tr}},\hat{c}_2^{\mathrm{tr}},\dots,c_{\hat{N}}^{\mathrm{tr}}]$  and  $X^{\mathrm{tr}} = [x_1^{\mathrm{tr}},x_2^{\mathrm{tr}},\dots,x_N^{\mathrm{tr}}]$ .
Training parameter  $\beta_{s}$  of structural model  $\mathrm{M_s}$ :
1. Extracting structural features  $\hat{s}_i^{\mathrm{tr}}$  from  $x_{i}^{\mathrm{tr}}$  by VQGAN:  $\hat{s}_i^{\mathrm{tr}} = f_\gamma (y_i^{\mathrm{tr}})$
2. Flatting  $\hat{s}_i^{\mathrm{tr}}$ , and training ridge regression parameters  $\beta_{s}$  by  $\hat{s}_i^{\mathrm{tr}}$  and  $x_{i}^{\mathrm{tr}}$ :  $\beta_{s} = (\hat{S}^{T}\hat{S} + \lambda I)^{-1}\hat{S}^{T}X^{\mathrm{tr}}$ , where  $\hat{S} = [\hat{s}_1^{\mathrm{tr}},\hat{s}_2^{\mathrm{tr}},\dots,\hat{s}_N^{\mathrm{tr}}]$ .
Inference:
1. Semantic module Predicting semantics from fMRI:  $c_i^{\mathrm{te}''} = \beta_c x_i^{\mathrm{te}}$ . Using momentum alignment to get  $c^{\mathrm{te}'}$  (section 3.2.1). Using linear re-projection to get  $c_i^{\mathrm{te}}$  (section 3.2.1).
2. Predicting structure from fMRI:  $s_i^{\mathrm{te}} = \beta_s x_i^{\mathrm{te}}$ .
3. MOE Estimating weighting parameters  $\pi_c$  by KDE and  $\pi_s = 1 - \pi_c$  (section 3.3.1).
4. Latent diffusion process by CS strategy (section 3.3.2):
(a) Initializing  $z_0$  by Gaussian.
(b) For  $t = 1,2,\dots,T$
i. Conditioned by cross attention:  $z_{t-1,c} \sim p_{\theta}(z_{t-1}|z_t, c^{\mathrm{te}})$
ii. Conditioned by CS strategy:  $z_{t - 1} = z_{t - 1,c} - \pi_s\mathrm{F}_s(z_{t - 1,c}) + \pi_s\mathrm{F}_s(z_{t - 1,s})$
(c)  $y_{i}^{\mathrm{te}} = f\gamma (z_{T})$
4.1 Dataset
We evaluated the performance of our model using two datasets: the General Object Decoding (GOD) dataset [19] and the Natural Scenes Dataset (NSD)[1]. The GOD dataset contains 1200 images from 150 categories for training and 50 images from 50 categories for testing. NSD uses images from the COCO dataset and roughly 10,000 fMRI-image pairs for one subject. During a visual experiment, researchers recorded subjects' brain activities by fMRI scanners while they are watching images on a screen. In GOD, there was no overlapping between the training and testing classes which matches the generalized scenario (supplementary materials 7.1).
However, the random split method used in [27] for NSD causes too much overlap between the training and testing semantics (Fig.4a left), so that a simple baseline model (Fig.4a left) by nearest searching can achieve comparable performance. To address this issue, we proposed a generalized split strategy for NSD in Section 4.4. We showed that while the simple baseline model performs worse under this strategy, our model is still able to achieve stable reconstruction in this situation.
4.2 Evaluation of image similarity
Comparing and estimating image similarity is a challenging task in computer vision as several factors like object dislocation, different viewpoints greatly increases pixel-wise distance like L2. To measure the similarity between the reconstructed and ground truth visual stimulus, we propose a combined comparison criteria that builds upon previous works with slight modifications. Specifically, we use: (1) a 2-way comparison task as in [19] with three different metrics, including RMSE, Spectral Angle Mapper(SAM[29], a physically-based metric), and a perceptual similarity metric based on a
pretrained CLIP (VIT-L/14), to make an overall assessment of reconstruction accuracy, and (2) visual comparison of the reconstructed images from different methods. Besides to remove the randomness caused by pairwise comparison, we repeat and averaged 10 trials for each sample.
We compare our method with several existing methods on GOD: GAN-based methods include [2], which use CycleGAN to directly decode images from fMRI data with augmentation and [22] use ICGAN as backbones. Diffusion-based methods include [27] use MAE to extract features unsupervisedly as conditions and inputs into a latent diffusion model, and [13] predict image labels from fMRI and use a diffusion model to reconstruct the image conditioned on the category information. Most previous methods approach LDM as a purely generative model without exercising tight control over the diffusion process.
4.3 Performance comparison
Figure 3: Performance comparison. Left: visual comparison of reconstructed images from different methods (more examples could be found in supplementary materials). Right: quantitative comparison of different methods by perceptual similarity.
We performed a quantitative and visual comparison between our model and several candidate models using the GOD dataset on three subjects  $(1\sim 3)$ . We used 1200 and 50 samples as the training and testing samples, respectively. Each sample has been averaged between 5 and 24 samples to improve signal-to-noise ratio. We also randomly sampled around 4000 images from ImageNet for augmentation. The final results are shown visually and quantitatively in Fig.3. We outperform the recent published methods [22, 27] by a large margin (3.8%, 13.8% by perceptual similarity, 7.2%, 5.3% by SAM, 1.1%, 16.3% by RMSE). As can be seen in Fig.3 left we generate high-quality and high-resolution  $(768\times 768)$  images that are more realistic in terms of their content than others. By explicitly taking the generalized scenario into account, our method outperforms the other methods visually and quantitatively.
4.4 Ablation study
Table 1: Abalation study on GOD dataset (Perceptual similarity).
Subjects	cte+LDM	cte+s+CS	cte+s+MOE+CS	cte+s+MOE+GG
Sub1	74.2%	69.2%	78.0%	76.2%
Sub2	77.6%	75.2%	84.8%	76.6%
Sub3	79.6%	77.2%	80.4%	82.6%
AVG	77.1%	73.9%	81.1%	78.4%
To evaluate the usefulness of the components in our approach, we conducted an ablation study on the GOD dataset. We compared the performance of different combinations of our components, mainly to evaluate the influence of semantic and structural information on reconstruction, and the importance of MOE. As shown in Table 1, equal-weighted feature integration without considering the semantic uncertainty may even lead to negative optimization. When the structural information is weighted as in columns 4 and 5, it can improve reconstruction accuracy by  $4\%$  and  $1.3\%$  on average compared to using only semantics, respectively. It can also be observed that the CS method achieves higher accuracy than the GG method, even though the latter produces more natural-looking images. In the
supplementary materials we provide detailed evaluations of both the individual effectiveness of each component and the associated computational costs (in Table 4 and Table 5).
4.5 A generalized split strategy
Figure 4: Left: visualize the difference of scenario in semantic space. Our model achieves stable reconstructions across scenarios visually. Right: quantitative results of baseline model and our model on different scenarios. RS:random scenario;GS:generalized scenario.
Method	SAM	RMSE	Perceptual
base+RS	60.8%	62.3%	81.0%
base+GS	50.8%	48.5%	52.0%
ours+RS	66.8%	63.5%	88.8%
ours+GS	60.5%	64.5%	63.3%
We emphasize that the generalized fMRI-to-Image setting is crucial in forcing the model to learn the visual mechanism of the brain, instead of relying on a pattern matching strategy that simply retrieves from the database. Here we propose a simple generalized scenario split strategy on NSD. We first encode the images into a semantic space and then cluster the images into several clusters (in our case 20 clusters) by K-means to obtain pseudo labels. Then we construct a graph with the cluster centers as nodes and their cosine similarity as edges, and apply a minimum cut algorithm to split the graph into two parts with the minimum number of disconnected edges. As visualized in TSNE in Fig.4 middle, a comparatively larger concept gap exists between the training and testing sets in the semantic space.
As in Fig.4, we propose a simple retrieval model based on k-nearest-neighbor (KNN) on NSD as a baseline model (supplementary materials 7.2), which can generate relatively accurate reconstructions based on  $1_{st}$  neighbor on the dataset that has been randomly split as [27] (Fig.4 left). Its performance deteriorates in a generalized experimental setting (Fig.4 right). Here we demonstrate that in such generalized scenario, our method still perform more stable reconstructions, even though the reconstruction accuracy has decreased by  $28.4\%$  compared to the baseline model  $(37.0\%)$ .
4.6 Discussion
In this paper, we explicitly define the generalized fMRI-to-image scenarios and the great semantic gaps that they bring. To explicitly address the semantic gap, we decompose it into inside-space and outside-space cases and propose a model called GESS (Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information) that adaptively integrates semantic and structural information within a diffusion process. Our experimental results demonstrate that GESS achieves state-of-the-art performance.
**Limitations.** Our model still suffers from several limitations including: (1) The significant noise exists in fMRI signals, which may cause variability in the quality of generated images. This is a common challenge in fMRI-based visual reconstruction and requires further research to improve robustness. (2) Sampling one image over multiple time steps in LDM still has greater computational cost compared to GANs. (3) The current model needs retraining for each specific subject and cannot handle cross-subject data. As collecting experimental recordings is expensive, designing a model capable of cross-subject prediction could reduce the difficulty of applying it on new subjects.
This work was supported by STI 2030 Major Projects (2021ZD0200400), Natural Science Foundation of China (No. 61925603, U1909202), the Key Research and Development Program of Zhejiang Province in China (2020C03004).
[1] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[2] Roman Belyi, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. In Advances in Neural Information Processing Systems, pages 6514-6524, 2019.
[3] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[4] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi-parametric neural image synthesis. In Advances in Neural Information Processing Systems, 2022.
[5] Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. Deepfacedrawing: Deep generation of face images from sketches. ACM Transactions on Graphics (TOG), 39(4):72-1, 2020.
[6] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06956, 1(2):4, 2022.
[7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938, 2021.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[10] Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape-semantic gan. Advances in Neural Information Processing Systems, 33:13038-13048, 2020.
[11] Tao Fang, Qian Zheng, Yu Qi, and Gang Pan. Extracting semantic-dynamic features for long-term stable brain computer interface. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5965-5973, 2023.
[12] Anja Feldmann and Ward Whitt. Fitting mixtures of exponentials to long-tail distributions to analyze network performance models. Performance evaluation, 31(3-4):245-279, 1998.
[13] Matteo Ferrante, Tommaso Boccato, and Nicola Toschi. Semantic brain decoding: from fmri to conceptually similar image reconstruction of visual stimuli. arXiv preprint arXiv:2212.06726, 2022.
[14] Prosper Agbesi Fiave, Saloni Sharma, Jan Jastorff, and Koen Nelissen. Investigating common coding of observed and executed actions in the monkey brain using cross-modal multi-variate fmri classification. Neuroimage, 178:306-317, 2018.