SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tao Fang $^{12}$ , Qian Zheng $^{12,*}$ , and Gang Pan $^{12}$
$^{1}$ College of Computer Science and Technology, Zhejiang University, Hangzhou, China  
 $^{2}$ The State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China {duolafang, qianzheng, gpan}  $@$  zju.edu.cn
Although existing fMRI-to-image reconstruction methods could predict high-quality images, they do not explicitly consider the semantic gap between training and testing data, resulting in reconstruction with unstable and uncertain semantics. This paper addresses the problem of generalized fMRI-to-image reconstruction by explicitly alleviates the semantic gap. Specifically, we leverage the pre-trained CLIP model to map the training data to a compact feature representation, which essentially extends the sparse semantics of training data to dense ones, thus alleviating the semantic gap of the instances nearby known concepts (i.e., inside the training super-classes). Inspired by the robust low-level representation in fMRI data, which could help alleviate the semantic gap for instances that far from the known concepts (i.e., outside the training super-classes), we leverage structural information as a general cue to guide image reconstruction. Further, we quantify the semantic uncertainty based on probability density estimation and achieve Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) within a diffusion process. Experimental results demonstrate that the proposed GESS model outperforms state-of-the-art methods, and we propose a generalized scenario split strategy to evaluate the advantage of GESS in closing the semantic gap. Our codes are available at https://github.com/duolala1/GESS.
Functional magnetic resonance imaging (fMRI) is a powerful tool for studying the human brain and visual system, as it provides a non-invasive way to measure neural activity. Image reconstruction from fMRI data is important for studying visual representation in the cortex and for developing the vivid "reading the mind" brain-computer interface (BCI) technology [10, 19, 11].
High-quality fMRI-to-image reconstruction is a typical cross-modality problem [14, 28] and suffers from severe ill-posedness [2]. Existing state-of-the-art methods leverage the data-driven scheme to address such ill-posed problems by learning data prior from training data. However, training data are often collected as a limited number of instances [19], and real-world images are distributed in a wide, broad semantic space with a long-tail distribution[12]. This brings about the problem of the semantic gap, the semantics of testing instances may be unknown in training stage.
Addressing the semantic gap between the training data collected from the laboratory and the testing instances in the real world helps develop reconstructions for generalized fMRI-to-image scenarios and significantly promote the application of BCI. However, previous methods put too much attention on improving the image quality while less focus on the semantic accuracy of the reconstructed images, which brings two problems. Unstable semantics: The limited samples in each super-class fails to
form a compact feature space, which may result in incorrect decision boundaries and inability to estimate robust semantics even located within the known concepts (Fig.1c). This is what we called the inside-space gap (ISG). Uncertain semantics: The concepts covered by the training set is not enough, resulting in uncertainty in the prediction of test samples with unknown concepts (more like a zero-shot problem), which is called the outside-space gap (OSG). Traditional methods assume that the training set covers all the semantics in the test set (Fig.1b), and ignore the semantic gap caused by unknown samples in reality.
To this end, this paper addresses the generalized fMRI-to-image reconstruction problem by explicitly alleviating the semantic gap. To deal with the instances within known semantic space (ISG problem), we map the fMRI signals to a compact semantic space via a pre-trained Contrastive Language-Image Pre-Training (CLIP) model[23]. To deal with the instances within unknown semantic space (OSG problem), we propose to use the structure information as a transferable cue to guide the reconstruction, which is inspired by the robust and redundant low-level representation in visual cortex[19]. As it is difficult to find a hard boundary to define ISG and OSG cases for a given instance, we quantify its semantic confidence by probability density estimation on the training semantics and adopt the likelihood as the contribution indicator. Finally, we achieve Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) in a diffusion process.
Our contributions in this paper could be summarized as:
- We explicitly address the generalized fMRI-to-image reconstruction problem and formulate its solution as alleviating the semantic gap within known and unknown semantic subspaces.
- We propose a CLIP based method to expand the fMRI features to a compact semantic space to alleviate the inside-space gap, and a structural information guided diffusion model to alleviate the outside-space gap.
- We construct a confidence indicator by quantifying the semantic similarity between a given instance and the training data, based on which we propose GESS to achieve generalized fMRI-to-image reconstruction by adaptively weighting the semantic and structural information.
- Our experimental results demonstrate that the proposed GESS model outperforms the classical and state-of-the-art methods. Additionally, we propose a dataset split method to construct a generalized fMRI-to-image scenario, which allows us to further evaluate the model's generalization ability.
2.1 Image reconstruction from fMRI
Decoding visual signals from the visual cortex has been investigated for a long time, ranging from early manual image feature extraction and linear mapping for prediction[20, 21] to later combining deep learning models (DNNs, GANs, DMs)[25, 15, 27, 10] and continuously improve the effect of generating images. Here we roughly divide the existing decoding methods into three categories:
(1) Reconstructing from the low-level features, such as [20] designs contrast patches as image stimuli and adopts linear mapping to predict from fMRI signals, [21] adopts Gabor features in reconstruction, and [2] directly establishes a bidirectional mapping between fMRI and images based on CycleGAN without explicitly modeling the semantics. Directly reconstructing the structural features is comparatively simple and efficient while suffers from the low quality results.
(2) Explicitly extracting high level conditions like categories for reconstructions. [6] refer to Masked Autoencoders (MAE)[16] to extract high-level features in a self-supervised manner, and [13] explicitly predicts category information by a classification task and uses it as conditions for a diffusion model. Such researches achieve vivid reconstructions due to the well-pretrained image generator, but have not considered the great semantic gap in reality. Obtaining incorrect semantics result in unrelated reconstructions that may look natural but meaningless.
(3) Hybrid methods that extract multiple-level features and jointly use them for reconstruction. [10] extracts high-level semantics and low-level shapes from different visual areas and adopt GAN for
Figure 1: Illustration of the semantic gap in generalized fMRI-to-image reconstruction: (a) Humans have excellent generalization ability for long-tail distributed categories in the real world because they have seen a lot, but for models that have only seen limited fMRI data, it is difficult to generalize to more complex scenarios. (b) Traditional methods implicitly assume that the training set contains the semantics present in the test set. (c) We consider that in a generalized scenario, the training set forms a semantic space called the known space, and there is an intersection between the semantics in the testing set and the known space. In the inside-space scenario, although there may be unstable semantics, if mapped correctly, similar semantics (e.g., felines) can still be found in the neighborhood. However, in the outside-space scenario, the semantic divergence of the testing samples is too large, leading to uncertain or even irrelevant estimated semantics.
reconstruction. [27] also extracts high-level and low-level features as conditions for a diffusion model. [15] injects category information within a GAN reconstruction phase. The mentioned hybrid methods still do not achieve satisfactory results due to incorrect semantics caused by the semantic gap. We propose a hybrid method that not only explicitly extracts semantic and structural information, but also adaptively integrate the features with based on the semantic uncertainty, alleviating the semantic gap and achieving general and vivid reconstructions.
2.2 Large-scale pretrained models
With the development of large-scale models, [24, 8, 23] prove that models with a large enough number of parameters trained with large-scale datasets show more generality and practicality for many downstream tasks like zero-shot identification. CLIP[23, 30] is one pretrained model which predicts a wide range of visual concepts using natural language supervision. It proposes that learning directly from raw text about images provides a broader source of supervision, and scalably learn state-of-the-art image representations from scratch on a dataset of 400 million. We obtain an image-text shared feature space through CLIP, where the point-wise distance indicates the potential semantic relationship, and implicitly provides an extending manifold for the limited training semantics and a potential interpolation ability in the prediction phase. Diffusion models[17] are one type of generative model that decompose the image formation process into a sequential denoising steps. The diffusion process destroys the image structure by gradually adding gaussian noise, and simulate the image generation as an inverse process using a UNet model[17]. Latent Diffusion Models (LDMs) [24] better handle high-resolution images with lower computational complexity in a latent diffusion process. We select LDM for image reconstruction due to the high-resolution and high-quality generation ability,
3.1 Problem Definition and Method Overview
Dividing the brain-image pair data as training  $X^{\mathrm{tr}}$ ,  $Y^{\mathrm{tr}}$  and testing set  $X^{\mathrm{te}}$  with the  $Y^{\mathrm{te}}$  are unknown, we aim to decode accurate and natural visual stimulus  $\hat{Y}^{\mathrm{te}}$  from the testing fMRI data  $X^{\mathrm{te}}$  and make  $\hat{Y}^{\mathrm{te}} \approx Y^{\mathrm{te}}$ . The visual features can be divided into two components: semantic information
Figure 2: Framework of GESS. The semantic module  $\mathrm{M_c}$  maps fMRI data onto a semantic space using CLIP and then reprojects it onto the dense manifold to obtain faithful semantics  $c^{\mathrm{te}}$ . Structural module  $\mathrm{M_s}$  extracts low-frequency features as structural information  $s^{\mathrm{te}}$ . Gating function  $\mathrm{M_g}$  adaptively assigns  $\pi_{c}$  for  $c^{\mathrm{te}}$  based on the semantic uncertainty, and assigns  $\pi_s = 1 - \pi_c$  for  $s^{\mathrm{te}}$ . LDM  $\mathrm{M_{dm}}$  reconstructs images conditioned on the two types of features weighted by  $\pi_c, \pi_s$ .
$c^{n \times d_c}$  and structural information  $s^{n \times d_s}$  that shared between fMRI and images (where  $n$  denotes the number of samples,  $d_c, d_s$  denotes the feature dimension). Here  $c$  is derived by a semantic module  $\mathrm{M_c}$  using CLIP (with parameter  $\phi$ ), and  $s$  is derived by a structural module  $\mathrm{M_s}$  using VQGAN (with parameter  $\gamma$ ). To describe the generalized fMRI-to-Image scenario, letting  $\mathcal{R}^d$  be the universal set of semantic space with  $d$  dimension we get  $c^{\mathrm{tr}} \in \mathcal{R}^{\mathrm{tr}}$  as the subspace of the training semantic features (that we call known space) and  $c^{\mathrm{te}} \in \mathcal{R}^{\mathrm{te}}$  is the subspace of the testing semantic features and  $\mathcal{R}^{\mathrm{tr}} \subset \mathcal{R}^d, \mathcal{R}^{\mathrm{te}} \subset \mathcal{R}^d$ . As in Fig.1c, in the generalized fMRI-to-Image scenario, there is overlapping that  $\mathcal{R}^{\mathrm{tr}} \cap \mathcal{R}^{\mathrm{te}} \neq \emptyset$  and  $\mathcal{R}^{\mathrm{te}} \not\subseteq \mathcal{R}^{\mathrm{tr}}$  (different from traditional assumption that  $\mathcal{R}^{\mathrm{te}} \subseteq \mathcal{R}^{\mathrm{tr}}$ ). To avoid confusion, from now on, when referring to space, it will be in the context of the training semantic space. Due to the different uncertainties of  $c^{\mathrm{te}} \in \mathcal{R}^{\mathrm{tr}}$  and  $c^{\mathrm{te}} \notin \mathcal{R}^{\mathrm{tr}}$  across samples caused by the generalized scenario, we introduce a gating function  $\pi_c(c^{\mathrm{te}})$  and  $\pi_s(s^{\mathrm{te}})$  in mixture of experts (MOE), to adaptively assign confidence for semantic and structural information. With the aforementioned components and their estimated weights, we obtain reconstructed images  $Y^{\mathrm{te}}$  from a conditioned latent diffusion model,  $f_{\theta}(c^{\mathrm{te}}, s^{\mathrm{te}}, \pi_c, \pi_s)$ .
In this paper we propose a two-stage decoding model: (1) To alleviate the semantic gap, we implement a semantic module  $(\mathrm{M_c}$  in Fig.2) that maps fMRI data onto a dense and continuous manifold constructed by CLIP and a structural module  $(\mathrm{M_s})$  to extract low-frequency features as a transferable clue. (2) To determine the ISG/OSG situation, we propose  $\mathrm{M_g}$  to estimate the inside-space likelihood to implicitly assess the situation and combine the two types of features using a weighted assignment approach in a diffusion process  $(\mathrm{M_{dm}})$ .
3.2 Semantic gap alleviation
The visual concepts are long-tail distributed in the real world so that the semantics that collected in a limited BCI experiment could not cover enough real world concepts. To achieve reconstruction in the generalized fMRI-to-image scenario, we further divided the semantic gap into two cases, which have been addressed by modules  $\mathrm{M_c}$ ,  $\mathrm{M_s}$  respectively.
3.2.1 Semantic module  $\mathrm{M_c}$ : inside-space case
Even in the known space  $\mathcal{R}^{\mathrm{tr}}$ , there may not be enough samples to continuously represent the space due to the limited number of fMRI samples. Gaps in this subspace without instance filling may result in classification failure, even if a sample has similar concepts (super-classes) in the training set. This is referred to as the inside-space gap (ISG). As ISG is still a problem within the same domain, we propose that the fundamental reason lies in the discontinuous manifold directly constructed from the limited number of fMRI samples. As mentioned before, the CLIP feature space shows a locally continuous and semantically reasonable manifold (i.e., the cosine similarity reveals semantic relationships). By appropriately projecting the fMRI samples onto this manifold, it becomes possible to implicitly extend the discontinuous fMRI features into a manifold that covers semantics of both of the collected and unseen fMRI samples.
Concretely we use a linear model (ridge regression [18]) to predict  $c$  from  $x$  with large regularization  $\lambda_{c}$  to limit the impact of noise on predicting semantics. We get  $c^{\mathrm{tr}} = f_{\phi}(x^{\mathrm{tr}})$  and fit the coefficients  $\beta_{c}$  of ridge regression with  $\phi$  of CLIP fixed. Although linear models and regularization may reduce the diversity of semantics and make it difficult to capture non-linearity, the simple relation assumption shrinks the coefficient variance and makes the model less prone to over-fitting to the noise.
However, the mean squared error (MSE) and ridge regression cannot guarantee that the estimated  $c^{\mathrm{te}''}$  are mapped onto the low-dimensional manifold in the high-dimensional space. Therefore, we propose two methods to alleviate this problem.
Momentum alignment We propose to align the statistics of the predictions and the image prior distributions to reduce semantic domain shift from a distributional perspective. To get the statistics of the manifold, we first sample  $C^{\mathrm{tr}}$  from the manifold by feeding the training images into CLIP models. However, the scarcity of training samples makes it difficult to accurately describe the manifold. Here we use augmentation  $C^{\mathrm{aug}} = f_{\phi}(X^{\mathrm{aug}})$  to help the manifold sampling (let  $C^{\mathrm{all}} = C^{\mathrm{tr}} \cup C^{\mathrm{aug}}$ ) and to satisfy the local linearity to some extent. Previously we get  $C^{\mathrm{te}''}$  by ridge regression  $\beta_{c}$ , and here we adjust its statistics by using a simple domain adaptation method named first-order momentum alignment like [13], which involves a whitening and an alignment process to get  $C^{\mathrm{te}'}$  which has the same mean and variance as  $C^{\mathrm{all}}$  (supplementary materials 6.2).
Linear re-projection Here we directly project  $C^{\mathrm{te}'}$  onto the manifold. We assume that in a small local area of the manifold, there are enough feature points such that a manifold point can be approximated as a linearly weighted sum of its K-nearest neighbor feature points (Fig. 2). The projection principle is that we re-project the object point  $c^{\mathrm{te}'}$  to  $c^{\mathrm{te}}$  using its K-nearest points  $c_k^{\mathrm{nb}} \subset \mathcal{K}$  on the manifold  $(\mathcal{K} \subset C^{\mathrm{all}})$  by a linear combination model:  $c^{\mathrm{te}} = \sum_{\mathcal{K}} w_k^{\mathrm{nb}} \cdot c_k^{\mathrm{nb}}$ , where  $w_k^{\mathrm{nb}}$  is the coefficients of each within-manifold points  $c_k^{\mathrm{nb}}$ . Here we solve a constrained least-squares problem:
$$
\underset {w _ {k} ^ {\mathrm {n b}}} {\arg \min } \| c ^ {\mathrm {t e}} - w _ {k} ^ {\mathrm {n b}} \cdot c _ {k} ^ {\mathrm {n b}} \| _ {2} ^ {2}, \text {s . t .} w _ {k} ^ {\mathrm {n b}} = 1, \tag {1}
$$
to get the  $c^{\mathrm{te}}$  that located in manifold for latter reconstruction (supplementary materials 6.1). This approach [5] essentially considers that the information that causes the components to deviate from the manifold is not important and could be removed.
3.2.2 Structural module  $\mathrm{M_s}$  : outside-space case
Although previous work has also considered extracting semantics [27], it has not addressed the extreme scenario where the image semantics are distributed outside the known space, which is hard to get trustworthy predictions. We call it outside-space gap (OSG). For a generalized scenario, we propose to use structural information  $S^{\mathrm{te}}$  to compensate for the insufficient generalization caused by solely depending on semantics. That is because structural information is a category-independent attribute, and is robustly and redundantly represented in multiple cortex areas[19]. We use the low quality  $S^{\mathrm{te}}$  to supplement the uncertain semantics in reconstructions.
To extract the low-frequency components as  $S$ , before fitting we conduct Gaussian filtering on visual stimuli Gaussian( $Y^{\mathrm{tr}}$ ,  $r$ ) with a large kernel  $r = 15$  to prevent the model from trying to extract the high-frequency components from fMRI. Besides this can be seen as to suppress the variance of the predictions and to some extent, suppress the noise. In this task a simple ridge regression could
already achieve acceptable  $s^{\mathrm{te}}$ , which is fast and easy to implement. Concretely, we use a pretrained VQGAN to extract the ground truth representation  $\hat{s}^{\mathrm{tr}} = f_{\gamma}(y^{\mathrm{tr}})$  of each image. Then, with a ridge regression  $\beta_{s}$ , we can establish the relationship between fMRI and the latent variables by minimizing  $|f_{\beta_s}(X^{\mathrm{tr}}) - \hat{S}^{\mathrm{tr}} + \lambda \beta_s|$ , where  $\hat{S}^{\mathrm{tr}}$  has been flattened here. Another approach is to use CycleGAN [2] that trained on  $Y^{\mathrm{tr}}$  to generate  $S^{\mathrm{te}}$ , which shows higher quality. Since a large number of epochs are needed to fit the model, it is computationally more expensive, so that the above two methods represent a trade-off between quality and efficiency.
3.3 Adaptive Integration with LDM
3.3.1 Gating function  $\mathrm{M_g}$  for Integration
Although in the first part, we have explicitly divided the problem into the ISG and OSG sub-problems, and construct the faithful manifold combined with structural information to alleviate the semantic gap, but in reality it is difficult to determine the boundary between the ISG and OSG cases. As the boundary, probably determined by a threshold, would introduce an extra hyper-parameter which is inaccessible for the testset. Considering that different features have their own strengths and inspired by mixture of experts (MOE [3]), we regard  $\mathrm{M_c(x)}$  and  $\mathrm{M_s(x)}$  as two experts, and use a gating function  $\pi_c(x),\pi_s(x)$  to indicate each expert's own regions within  $X^{\mathrm{te}}$ . We assume that there exists an integrated intermediate information  $h^{\mathrm{te}}$  which could be marginalized as a weighted sum of:
$$
p \left(h _ {i} ^ {\mathrm {t e}} \mid x _ {i} ^ {\mathrm {t e}}\right) = \pi_ {c} \left(x _ {i} ^ {\mathrm {t e}}\right) p \left(c _ {i} ^ {\mathrm {t e}} \mid x _ {i} ^ {\mathrm {t e}}\right) + \pi_ {s} \left(x ^ {\mathrm {t e}}\right) p \left(s _ {i} ^ {\mathrm {t e}} \mid x _ {i} ^ {\mathrm {t e}}\right), s. t. \pi_ {s} + \pi_ {c} = 1, \tag {2}
$$
where  $i = \{1, \dots, n\}$ ,  $\pi_s$  and  $\pi_c$  are gated functions of the semantic and structural modules which works as adaptive weights for each component, and  $p_c(c_i^{\mathrm{te}} | x_i^{\mathrm{te}})$ ,  $p_s(s_i^{\mathrm{te}} | x_i^{\mathrm{te}})$  describes the intermediate features derived from the two modules respectively.
Here we propose to estimate the gating function in the form of likelihood of inside known space as  $\pi_c(x_i^{\mathrm{te}}) = p(c_i^{\mathrm{te}}|C^{\mathrm{tr}})$  and compute  $\pi_s(x_i^{\mathrm{te}}) = 1 - \pi_c(x_i^{\mathrm{te}})$ . This could be interpreted as samples that are far away from the manifold that the training semantics covered are more uncertain. Concretely, we propose a non-parametric density estimation method based on kernel density estimation (KDE[3] with bandwidth=1.5, and then normalized to [0, 1]), which uses training samples to estimate the underlying probability distribution to approximately estimate  $p(c_i^{\mathrm{te}})$  on  $C^{\mathrm{tr}}$  (supplementary materials 6.3).
3.3.2 Diffusion model  $\mathrm{M_{dm}}$  for generation
Considering the powerful generality of LDM, we propose to implement MOE within the diffusion process. But directly taking the conditions in the form of cross attention inputs is not appropriate as it is difficult to realize feature weighting and a structural guidance containing spatial information. For MOE model, we need a common space to integrate the different modalities of  $c^{\mathrm{te}}$  and  $s^{\mathrm{te}}$ . We first consider weighting in the gradient space and introduce a gradient guided method (GG for short), and inspired by [8] we get:
$$
p _ {\theta , \phi} \left(z _ {i, t} \mid z _ {i, t + 1}, h _ {i}\right) = A p _ {\theta} \left(z _ {i, t} \mid z _ {i, t + 1}\right) p _ {\eta} \left(h _ {i} \mid z _ {i, t}\right), \tag {3}
$$
where  $A$  is the normalization term,  $\theta, \eta$  denote the parameters of the UNet and the guidance module (like a classifier),  $z_{i,t}$  means the latent features of sample  $i$  at time step  $t$  in the LDM. For the sake of readability, we omit the 'i' and 'te' tags in the following. As mentioned before we have estimated the confidence  $\pi_c$  and  $\pi_s$  from  $x_i$  and get  $p_{\eta}(h|z_t) = \pi_c p_c(c|z_t) + \pi_s p_s(s|z_t)$  with the confidence fixed during the diffusion process. By Jensen's inequality we obtain the lower bound of log likelihood: