<figure><img src="image_1.png" /></figure>

HAL
open science

# VAE-Based Generic Decoding via Subspace Partition and Priori Utilization

Mingyang Sheng, Yongqiang Ma, Kai Chen, Nanning Zheng

► To cite this version:

Mingyang Sheng, Yongqiang Ma, Kai Chen, Nanning Zheng. VAE-Based Generic Decoding via Subspace Partition and Priori Utilization. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.220-232, 10.1007/978-3-031-34107-6_18. hal-05498086

HAL Id: hal-05498086

https://inria.hal.science/hal-05498086v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
VAE-Based Generic Decoding
via Subspace Partition and Priori
Utilization

Mingyang Sheng, Yongqiang Ma, Kai Chen, and Nanning Zheng(✉)

Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University,
Xi'an 710049, Shaanxi, China
smysmy2016@stu.xjtu.edu.cn, nnzheng@mail.xjtu.edu.cn

**Abstract.** Generic decoding is a challenging problem in visual neural decoding. The existing methods based on generative models ignore the application of prior knowledge, which leads to poor interpretability, and few pay attention to fMRI (functional Magnetic Resonance Imaging) processing. To tackle these problems, a novel framework for generic decoding has been proposed named GD-VAE. GD-VAE is based on Variational Auto-Encoder (VAE) which is capable of meaningful latent space, and contains four modules: feature extractor, feature VAE, Prior Knowledge Network (PKN) and Latent Space Disentangling Network (LSDN). The feature extractors extract features of raw visual and cognitive data, and feature VAE implements decoding with a shared latent space for both modalities. The PKN and LSDN constrain the latent space of VAE with delicate structure, in order to apparently reveal the information in the subspace. Benefiting from these modules, the alignment between visual and cognitive modality can be achieved, and greater interpretability can be acquired. Experiments on Generic Decoding Dataset validate the effectiveness and interpretability of the proposed method.

**Keywords:** Generic decoding · Variational Auto-Encoder · Subspace disentanglement

# 1 Introduction

One of the most important challenges in the field of neuroscience and artificial intelligence is to understand how the brain works, which is inseparable from “neural decoding”. Visual brain decoding is an important branch of neural decoding, aiming to extract information of visual tasks from brain activities in visual cortices, of which signal contains useful message highly related with visual perception [10]. With the development of neuroimaging technology such as fMRI (functional magnetic resonance imaging), we can obtain enough indirect or direct reflection of brain activity.

According to the granularity of decoding, visual brain decoding can be divided into visual reconstruction and visual identification [14]. The former one
implements on the image reconstruction from pixel to pixel, while the latter one focuses on image classification or identification based on decoding features. Compared with visual reconstruction, the advantage of visual identification lies in a better understanding about how the brain generates object-based information, which is usually more directly related to our visual guidance behavior than raw images in the real world. This paper is based on an extended paradigm of visual identification, namely the generic decoding paradigm [9], which requires decoding the corresponding image representation from the given fMRI data, comparing the similarity with the image features of all categories in the candidate set, and selecting the most similar category as the classification result. The categories in training set and those in test set do not intersect at all, which is also known as zero-shot learning.

Currently, generic decoding is still in its infancy. This paradigm was first proposed by Horikawa et al. [9]. It uses sparse linear regression model to learn the mapping from selected fMRI voxels to image features. Recently, some methods have applied the generation model to this problem, using probability graph model to model the data generation process with the assistance of semantic information [1, 3, 7]. In addition, DGMM combines probability graph model with neural network [5]. There are still a few literatures regarding it as multi-view learning, learning the linear or nonlinear mapping of two modality data to the shared feature space.

These methods ignore the processing of fMRI data and feature extraction. The high dimensional and strong noise characteristics of fMRI data make it necessary to extract useful information for decoding. Besides, insufficient interpretability of the model remains further consideration. The trivial latent space in generative model does not completely disentangle different information. We still know little about the information encoded in the latent space.

In this paper, we design a VAE-based visual decoding framework, which consists of four modules: visual and cognitive feature extractors, feature VAEs, prior knowledge network (PKN) and Latent Space Disentangling Network (LSDN). Feature extractors extract features of input images and raw fMRI data, and feature VAEs are used to create a meaningful latent space. In order to generate a more interpretable latent space, latent space is divided into two subspaces and default prior distribution (normal distribution) is substituted by the distribution encoded by PKN from prior knowledge. Meanwhile, latent space disentangling network helps regularize the two subspace to be disentangled. Our main contributions are as follows:

- A VAE based decoding framework is proposed, which is the first deep learning based framework for generic decoding.

- The designed fMRI feature extractor incorporates prior knowledge about brain connections and data, and is more general and reasonable for decoding.

- We propose prior knowledge network and latent space disentangling network for a more interpretable latent space. Experiments verify the effectiveness of the method.
Other sections are organized as follows. Section 2 summarizes related work about generic decoding, zero shot learning and VAE. The proposed framework and method are explained in Sect. 3. Experiment results are shown in Sect. 4. Section 5 concludes our contribution and prospect for future work.

# 2 Related Work

## 2.1 Generic Decoding

Generic decoding was formally proposed by Horikawa et al. [9], who used sparse linear regression model, to extend visual identification to unseen classes in training phase. MVBGM [1] proposes to solve this problem from the perspective of multi-view learning, introducing probability graph model, which used shared hidden variables to explain the data generation process. On this basis, MS-MVBGM [3] was proposed to solve the function alignment problem of multiple subjects. Semi-MVBGM [2] used semi-supervised learning to utilize unpaired images to increase the scale of dataset. Some methods related to generic decoding are summarized in [14]. The current methods lack the interpretability of decoding process due to the lack of prior knowledge. Injecting prior knowledge into current model is promising.

## 2.2 Zero-Shot Learning (ZSL)

ZSL is proposed to solve the dilemma that traditional classification can only classify limited categories in training set, expecting knowledge transfer from visual to semantic and from seen to unseen. ZSL can be divided into embedding model [6] and generative model [17]. The embedding model learns a mapping function to transform visual features and semantics into the shared embedded space, while the generative model generates image features for unseen classes using semantic attributes. Our approach combines embedding and generative model. To be more specific, our approach learns the projections of visual and cognitive spaces to a shared latent space and knowledge transfer from cognitive to visual features by generation process of data.

## 2.3 Variational Auto-Encoders (VAE)

VAE has emerged as an effective method for modeling the generation process of data [12]. The prior in KL divergence was modified to capture complicated structure of latent space [19]. In order to achieve disentanglement, some methods deformed KL divergence into different forms [8,16]. AAE [13] and WAE [18] substituted KL divergence with adversarial learning or Wasserstein distance to achieve better compactness in latent space. As for more than one modality input, [20] built paralleled single-modality VAE with strong correlation regularization between corresponding layers. However, few consider disentanglement performance on complex datasets. We aim to address this issue.
<figure><img src="image_4.png" /><figcaption>Fig. 1. Generic decoding framework.</figcaption></figure>

# 3 Method

**Problem Settings.** Suppose we have $N_{\text{tr}}$ training and $N_{\text{te}}$ testing image-fMRI pairs and $N_{\text{ca}}$ candidate images. For training pairs, semantic vectors (usually word embeddings corresponding to the image label) are also available. Denote $\mathcal{V}$ as visual space constructed by visual images, $\mathcal{C}$ and $\mathcal{S}$ as cognitive space (constructed by fMRI) and semantic space (constructed by semantic vectors) respectively. Let $D_{\text{tr}} = \{(v_i, c_i, s_i)|v_i \in \mathcal{V}, c_i \in \mathcal{C}, s_i \in \mathcal{S}\}_{i=1}^{N_{\text{tr}}}$ be the training set, and $D_{\text{te}} = \{(v_i, c_i)|v_i \in \mathcal{V}, c_i \in \mathcal{C}\}_{i=1}^{N_{\text{te}}}$, $D_{\text{ca}} = \{v_i|v_i \in \mathcal{V}\}_{i=1}^{N_{\text{ca}}}$ be the test set and the candidate set respectively. The aim of generic decoding is to learn a mapping (decoder) from space $\mathcal{C}$ to $\mathcal{V}$, and then decodes $c$ in the test set into $\hat{v}$, finally classified by nearest neighbour searching in the candidate set under certain similarity metric. The training and testing phase of generic decoding can be summarized as Fig. 1.

## 3.1 Overall Model Architecture

The overall architecture is shown as Fig. 2. The framework mainly consists of four modules: visual and cognitive feature extractors (EXT), feature VAEs, PKN and LSDN. In training phase, given an image-fMRI pair $(v, c)$, firstly the pair is fed into feature extractors to obtain features $(x_v, x_c)$. Secondly, prior knowledge about the image $v$ can be obtained by pretrained models, and the corresponding
prior distribution is encoded by PKN. Finally, $(x_v, x_c)$ pass through feature VAEs jointly with the regularization of LSDN, forming an interpretable latent space. The following subsections describe the modules in detail.

<figure><img src="image_5.png" /><figcaption>Fig. 2. The overall architecture.</figcaption></figure>

## 3.2 Feature Extractor

The first step of framework is feature extraction. As mentioned before, the reason for using extracted features instead of raw data is to filter out useless information in raw data. For visual and cognitive data, two different extractors should be applied.

**Visual Feature Extractor:** A large number of studies have shown that VGG19 is more consistent with the working process of human visual pathway. Therefore, the pretrained VGG19 is used as the visual feature extractor.

**Cognitive Feature Extractor:** There is no widely recognized feature extractor in terms of fMRI feature extraction. Although simple MLP with shallow layers (or linear model) as the extractor is effective, we expect a feature extractor that combines the fitting ability of nonlinear model. In addition, we have prior knowledge about fMRI and brain structure. The fMRI data is high-dimensional and noisy. In addition, the cortices in the visual pathway are bidirectional connected
[11, 15], and there is also a direct pathway from lower cortices to higher cortices. Therefore, this work proposes a feature extractor that combines our prior knowledge of fMRI data and brain connection, as shown in Fig. 3.

<figure><img src="image_6.png" /><figcaption>Fig. 3. The fMRI feature extractor. We divide the whole visual cortices into five regions of interest (ROI), V1, V2, V3, V4 and HVC (Higher visual cortices). Firstly, {$f_i$}$_{i=1}^5$ pass through DAEs and hidden states of DAEs {$h_i$}$_{i=1}^5$ can be obtained. Secondly, {$h_i$}$_{i=1}^5$ are fed into modified Bi-LSTM. Finally, hidden states of Bi-LSTM {$\hat{h}_i$}$_{i=1}^5$ are averaged by self-attention mechanism and the averaged vector as the extracted features.</figcaption></figure>

The extractor consists of three parts: denoised auto-encoder (DAE), modified bidirectional LSTM (Bi-LSTM) and self-attention mechanism (SA). Hidden states of DAEs are dimension-reduced and noise-free. Modified Bi-LSTM adds connections cross cells, aiming to simulate the direct pathway and bidirectional connection in the brain as well. Self-attention can dynamically adjust the average weights according to different cognitive tasks. The fMRI extractor loss $\mathcal{L}_{EXT}$ is:

$$ \mathcal{L}_{EXT} = \sum_{i=1}^{5} \| \hat{f}_i - f_i \|_2^2 \qquad (1) $$

## 3.3 Subspace Partition and Prior Knowledge Network (PKN)

Images contain low-level information (LLI) and high-level information (HLI). LLI generally refers to edges, colors, textures, etc.; while HLI is mainly semantic. The two kinds of information are intrinsically different, and should not be blended and represented by the same whole latent space, which is a common setting in VAEs.
Inspired by the different level of information in images, latent space should be carefully designed, where we introduce subspace partition, dividing the original whole latent space z into two subspaces z^l and z^h, each representing only one kind of information. Accordingly, it is necessary to generate reliable prior distribution specific to the subspaces, where prior knowledge should be introduced.

* LLI: An image maintains the main color by blurring, and preserves edges by edge detector. Thus blending the two processed images can retain LLI. We denote LLI as $I_l$.

* HLI: Word2vec embedding of the image label is used here as HLI, which is a usually choice. We denote HLI as $I_h$.

After getting prior knowledge, PKN encodes the prior knowledge into low-level and high-level prior distribution, which is assumed normal distribution. Two MLP encoders $\mu(\cdot)$ and $\sigma(\cdot)$ encode the mean and variance. The encoded low-level and high-level distributions are as follows:

$$p_l(z^l|I_l) = \mathcal{N}(\mu_l(I_l), \sigma_l(I_l)), \quad p_h(z^h|I_h) = \mathcal{N}(\mu_h(I_h), \sigma_h(I_h)) \quad (2)$$

## 3.4 Feature VAE

Denote $x_m$ ($m \in \{v, c\}$) as the extracted features of modality $m$. The goal of feature VAE$_m$ is to maximize the ELBO:

$$\begin{aligned} \log p(x_m) &\ge \mathbb{E}_{\mathbf{z}_m \sim q_{\phi}(\mathbf{z}_m | x_m)} [\log p_{\theta}(x_m | \mathbf{z}_m)] - \text{KLD}(q_{\phi}(\mathbf{z}_m | x_m) \| p(\mathbf{z}_m)) \\ &= \mathbb{E}_{\mathbf{z}_m} [\log p_{\theta}(x_m | \mathbf{z}_m)] - \text{KLD}(q_{\phi}(\mathbf{z}_m | x_m) \| p(\mathbf{I})) \end{aligned} \quad (3)$$

where $\mathbf{z}_m = [z_m^l; z_m^h]$ is the concatenated latent variables, $\mathbf{I} = [I_l, I_h]$ is the prior knowledge and $q_{\phi}(\mathbf{z}_m|x_m)$ is the encoder parametrized by $\phi$, and $p_{\theta}(x_m|\mathbf{z}_m)$ is the decoder parametrized by $\theta$. The first term is the reconstruction loss of features of modality $m$. The second term is the KL-divergence loss regularizing the latent space $\mathbf{z}_m$. By adopting the assumption that $z^l$ and $z^h$ are independent, the second term can be rewritten as follows:

$$\begin{aligned} \text{KLD}(q_{\phi}(\mathbf{z}_m|x_m) \| p(\mathbf{I})) &= \text{KLD}(q(z_m^l|x_m) \| p(I_l)) \\ &\quad + \text{KLD}(q(z_m^h|x_m) \| p(I_h)) \end{aligned} \quad (4)$$

In order to alleviate posterior collapse, unlike traditional VAEs, VAE$_m$ adopts skip connection [4] from hidden layer to layers in decoder, avoiding mode collapse by maximizing mutual information between latent variable and output. Also the raw KL loss is substituted by free-bit KL loss. The final loss of feature VAE$_m$ is:

$$\begin{aligned} \mathcal{L}_{\text{VAE}_m} &= \mathcal{L}_{\text{rec}_m} + \alpha \mathcal{L}_{\text{KLD}_m} \\ &= -\mathbb{E}_{\mathbf{z}_m} [\log p_{\theta}(x_m | \mathbf{z}_m)] + \alpha \text{KLD}(q_{\phi}(\mathbf{z}_m | x_m) \| p(\mathbf{I})) \end{aligned} \quad (5)$$

Visual and cognitive feature VAEs are jointly learned, maximizing the joint log-likelihood:

$$\begin{aligned} \log p(\mathbf{x}) &= \log p(x_v) + \log p(x_c) \\ &\approx \log p(x_v) + \mathbb{E}_{\mathbf{z}_c} [\log p_{\theta}(x_c | \mathbf{z}_c)] - \alpha \text{KLD}(q_{\phi}(\mathbf{z}_c | x_c) \| q_{\phi}(\mathbf{z}_v | x_v)) \end{aligned} \quad (6)$$
where $\mathbf{x} = [x_v, x_c]$. By revising the term $\log p(x_c)$ (or $\log p(x_v)$), a strong connection is established between $\text{VAE}_v$ and $\text{VAE}_c$.

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** (a): The first step of hierarchical classification is to discriminate whether two set of variables are from different subspace, if they are from different subspace, then the second step is to discriminate whether two set of variables are from different labels. (b): LSDN receives two set of variables $z_1$ and $z_2$, and has a shared backbone and two prediction branches $\sigma_1()$ and $\sigma_2()$ for predicting the probability of different subspace and labels, respectively. ①: whether the two set are from different subspace, ②: whether the two set are from different sample.</figcaption></figure>

## 3.5 Latent Space Disentangling Network

Although PKN is proposed to generate different subspace, there is no explicit regularization to disentangle the two subspace. LSDN is an explicit constraint to force the subspaces to be discriminative. Because the two subspaces may overlap, our aim is to discriminate subspaces between samples within the same label rather than across labels. Our sample selection strategy is: given one pair $(v_i, c_i)$ with label $y_i$ and $z_i$ the corresponding latent variables, $\{(z_i^l, z_j^h) \mid y_j = y_i\}$ is seen as positive pairs $z^+$ and $\{(z_i^l, z_j^h) \mid y_j \neq y_i\}$ are negative pairs $z_1^-$. Also $\{(z_i^l, z_j^l) \mid y_j = y_i\}$ are negative pairs $z_2^-$. The reason for introducing $z_2^-$ is to avoid LSDN discriminating according to labels only rather than subspaces. The difficulty of distinguishing different labels and subspaces are not equal. Thus, we proposed hierarchical classification, which convert one-step classification of positive and negative samples to two-step classification. The schematic is shown in Fig. 4(a).

The architecture of LSDN is shown in Fig. 4(b). Assuming there are $P_i$ positive samples $z^+$ and $M_i$ negative samples $z_1^-$ and $N_i$ negative samples $z_2^-$, the
loss of LSDN is:

$$ \mathcal{L}_{\text{LSDN}} = \sum_{i=1}^{P_i} \log(\sigma_1(z^+) \sigma_2(z^+)) + \sum_{j=1}^{M_i} \log(\sigma_1(z_1^-)[1 - \sigma_2(z_1^-)]) + \sum_{j=1}^{N_i} \log(1 - \sigma_1(z_2^-)) \quad (7) $$

## 3.6 Training and Inference

**Training.** The proposed method includes EXT loss, VAE loss and LSDN loss. The total loss is as follows:

$$ \mathcal{L} = \mathcal{L}_{\text{EXT}} + \alpha \mathcal{L}_{\text{VAE}_v} + \beta \mathcal{L}_{\text{VAE}_c} + \gamma \mathcal{L}_{\text{LSDN}} \quad (8) $$

where $\alpha$, $\beta$ and $\lambda$ are hyper-parameters adjusting the weights of each loss term. All of the modules with parameters except visual feature extractor are trained together.

**Inference.** A cascaded encoder of VAE$_c$ and decoder of VAE$_v$ is used as the decoder of visual decoding. Given a piece of fMRI $c$, the corresponding visual feature vector $\hat{v}$ is computed, then the identified category $\hat{y}$ is retrieved by $\arg\max_{i} \{sim(\hat{v}, v_i)\}_{i=1}^{N_{ca}}$ where $sim(\bullet, \bullet)$ is some kind of similarity metric.

# 4 Experiments

## 4.1 Experimental Settings

**Dataset.** The method is evaluated on GD dataset [9] which collects data of five subjects, each containing 1200 image-fMRI pairs with 150 classes for training and 50 pairs with 50 classes for testing. As for semantic vectors, we use 300-dimensional word2vec embeddings corresponding to image labels. We select several images for each of 10000 categories from ImageNet as the candidate set. Images features within the same category in the candidate set are used as category features.

**Evaluation Metrics.** We choose mean rank as the metric which is a common metric in other works. The mean rank is computed as follows: First sort similarity scores in descending order for each test image, and then fetch the rank of correct category, finally average across all images. Also, we use T-SNE to visualize the clustering of latent space.
**Implementation Details.** Visual feature extractor is a pretrained VGG19 network, and auto encoders in cognitive feature extractor is one-layer MLP with hidden dimension 512. The PKN consists of two MLPs with one fully-connected layer. The backbone of LSDN is a two-layer MLP and the two branches is one fully-connected layer. Both of the encoders of feature VAEs are two fully-connected layer with hidden dimension [300, 200], and the structure of decoders are symmetric with skip connection. Dimension of $z^l$ and $z^h$ are set to be equal. The proposed model is trained using Adam with default settings in Pytorch. $\alpha$, $\beta$ and $\gamma$ are set to $1e^{-4}$, $1e^{-3}$ and $1e^{-3}$. The learning rate $1e^{-3}$ is controlled by cosine annealing with $T_0 = 5$ and $T_{mult} = 2$. $sim(\bullet, \bullet)$ is cosine distance. The model is trained 175 epochs.

<table><caption>**Table 1.** Comparison of different methods.</caption><thead><tr><th></th><th>Ref. [9]</th><th>BCCA-V</th><th>BCCA-C</th><th>MVBGM</th><th>MS-MVBGM</th><th>GD-VAE</th></tr></thead><tbody><tr><td>Subject 1</td><td>867.38</td><td>988.76</td><td>1883.9</td><td>731.88</td><td>672.82</td><td><strong>650.32</strong></td></tr><tr><td>Subject 2</td><td>646.16</td><td>586.46</td><td>1496.2</td><td>436.84</td><td>420.92</td><td><strong>415.44</strong></td></tr><tr><td>Subject 3</td><td>406.78</td><td>421.84</td><td>1484.2</td><td>373.90</td><td><strong>351.56</strong></td><td>370.11</td></tr><tr><td>Subject 4</td><td>464.94</td><td>515.92</td><td>1545.1</td><td>451.26</td><td>392.02</td><td><strong>390.94</strong></td></tr><tr><td>Subject 5</td><td>625.28</td><td>585.22</td><td>1692.0</td><td>448.34</td><td><strong>410.36</strong></td><td>418.21</td></tr></tbody></table>

<table><caption>**Table 2.** Effectiveness of prior distribution ($p_N$ means standard normal distribution).</caption><thead><tr><th>p<sub>l</sub></th><th>p<sub>h</sub></th><th>p<sub>N</sub></th><th>Mean rank</th></tr></thead><tbody><tr><td>✓</td><td>✓</td><td></td><td>370.11</td></tr><tr><td></td><td>✓</td><td>✓</td><td>1284.28</td></tr><tr><td>✓</td><td></td><td>✓</td><td>1530.46</td></tr><tr><td></td><td></td><td>✓</td><td>2521.42</td></tr></tbody></table>

## 4.2 Results

We compare our proposed GD-VAE with the methods in Refs. [1,3,9] and BCCA-V and BCCA-C. In those methods, sparse linear regression (SLR) and probability graph model (PGM) are applied to map fMRI activity into visual features. BCCA-V and BCCA-C translate fMRI into visual and category features respectively. As shown in Table 1, the mean rank of our method improves in subject 1, 2 and 4, which shows the superiority of our method.
## 4.3 Ablation Study

**The Effect of Prior Distribution.** From Table 2, it can be observed that standard normal distribution will cause severe mode collapse, which is the joint effect of strong expression ability of VAE and scarce data pairs. Only applying both low-level and high-level prior distribution can achieve better performance, which implies the disentanglement of latent subspaces. T-SNE of latent subspace $z^h$ (Fig. 5) has also shown that high-level prior distribution helps form compact latent space (we don't observe the same obvious compactness with low-level prior distribution).

**The Effect of LSDN.** By setting $\lambda$ to 0 during training, we can verify the impact of LSDN. It can be observed that mean rank of four of five subjects (except subject 2) drops and the cross entropy loss (Eq. 7) increase about 0.05, indicating the effectiveness of LSDN for subspace disentanglement.

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Visualization of latent space $z^h$ (Image categories randomly from ILSVRC-2012 dataset).</figcaption></figure>

## 5 Conclusion

In this paper, we propose a generic decoding framework named GD-VAE, aiming to achieve mapping from fMRI activity to visual features based on VAE. fMRI feature extractor is designed to purify fMRI activity from noise and simulate brain connections. We introduce prior knowledge and corresponding distribution to substitute the original standard normal distribution to get greater interpretability of latent space, with the regularization of LSDN. Experiments and ablation studies validate the effectiveness of the proposed method. We will continue to analyze the effect of nonlinear model in fMRI feature extractor and improve the interpretability of decoding process in the future.
**Acknowledgements.** This work is supported by the National Science Foundation of China (No. 62088102), China National Postdoctoral Program for Innovative Talents from China Postdoctoral Science Foundation (No. BX2021239).

## References

1. Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Estimating viewed image categories from fMRI activity via multi-view Bayesian generative model. In: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), pp. 127–128. IEEE (2019)

2. Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Brain decoding of viewed image categories via semi-supervised multi-view Bayesian generative model. IEEE Trans. Sig. Process. **68**, 5769–5781 (2020)

3. Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Multi-view Bayesian generative model for multi-subject fMRI data on brain decoding of viewed image categories. In: ICASSP 2020–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1215–1219. IEEE (2020)

4. Dieng, A.B., Kim, Y., Rush, A.M., Blei, D.M.: Avoiding latent variable collapse with generative skip models. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397–2405. PMLR (2019)

5. Du, C., Du, C., Huang, L., He, H.: Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Trans. Neural Netw. Learn. Syst. **30**(8), 2310–2323 (2018)

6. Frome, A., Corrado, G., Shlens, J., et al.: A deep visual-semantic embedding model. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 2121–2129 (2013)

7. Higashi, T., Maeda, K., Ogawa, T., Haseyama, M.: Estimation of visual features of viewed image from individual and shared brain information based on fMRI data using probabilistic generative model. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1335–1339. IEEE (2021)

8. Higgins, I., et al.: β-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations (2017)

9. Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. **8**(1), 15037 (2017)

10. Huang, S., Shao, W., Wang, M.L., Zhang, D.Q.: fMRI-based decoding of visual information from human brain activity: a brief review. Int. J. Autom. Comput. **18**(2), 170–184 (2021). https://doi.org/10.1007/s11633-020-1263-y

11. Huang, W., et al.: Long short-term memory-based neural decoding of object categories evoked by natural images. Hum. Brain Mapp. **41**(15), 4442–4453 (2020)

12. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

13. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)

14. Papadimitriou, A., Passalis, N., Tefas, A.: Visual representation decoding from human brain activity using machine learning: a baseline study. Pattern Recogn. Lett. **128**, 38–44 (2019)

15. Qiao, K., et al.: Category decoding of visual stimuli from human brain activity using a bidirectional recurrent neural network to simulate bidirectional information flows in human visual cortices. Front. Neurosci. **13**, 692 (2019)
16. Rodriguez, E.G.: On disentanglement and mutual information in semi-supervised variational auto-encoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1257-1262 (2021)

17. Schonfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., Akata, Z.: Generalized zero- and few-shot learning via aligned variational autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8247-8255 (2019)

18. Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017)

19. Tomczak, J., Welling, M.: VAE with a VampPrior. In: International Conference on Artificial Intelligence and Statistics, pp. 1214-1223. PMLR (2018)

20. Wang, X., Peng, D., Hu, P., Sang, Y.: Adversarial correlated autoencoder for unsupervised multi-view representation learning. Knowl.-Based Syst. 168, 109-120 (2019)