# Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network

Siyu Yu, Nanning Zheng, Hao Wu, Ming Du, Badong Chen

► To cite this version:

Siyu Yu, Nanning Zheng, Hao Wu, Ming Du, Badong Chen. Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network. 15th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), May 2019, Hersonissos, Greece. pp.223-235, 10.1007/978-3-030-19823-7_18. hal-02331335

HAL Id: hal-02331335

https://inria.hal.science/hal-02331335

Submitted on 24 Oct 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network*

Siyu Yu¹,², Nanning Zheng¹,²,³, Hao Wu¹,², Ming Du¹,², and Badong Chen¹,²

¹ Institute of Artificial Intelligence and Robotics,
Xi’an Jiaotong University, Xi’an, Shaanxi, P.R.China

² National Engineering Laboratory for Visual Information Processing and Applications, Xi’an
Jiaotong University, Xi’an, Shaanxi, P.R.China

{yusiyu, xuan.zhi, x1978626209}@stu.xjtu.edu.cn;
chenbd@mail.xjtu.edu.cn

³ Correspondence: nnzheng@mail.xjtu.edu.cn

**Abstract.** Brain-inspired computing is a research hotspot in artificial intelligence (AI). One of the key problems in this field is how to find the bridge between brain connectivity and data correlation in a connection-to-cognition model. Functional magnetic resonance imaging (fMRI) signals provide rich information about brain activities. Existing modeling approaches with fMRI focus on the strength information, but neglect structural information. In a previous work, we proposed a monolayer correlation network (CorrNet) to model the structural connectivity. In this paper, we extend the monolayer CorrNet to a hierarchical correlation network (HcorrNet) by analysing visual stimuli of natural images and fMRI signals in the entire visual cortex, that is, V1, V2 V3, V4, fusiform face area (FFA), the lateral occipital complex (LOC) and parahippocampal place area (PPA). Through the HcorrNet, the efficient connectivity of the brain can be inferred layer by layer. Then, the stimulus-sensitive activity mode of voxels can be extracted, and the forward encoding process of visual perception can be modeled. Both of them can guide the decoding process of fMRI signals, including classification and image reconstruction. In the experiments, we improved a dynamic evolving spike neuron network (SNN) as the classifier, and used Generative Adversarial Networks(GANs) to reconstruct image.

**Keywords:** Brain-inspired computing · visual perception · functional magnetic resonance imaging (fMRI) · Hierarchical correlation network (HcorrNet) · connection.

## 1 Introduction

A better understanding of the brain can inspire researchers for efficient methods in brain-inspired computing, which can help to develop more robust artificial intelligence (AI) technologies. [23]. In recent decades, brain-inspired computing with functional magnetic resonance imaging (fMRI) signals have driven plenty of studies [5, 12].

* This work was supported by the National Natural Science Foundation of China (NO. 61773312, L1522023 and No. 91648208) and the 973 Program (No. 2015CB351703).
One key problem is how to analyze the correlations in collected data for modeling connection-to-cognition process. Much of the literature are devoted to modeling the correlations between neural activities in visual perception and revealing neural connections in brain [13, 8, 6]. In early studies, voxel receptive-field models have been used to analyze the correlations by locating a small visual stimulus that elicits activities spatially spread over voxels. Some researchers combined linear regression and Bayesian inference algorithms with correlation models [20, 15]. In recent years, deep neural network (DNN), VAEs [9] and DCCAE [18] have been used to study this problem. As is well known, neural connections can be divided into three types of connections between brain nodes: functional connectivity, structural connectivity, and effective connectivity [14, 21]. However, most existing correlation models [3, 10] have left out the structural information of voxel, which may provide information about the intracortical or intrinsic connections, i.e., the structural connectivity. To reveal the working mechanism of the human brain (connection-to-cognition process), in a previous work, we proposed a monolayer correlation network (CorrNet) by considering the three correlations to simulate the rich effective connectivity of the early visual cortex and establish mapping from visual perceptual tasks to voxels [21].

Recently, some DNNs [17] and hierarchical structure [2] are utilized to build brain-inspired models. As for image reconstruction, some researchers use fMRI signals as the input of Gans [16]. Inspired by these works, in this paper, we propose a hierarchical correlation network (HcorrNet) by extending our monolayer CorrNet [21] to deep framework case. The study rests on analyzing hierarchical correlations to explore effective connectivity in the entire visual cortex (VC), including V1, V2, V3, V4, fusiform face area (FFA), lateral occipital complex (LOC), parahippocampal place area (PPA). In the HcorrNet, the correlations of collected data are obtained, including perception-voxel corresponding correlation and voxel-voxel correlation. The voxel-voxel correlation can be modeled as voxel correlation between voxels in the same visual cortex and the hierarchical voxel correlation between voxels of two layers. Then, the correlations are utilized to model the stimulus-sensitivity activation patterns of brain and the layer-to-layer forward propagation process, that is, the forward encoding information flow, both of which help to explore the effective connectivity in visual perception. Then, based on the inferred effective connectivity, we location the most activate voxels for the decoding process, including classification and image reconstruction. In our experiments, a improved dynamic evolving spike neuron network (SNN) [4, 7] and Wasserstein GAN (WGAN) [1] are used to classify and reconstruct image, respectively. The experimental dataset is the public fMRI dataset from ATR Computational Neuroscience Laboratories of Kyoto University [6].

The rest of the paper is organized as follows. Section 2 describes the monolayer CorrNet. Section 3 proposes the HcorrNet framework. Section 4 describes the decoding process, including classification and reconstruction. In Section 5, experiments are conducted to demonstrate the performance of our method. Finally, conclusion is given in Section 6.
## 2 Monolayer Correlation Network

Our previous work rested on how the visual cortex organizes addresses different visual stimuli based on function-structure relationships in the brain. Three kinds of correlations were modeled, including the correlation among brain activities, the correlation between brain activities and visual stimuli, and the correlation among visual stimuli. We transformed the correlation degree into correlation probability, and defined the following novel concepts:

*Voxel correlation probability* measures the voxel correlations, whose element is the probability that any two voxels are of functional and structural correlation.

*Strength correlation probability* measures the strength correlation, which means the similarity or opposite reflection to the same stimuli. Its element presents the probability of strength correlation.

*Topological correlation probability* measures the topological correlation, which means any two voxels are in the same topological space. Its element presents the probability that any two voxels are of topological correlation.

*Corresponding correlation* is viewed as a significance measure of the corresponding correlation between one voxel and one pixel, which is not of simple linear relationship, and is different from the traditional correlation measure, such as Pearsons Correlation.

*Corresponding correlation probability* is the probability that one voxel of is in the inverse receptive field of one pixel. Its values determines corresponding correlation.

In the monolayer CorrNet, above correlations refers to the functional connectivity, the structural connectivity and the effective connectivity. So, we introduced the topological correlation to measure the structural connectivity. Notably, Delaunay triangulations are utilized to reveal the topological correlation, because Delaunay triangulations can reveal present superficial connectivity between voxels, which is consistent with the inner structure of the cerebral cortex [22]. The functional connectivity was measured by the Strength correlation, which is computed in a full covariance matrix. Then, Bayes rules were used to combine topological correlation with strength correlation between voxels to obtain voxel correlation. Later, we computed the corresponding correlation between voxels and pixels in a probabilistic correlation graph. In the graph, the corresponding correlation probability was initialized as strength correlation probability between voxel and pixel. Next, considering the voxel correlation, the graph was updated, iteratively, by computing the joint probability of voxel correlation and strength correlation between voxel and pixel. Then, a threshold parameter was used to determine the degree of the corresponding correlation. When the corresponding correlation probability of certain voxel-pixel pair is equal to or greater than the threshold parameter, the voxel and pixel are of significant corresponding correlation, that is, the voxel is in the inverse receptive field of the pixel. Finally, the inverse receptive field of the pixel in brain was located, which revealed the effective connectivity in visual cognition tasks.

## 3 Hierarchical Correlation Network

As Fig. 1 shows, to explore the entire visual cortex, this paper focuses on how to model the correlations between stimuli and the voxels in the entire visual cortex, including
<figure><img src="image_2.png" /><figcaption>Fig. 1. The framework of our methods.</figcaption></figure>

V1, V2, V3, V4, FFA, LOC, PPA. Considering that the cognition information of visual stimuli is forward encoded from lower visual cortex to higher visual cortex, we propose a deep framework with hierarchical correlation network (HcorrNet) to address this problem. Thus, the original problem, exploring the effective connectivity in visual perception, is translated into three sub-problems:

How to obtain the correlations inferred by the collected data, including voxel correlation between voxels in the same visual cortex and the voxel correlation between voxels in two layers (such as V1 and V2) and perception-voxel corresponding correlation (pixel-voxel corresponding correlation). We call the former *inner-cortex voxel correlation*, and compute the probability that two voxels are of inner-cortex voxel correlation, denoted as *Prob*<sub>vinn</sub>. Similarly, the latter is called *inter-cortex voxel correlation*, *Prob*<sub>vint</sub> is defined as the probability that two voxels are of inter-cortex voxel correlation;

How to use the correlations to explore the stimulus-sensitivity activation patterns;

How to represent the correlations in the forward encoding process.

## 3.1 Voxel Correlation in the HcorrNet

The inner-cortex voxel correlation in each layer, denoted as $CV_{inn}^{lc}$, can be obtained by our previous work of CorrNet [21], easily, which exactly analyses the voxel correlation in single layer.
As for the inter-cortex voxel correlation, the computation procedure is differen-
t from that of the inner-cortex voxel correlation. In this paper, lc, derived from V1, V2,
V3, V4, FFA, LOC, PPA, is denoted a certain visual cortex. vn$^{lc}$ denotes the number of
voxels in the lc$^{th}$ layer. $V_i^{lc} \in \mathbb{R}^N$, $i = 1, ..., vn^{lc}$ stands for the strength value vector
of the i$^{th}$ voxel in the lc$^{th}$ visual cortex, N is the number of trials. Vnum is the number
of all voxels in the entire visual cortex. $v_{i,m}^{lc}, i = 1, ..., vn^{lc}, m = 1, ..., N$ represents
the m$^{th}$ element of $V_i^{lc}$. Prob$_{vint}$(V$_i^{lc}$, V$_j^{lc+1}$) is denoted as the inter-cortex voxel cor-
relation between the i$^{th}$ voxel in the lc$^{th}$ layer and the j$^{th}$ voxel in the lc + 1$^{th}$ layer.
The strength correlation probability of any two voxels from adjacent visual cortex is
denoted as Prob$_{vstr}$, given by:

$$
\mathrm{Prob}_{\mathrm{vstr}}(\mathbf{V}_{i}^{\mathrm{lc}}, \mathbf{V}_{j}^{\mathrm{lc}+1}) = \frac{\left| \sum_{m=1}^{N} (v_{i,m}^{\mathrm{lc}} v_{j,m}^{\mathrm{lc}+1}) - \frac{\sum_{m=1}^{N} v_{i,m}^{\mathrm{lc}} \sum_{m=1}^{N} v_{j,m}^{\mathrm{lc}+1}}{N} \right|}{\sqrt{\sum_{m=1}^{N} v_{i,m}^{\mathrm{lc}}^2 - \frac{\sum_{m=1}^{N} 2 v_{i,m}^{\mathrm{lc}}}{N}}} \sqrt{\sum_{m=1}^{N} v_{j,m}^{\mathrm{lc}+1}^2 - \frac{\sum_{m=1}^{N} 2 v_{j,m}^{\mathrm{lc}+1}}{N}} \quad (1)
$$

Then, Prob$_{vint}$ of two voxels $\mathbf{V}_i^{lc}, i = 1, ..., vn^{lc}$ and $\mathbf{V}_j^{lc+1}, j = 1, ..., vn^{lc+1}$ is
given by Algorithm .1.

A voxel correlation matrix $\mathbf{Cv}_{inter}^{lc} \in \mathbb{R}^{vn^{lc} \times vn^{lc+1}}$ is introduced to simplify $\text{Prob}_{vint}$.
$\mathbf{Cv}_{inter}^{lc}(\mathbf{V}_i^{lc}, \mathbf{V}_j^{lc+1})$ defines the correlation between voxel $\mathbf{V}_i^{lc}$ and voxel $\mathbf{V}_j^{lc+1}$, whose
computation process is detailed in [21]. When $\text{Prob}_{vint}(\mathbf{V}_i^{lc}, \mathbf{V}_j^{lc+1}) >= \text{thre}_{vint}$,
$\mathbf{Cv}_{inter}^{lc}(\mathbf{V}_i^{lc}, \mathbf{V}_j^{lc+1}) = 1$, otherwise, $\mathbf{Cv}_{inter}^{lc}(\mathbf{V}_i^{lc}, \mathbf{V}_j^{lc+1}) = 0$, where $\text{thre}_{vint}$ is a
free parameter, used to determine the degree of correlation and set to 0.5 in the present
study.

3.2 Corresponding Correlation in the HcorrNet

Following our previous work [21], we compute the corresponding correlation between
visual perception and voxels layer by layer. More precisely, we compute the corre-
sponding correlation by using the monolayer algorithm, independently and repeatedly.
For each layer, a binary matrix Corr$^{lc} \in \mathbb{R}^{Pnum \times vn^{lc}}$, lc \in \{1, ..., 6\} is denoted as
the corresponding correlation matrix of the lc$^{th}$, where Pnum is the number of pixel-
s. Thus, Corr$^{lc}(p_k, V_i^{lc}) = 1$ means the i$^{th}$ voxel of the lc$^{th}$ visual cortex is in the
inverse receptive field of the k$^{th}$ pixel, where p_k is the k$^{th}$ pixel of images.

3.3 Stimulus-sensitivity Activation Patterns

As is known, certain fixed activation patterns of Stimulus-sensitivity character can be
inferred from human brain when the stimuli are of the same category, and can be
viewed as stimulus-sensitivity cognition unit [19]. In this paper, to explore how the
different stimuli affect the brain connectivity, we compute the joint probability distri-
bution of all voxel correlation probability matrixes with category labels, denoted as
**Algorithm 1** Generating Prob<sub>vint</sub>.

**Require:**

Prob<sub>vinn</sub> of all voxels in the lc<sup>th</sup>, Prob<sub>vstr</sub> of the lc and lc+1 layers, a threshold, V<sub>i</sub><sup>lc</sup>, i = 1, ..., vn<sup>lc</sup> and V<sub>j</sub><sup>lc+1</sup>, j = 1, ..., vn<sup>lc+1</sup>, strength correlation threshold ε<sub>vstr</sub>, inter-cortex voxel correlation threshold ε<sub>vinn</sub>.

**Ensure:**

Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>)

1: Initial Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>) = Prob<sub>vstr</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>)

2: **for** i = 1 to vn<sup>lc</sup> **do**

3:     temp = 0; L<sub>i</sub> = φ;

4:         **for** j = 1 to vn<sup>lc+1</sup> **do**

5:             temp = max(temp, Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>));

6:         **if** Prob<sub>vstr</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>) ≥ ε<sub>vstr</sub> **then**

7:             k<sub>t</sub> = MAXNUM;

8:         **for** j* = 1 to vn<sup>lc+1</sup> **do**

9:             **if** Prob<sub>vinn</sub>(V<sub>j</sub><sup>lc+1</sup>, V<sub>j*</sub><sup>lc+1</sup>) ≥ ε<sub>vinn</sub> **then**

10:         Update L<sub>i</sub> = L<sub>i</sub> ∪ {j*};
11:         k<sub>t</sub> = min(k<sub>t</sub>, d(V<sub>j</sub><sup>lc+1</sup>, V<sub>j*</sub><sup>lc+1</sup>));
12:         for all m ∈ L<sub>i</sub>, calculate:
13:             pinn = k<sub>t</sub>/d(V<sub>j</sub><sup>lc+1</sup>, V<sub>m</sub><sup>lc+1</sup>));
14:             Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>m</sub><sup>lc+1</sup>) = pinn * Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>));
15:             Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>m</sub><sup>lc+1</sup>) = max(temp, Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>m</sub><sup>lc+1</sup>));

11:     **end if**

12: **end for**

13:     **end if**

14: **end for**

15: **if** L<sub>(i)</sub> == ∅ **then**

16:         j = arg max<sub>j* ∈ {1,...,vn<sup>lc+1</sup>}</sub> Prob<sub>vint</sub>(V<sub>i</sub><sup>lc</sup>, V<sub>j*</sub><sup>lc+1</sup>);

Return to step 5;

17: **end if**

18: **end for**

Cv<sub>ss</sub><sup>lc</sup> ∈ ℝ<sup>vn<sup>lc</sup>×vn<sup>lc+1</sup></sup>. It is derived from the stimuli within the same category with Hadamard products. Cv<sub>ss</sub><sup>lc</sup> is a binary matrix, given by Algorithm 2.

Cv<sub>ss</sub><sup>lc</sup>(V<sub>i</sub><sup>lc</sup>, V<sub>j</sub><sup>lc+1</sup>) ∈ {0, 1} can be viewed as the common effective connectivity with category guidance, measuring whether the i<sup>th</sup> voxel of the lc<sup>th</sup> visual cortex and the j<sup>th</sup> voxel of the (lc + 1)<sup>th</sup> visual cortex are of stimulus-sensitivity voxel correlation or not.

Similarly, the common effective connectivity relying on the pixel-voxel corresponding correlation, denoted as Cpv<sub>ss</sub><sup>lc</sup> can be obtained by computing the joint probability distribution of all corresponding correlation matrices Corr<sup>lc</sup> derived from the stimuli within the same category with Hadamard products. It is a binary matrix Cpv<sub>ss</sub><sup>lc</sup>(p<sub>k</sub>, V<sub>i</sub><sup>lc</sup>) = 1 measures whether the i<sup>th</sup> voxel of the lc<sup>th</sup> visual cortex and the k<sup>th</sup> pixel are of stimulus-sensitivity corresponding correlation or not. For each pixel p<sub>k</sub>, a set Cr<sub>k</sub><sup>lc</sup> = {i | Cpv<sub>ss</sub><sup>lc</sup>(p<sub>k</sub>, V<sub>i</sub><sup>lc</sup>) = 1} is defined to present the set of voxels in the lc<sup>th</sup> visual cor-
**Algorithm 2** obtain $Cv_{ss}^{lc}$.

**Require:**

$Cv_{inter}^{lc}$, $Label \in \mathbb{R}^N$ is the category label, $classnum = \text{MAX}(Label)$, $v_{i,m}^{lc}, i = 1, ..., vn^{lc}, m = 1, ..., N, v_{j,m}^{lc+1}, j = 1, ..., vn^{lc+1}, m = 1, ..., N$.

**Ensure:**

$Cv_{ss}^{lc} \in \mathbb{R}^{vn^{lc} \times vn^{lc+1}}$

1: Initial $Cv_{ss}^{lc}(V_i^{lc}, V_j^{lc+1}) = 1$,

$\text{temp\_Cv}_{ss}^{lc} = Cv_{ss}^{lc}$.

2: **for** $k = 1$ to classnum **do**

3: &nbsp;&nbsp;&nbsp;&nbsp;list = {tag | Label tag} = k, tag = 1, ..., N;

4: &nbsp;&nbsp;&nbsp;&nbsp;**for** $i = 1$ to $vn^{lc}$ **do**

5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\bar{V}_i^{lc} = V_i^{lc}(list)$

6: &nbsp;&nbsp;&nbsp;&nbsp;**for** $j = 1$ to $vn^{lc+1}$ **do**

7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\bar{V}_j^{lc+1} = V_j^{lc+1}(list);$

$\text{temp\_Cv}_{ss}^{lc}(i, j) = Cv_{inter}^{lc}(V_i^{lc}, V_j^{lc+1});$

8: &nbsp;&nbsp;&nbsp;&nbsp;**end for**

9: &nbsp;&nbsp;&nbsp;&nbsp;**end for**

$Cv_{ss}^{lc} \times \text{temp\_Cv}_{ss}^{lc};$

10: **end for**

tex which have stimulus-sensitivity corresponding correlation with $p_k$. The voxels in $Cr_k^{lc}$ can be viewed as active nodes in terms of stimulus-sensitivity activation patterns.

## 3.4 Forward Encoding Process Representation in the HcorrNet

It is believed that, the higher visual cortex receives the information from the lower visual cortex. In this paper, we try to model the information flow process.

Firstly, we build a directed probability graph to figure out which voxels in the higher visual cortex will be passed by the information from certain voxel in the lower visual cortex. Notably, our method with bayes reference is of data-diven analysis, which is totally different from the anatomical methods. Those methods mainly tend to use biomarkers to trace the information flow. In the garph, $Prob_{fw}$ is denoted as information-reachable probability, presenting the binary probability that any two nodes (voxels) are information-reachable. $Prob_{fw}(V_i^{lc1}, V_j^{lc2})$, $lc1 < lc2$ presents whether the information can flow from the voxel $V_i^{lc1}$ in the lower layer to $V_j^{lc2}$ in the higher layer or not. $Prob_{fw}$ is given by:

$$ Prob_{fw}(V_i^{lc1}, V_j^{lc2}) = \prod_{k=lc1}^{lc2-1} C v_{ss}^{lc}(V_i^k, V_j^{k+1}) \quad (2) $$

Next, we complete the whole forward encoding process by viewing the visual image as the 0th layer and computing $Prob_{fw}(p_k, V_m^{lc})$, $k = 1, ..., Pnum, m = 1, ..., Vnum$, $lc$ is the layer tag of the $m$th voxel within all voxels. When $lc = 0, Prob_{fw}(p_k, V_m^{lc}) = C p v_{ss}^{lc}(p_k, V_m^{lc+1})$, otherwise:

$$ Prob_{fw}(p_k, V_m^{lc}) = \max_{m^*=1}^{m^* \le vn^{lc}} (Prob_{fw}(p_k, V_{m^*}^{lc-1}) \cdot Prob_{fw}(V_{m^*}^{lc-1}, V_m^{lc})) \quad (3) $$
When Prob<sub>fw</sub>(p<sub>k</sub>, V<sub>m</sub><sup>lc</sup>) = 1, the k<sup>th</sup> pixel and the m<sup>th</sup> voxel with layer tag lc are information-reachable pair. For each pixel p<sub>k</sub>, a set Fr<sub>k</sub><sup>lc</sup> = {m | Prob<sub>fw</sub>(p<sub>k</sub>, V<sub>m</sub><sup>lc</sup>) = 1} is defined to represent the set of voxels in the lc<sup>th</sup> visual cortex which are information-reachable to p<sub>k</sub>. The voxels in Fr<sub>k</sub><sup>lc</sup> can be viewed as effective nodes in the forward encoding process. The more voxels Fr<sub>k</sub><sup>lc</sup> has, the more voxels the effective connectivity in terms of the corresponding pixel p<sub>k</sub> is related to. In addition, a chain from certain pixel to certain voxel in the HVC can be revealed, which can be viewed as the whole the information flow from visual perception to brain response. Notably, to simplify the computation, the forward encoding process is computed in chain V1 → V2 → V34 → HVC.

Based on the pixel-wise inverse receptive field, we compute the pixel-voxel correlation pairs by combining the forward encoding process representation and stimulus-sensitivity activation patterns, denoted as **bin**<sub>k</sub> for k = 1, ..., *Pnum*.

**bin**<sub>k</sub> presents the set of voxels that are both within the information flow of pixel p<sub>k</sub>, and are of similar stimulus-sensitivity activation pattern. **bin**<sub>k</sub> is:

$$ \mathbf{bin}_k = \{i | \mathbf{C}\mathbf{v}_{ss}^{hvc}(\mathbf{p}_k, \mathbf{V}_i) = 1 \& \mathrm{Prob}_{fw}(\mathbf{p}_k, \mathbf{V}_i) = 1\} \quad (4) $$

Then, to locate the voxels with the most high active stimulus-response pattern, we use sliding windows in the image coordinate to union the intersectional voxels in of **bin**<sub>k</sub> within each window, denoted **Voxel**<sub>act</sub>. **Voxel**<sub>act</sub> contains all voxels with the most high active stimulus-response pattern, and can be used in decoding process, including classification and reconstruction.

## 4 Decoding Process

In the decoding process, only voxels in **Voxel**<sub>act</sub> are used. As is known, the learned connections of SNNs [7] represent dynamic spatiotemporal relationships, making SNNs more biomimetic than artificial neural networks (ANNs) in both information transmission and structure.

<figure><img src="image_3.png" /><figcaption>Fig. 2. The structure of our SNN.</figcaption></figure>
In our previous work, SNNs have been used to classify MNIST Dataset and fMRI signals of simple images, yielding satisfying results. Therefore, in this paper, a similar SNN structure is also used as the classifier. Fig. 2 shows the network structure of the SNN. In the training process, the input of the SNN is a $N \times M$ matrix with $N$ training samples($N$ training trials) and $M$ voxels response attributes ($M$ is the number of voxels in $\textbf{Voxel}_{act}$), the output is the class label of corresponding stimulus (50 in this paper). The detail is in [21, 11].

To obtain reconstruction images, an improved WGAN [1] with gradient penalty is trained to generate $32 \times 32 \times 3$ image. The input of generator is fixed voxel response value scalars of all $M$ voxels in $\textbf{Voxel}_{act}$ corresponding to the related stimulus, instead of random noise. The input images ($32 \times 32 \times 3$) is transformed from the ordinary images (RGB, $800 \times 800 \times 3$).

# 5 Experiments

The experimental data comes from the public fMRI dataset from ATR Computational Neuroscience Laboratories of Kyoto University, which contains training set and test set. The training set includes 1200 nature images of 150 categories (8 images from each category) and the corresponding fMRI signals. In the process with HCorrNet, we only used the training set to obtain the voxel tag $\textbf{Voxel}_{act}$. Firstly, the nature images (RGB, $800 \times 800 \times 3$) were transformed into grayscale images and resized into $224 \times 224$ with normalization process. Moreover, V3 is merged into V4 as V34 layer. In the decoding process, we use the test set which includes 50 nature images of new 50 categories (never intersect with the category in the training set). Each image in the test set is showed 35 times to subjects, repeatedly. The first 80% of the test set is used as the training set for classification and reconstruction, the rest is used as the test set in the decoding process.

As detailed before, $\textbf{Cr}_k^{lc} = \{i | \text{Corr}_{ss}(\mathbf{p}_k, \mathbf{V}_i^{lc}) = 1\}$ is defined to represent the tag set of voxels in the $lc^{th}$ layer which have stimulus-sensitivity corresponding correlation with $\mathbf{p}_k$. $\textbf{Fr}_k^{lc} = \{m | \text{Prob}_{fw}(\mathbf{p}_k, \mathbf{V}_m^{lc}) = 1\}$ is defined to indicate the tag set of voxels in the $lc^{th}$ layer which are information-reachable to $\mathbf{p}_k$. When $lc$ presents the HVC region, the forward encoding is accomplished. So, the reservation rate of voxels in the $\textbf{Voxel}_{act}$ is showed in Tabel 1(Note: there is overlap between the layers.):

<table><caption>**Table 1.** The reservation rate of voxels in the $\textbf{Voxel}_{act}$.</caption><thead><tr><th>Visual Cortex</th><th>V1</th><th>V2</th><th>V3</th><th>V4</th><th>FFA</th><th>PPA</th><th>LOC</th><th>LVC</th><th>HVC</th><th>Entire</th></tr></thead><tbody><tr><td>All Voxel Number</td><td>1004</td><td>1018</td><td>759</td><td>740</td><td>540</td><td>568</td><td>356</td><td>2281</td><td>2049</td><td>4466</td></tr><tr><td>Reserved Voxel Number</td><td>69</td><td>132</td><td>18</td><td>2</td><td>0</td><td>3</td><td>14</td><td>136</td><td>27</td><td>163</td></tr><tr><td>Reservation(%)</td><td>6.87</td><td>12.97</td><td>2.37</td><td>0.27</td><td>0.00</td><td>0.51</td><td>3.93</td><td>5.96</td><td>1.32</td><td>3.65</td></tr></tbody></table>

To visualize the information flow, five heat maps are drawn to reveal the quantitative distribution of voxels in the $\textbf{Fr}_k^{lc}$, $k = 1, 2, ..., 224 \times 224$ by mapping the number of
<figure><img src="image_4.png" /><figcaption>Fig. 3. Quantitative distribution of voxels.</figcaption></figure>

voxels in $Fr_k^{lc}$ into the coordinates of $p_k$ in the image coordinate frame in chain V1 → V2 → V34 → HVC, as shown in Fig.3. The first four images is the quantitative distribution of V1, V2, V34 and HVC from $Fr_k^{lc}$, and the last one is the quantitative distribution of the union set of voxels of $Fr_k^{lc}$. The more red the pixel patch, the more voxels $p_k$ is information-reachable to. The more red the $k^{th}$ pixel patch, the more voxels $Fr_k^{lc}$ contains. In the last heat map, the middle region contains the most voxels, which conforms to the attention mechanism in human perception. Additionally, more higher the layer is, more conspicuous the tendency that most voxels concentrate on the middle region is. Therefore, our HCorrNet framework can reveal the attention mechanism in the forward encoding process well.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Reconstructions result (Subject 1) of some images in the test set. The perceptual images are shown in the first row. The second is the processed images which are the input images of the WGAN, the rest four rows are the RGB reconstruction images of different test trials.</figcaption></figure>
The classification result of our SNN is 68.43%. We failed to find other similar research of fMRI classification by using nature images as stimuli. Thus, there is no comparison. However, it is hard to process fMRI signals because of its severe noise and high dimension. Thus, 68.43% for 50 categories is accredited and maybe yield the state of art performance. In addition, the reconstruction result is shown in Fig.4. This reconstruction result can be nearly recognizable and significantly different, which demonstrates the HCorrNet reserves important voxels, and reveals the efficient connectivity and data correlations, effectively.

Through HCorrNet, less than 4% voxels are selected (see Tabel 1), which confirms the excellent denoising performance of our methods. The decoding result demonstrates the importance of HCorrNet in the entire visual cortex as it not only refines the high-dimension fMRI signals for decoding task, but also enhances the data correlations. Additionally, the information flow in the forward encoding process has been revealed both in the algorithm level and in the visualization level (see Fig.3).

# 6 Conclusion

In this paper, authors propose a hierarchical correlation network (HcorrNet) to explore the effective connectivity in visual perception by modeling the correlations between perceptual images and voxels from fMRI signals in the entire visual cortex. Our HcorrNet addresses three key problems: how to obtain the correlations, how to explore the stimulus-sensitivity activation patterns on the basis of correlations, and how to model the forward encoding process. Then, by combining the stimulus-sensitive activity patterns and the forwarding process, the most activate voxels in the inferred effective connectivity can be located to guide the decoding process, including classification and image reconstruction. The experimental results of classification and image reconstruction demonstrate that the HcorrNet refines the fMRI signals and enhances the data correlations. Our work can contribute to revealing the working mechanism of the human brain and establishing mapping from perceptual tasks to voxels. However, reliable contrast experiments are desired, and experiments on other fMRI datasets are required to further improve and generalize the HCorrNet framework for practical applications in the future.

# References

1. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning. pp. 214–223 (2017)

2. Eliasmith, C., Stewart, T.C., Choo, X., Bekolay, T., Dewolf, T., Tang, Y., Rasmussen, D.: A large-scale model of the functioning brain. Science 338(6111), 1202–1205 (2012)

3. Fujiwara, Y., Miyawaki, Y., Kamitani, Y.: Modular encoding and decoding models derived from bayesian canonical correlation analysis. Neural computation 25(4), 979–1005 (2013)

4. Gütig, R., Sompolinsky, H.: The tempotron: a neuron that learns spike timing-based decisions. Nature neuroscience 9(3), 420–428 (2006)

5. Hausfeld, L., Valente, G., Formisano, E.: Multiclass fmri data decoding and visualization using supervised self-organizing maps. NeuroImage 96, 54–66 (2014)
6. Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchi-cal visual features. Nature communications 8 (2017)

7. Hu, J., Tang, H., Tan, K.C., Li, H.: How the brain formulates memory: A spatio-temporal model research frontier. IEEE Computational Intelligence Magazine 11 (2), 56–68 (2016)

8. Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L.: Identifying natural images from human brain activity. Nature 452 (7185), 352 (2008)

9. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

10. Kuang, D., Guo, X., An, X., Zhao, Y., He, L.: Discrimination of adhd based on fmri data with deep belief network. In: International Conference on Intelligent Computing. pp. 225–232. Springer (2014)

11. Ma, Y.q., Wang, Z.r., Yu, S.y., Chen, B.d., Zheng, N.n., Ren, P.j.: A novel spiking neural network of receptive field encoding with groups of neurons decision. Frontiers of Information Technology &amp; Electronic Engineering 19(1), 139–150 (2018)

12. Naselaris, T., Olman, C.A., Stansbury, D.E., Ugurbil, K., Gallant, J.L.: A voxel-wise encod-ing model for early visual areas decodes mental images of remembered scenes. Neuroimage 105, 215–228 (2015)

13. Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V.: Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends in cognitive sciences 10(9), 424–430 (2006)

14. Park, H.J., Friston, K.: Structural and functional brain networks: from connections to cogni-tion. Science 342(6158), 1238411 (2013)

15. Penny, W.D., Friston, K.J., Ashburner, J.T., Kiebel, S.J., Nichols, T.E.: Statistical parametric mapping: the analysis of functional brain images. Academic press (2011)

16. Shen, G., Dwivedi, K., Majima, K., Horikawa, T., Kamitani, Y.: End-to-end deep image reconstruction from human brain activity. BioRxiv p. 272518 (2018)

17. Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS computational biology 15(1), e1006633 (2019)

18. Wang, W., Arora, R., Livescu, K., Bilmes, J.: On deep multi-view representation learning. In: Proceedings of the 32nd International Conference on Machine Learning (ICML-15). pp. 1083–1092 (2015)

19. Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J., Liu, Z.: Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral Cortex 28(12), 4136–4160 (2017)

20. Yamashita, O., Sato, M.a., Yoshioka, T., Tong, F., Kamitani, Y.: Sparse estimation automat-ically selects voxels relevant for the decoding of fmri activity patterns. NeuroImage 42(4), 1414–1429 (2008)

21. Yu, S., Zheng, N., Ma, Y., Wu, H., Chen, B.: A novel brain decoding method: a correlation network framework for revealing brain connections. IEEE Transactions on Cognitive and Developmental Systems (2018)

22. Zeeman, E.C.: Topology of the brain (1965)

23. Zheng, N., Liu, Z., Ren, P., Ma, Y., Chen, S., Yu, S., Xue, J., Chen, B., Wang, F.: Hybrid-augmented intelligence: collaboration and cognition. Frontiers of Information Technology & Electronic Engineering 18(2), 153–179 (2017)