Foreground-attention in neural decoding: Guiding Loop-Enc-Dec to reconstruct visual stimulus images from fMRI
Authors: Kai Chen
Yongqiang Ma
Mingyang Sheng
Nanning Zheng
Venue: ICLR 2022 Submitted (2022)
Keywords: neural decoding
visual stimulus image reconstruction
visual attention
encoder-decoder
fMRI

Abstract: The reconstruction of visual stimulus images from functional Magnetic Resonance Imaging (fMRI) has received extensive attention in recent years, which provides a possibility to interpret the human brain. Due to the high-dimensional and high-noise characteristics of fMRI data, how to extract stable, reliable and useful information from fMRI data for image reconstruction has become a challenging problem. Inspired by the mechanism of human visual attention, in this paper, we propose a novel method of reconstructing visual stimulus images, which first decodes the distribution of visual attention from fMRI, and then reconstructs the visual images guided by visual attention. We define visual attention as foreground attention (F-attention). Because the human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not connected in practice. Therefore, it is necessary to consider the global information when decoding fMRI, so we introduce the self-attention module for capturing global information into the process of decoding F-attention. In addition, in order to obtain more loss constraints in the training process of encoder-decoder, we also propose a new training strategy called Loop-Enc-Dec. The experimental results show that the F-attention decoder decodes the visual attention from fMRI successfully, and the Loop-Enc-Dec guided by F-attention can also well reconstruct the visual stimulus images.

Under review as a conference paper at ICLR 2022
FOREGROUND-ATTENTION IN NEURAL DECODING: GUIDING LOOP-ENC- TO RECONSTRUCT VISUAL STIMULUS IMAGES FROM FMRI
Anonymous authors Paper under double-blind review
ABSTRACT
The reconstruction of visual stimulus images from functional Magnetic Resonance Imaging (fMRI) has received extensive attention in recent years, which provides a possibility to interpret the human brain. Due to the high-dimensional and high- noise characteristics of fMRI data, how to extract stable, reliable and useful in- formation from fMRI data for image reconstruction has become a challenging problem. Inspired by the mechanism of human visual attention, in this paper, we propose novel method of reconstructing visual stimulus images, which first decodes the distribution of visual attention from fMRI, and then reconstructs the visual images guided by visual attention. We define visual attention as foreground attention (F-attention). Because the human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not connected in practice. Therefore, it is necessary to consider the global information when decoding fMRI, so we intro- duce the self-attention module for capturing global information into the process addition, in order to obtaín more loss constraints
of decoding F-attention. the training process of encoder-decoder, we also propose new training strat- egy called Loop-Enc-Des The experimental results show that the F-attention de- coder decodes the visual attention from fMRI successfully, and the Loop-Enc-Dec guided by F-attention can also well reconstruct the visual stimulus images.
INTRODUCTION
In recent years, reconstructing visual stimulus images from fMRI has gradually gained attention, which provides the possibility of "mid reading" in the future (Fig. D. Existing work has shown that there is certain mapping relationship between visual stimuli and brain activity (Poldrack & Farah, 2013). which provides us with the possibility and basis for reconstructing visual stimuli from fMRI data. The fMRI data which is collected from the human brain records the variations in blood oxygen level dependent (BOLD) and reflects the activity of nerves in human brain. Through the analysis of fMRI data, the correlation between brain activity and visual tasks be explored, helping to understand human visual mechanism better.
Prior studies. In recent years, there has been lot of research in this field. The main methods can be roughly divided into the following two categories: Linear regression models, which encode fMRI data into image pixel values, and finally achieve image reconstruction. Deep learning models, such as DCNN and GAN model frameworks.
For the first type of methods, Yoichi Miyawaki et al. applied multi-scale linear weighting model to predict the pixel value of each image and obtain the reconstruction results of the black and white simple images (Miyawaki
al., 2008). Fujiwara et al. proposed to use Bayesian canonical cor- relation analysis (BCCA) to build reconstruction model, which extracts image information from measured data and reconstruct images (Fujiwara et al. 2013). Marcel AJ van Gerve et al. adopted a hierarchical generative model composed of Boltzmann machines with restricted conditions, which explores the reconstruction of feature hierarchies based on learning (Van Gerven et al. 2010). Yu et al. advanced correlation network framework that could be flexibly combined with diverse pattern representation models. They revealed the effective connectivity of human brain and reconstructed the visual stimulus images (Yu & Zheng. 2017
Yu al. 2018).

---

Under review as a conference paper at ICLR 2022
For the second type of methods, Guohua Shen et al. proposed an end-to-end deep convolution model to reconstruct visual stimulus images directly by using fMRI data, and at the same time they used GAN framework to optimize the training of the model (Shen et al. 2019a). In response to the current problem of the lack of data sets used to reconstruct stimulus images from fMRI data, Roman Beliy al. suggest an approach for self-supervised training on unlabeled fMRI data (Beliy et al.) 2019). Yunfeng Lin et al. proposed model called DCNN-GAN by combining a reconstruction network and GAN module. They utilized the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images (Lin et al.) (019). Tao Fang et al. defined a Shape- Shape-Semantic GAN, considering the functional differences of different visual cortex areas, and reconstruct visual stimulus images under the guidance of shape and semantic information (Fang et al. 2020). The ideas of DCNN and GAN have also been applied to various other model frameworks to achieve the reconstruction of visual stimulus images (Seeliger et al., 2018 VanRullen & Reddy 2019 Zhang
ct al. 2020 Qiao et al. 2020; Mozafari et al. 2020).
Visual stimulus images
FMRI
Reconstruction images
reconstruct
Figure 1: The work in this paper. The purpose of this article is to extract useful information from the visual cortex of the human brain, and build model to reconstruct visual stimulus images from fMRI data by using the uesful information.
Our contributions. Because of the complex neural information in the visual cortex, how to extract valuable information from high-dimensional fMRI data has been a challenging problem. Inspired by human visual attention mechanism, we introduce visual attention to this work for the first time. We first decode the human visual attention from small amount of fMRI data successfully, and then use visual attention to guide the work of reconstructing visual stimulus images from fMRI. Existing studies have shown that the human visual system is more inclined to focus on prominent objects. and the neural representation of these prominent objects in the brain is more obvious (Ungerleider & Gj 2000 Braun al. 200T). The distribution of attention leads to an information bottleneck, only the most prominent objects are allowed to appear in the inferior temporal cortex, especially the ventral visual stream that encodes the identity of the object. The visual attention mechanism is crucial for simulating the neural response of the higher visual system (Poggio & Anselmi 2016 Khosla al. 2020). Based on the above conclusions, we propose Foreground-atten -attenttonnnttntiionnnn (F-attention). We abstract the distribution of human visual attention as the most prominent foreground object distribution area in the images. Through the constructed F-attention decoder, the fMRI corresponding to the natural stimulus images can be decoded into visual attention distribution (Fig. 2))
Because human brain is strongly wound into sulci and gyri, some spatially adjacent voxels are not directly connected in the human brain (Tallinen et al. 2014). the correlation between voxels is not only determined by location. Therefore, we need to consider global information when decoding IMRI, so we first introduce the self-attention module (Vaswani et al. 2017 Wang al. 2018 Zhang et al.) 2019) into the fMRI decoding process, the self attentooo module can capture global information of the input data and expand the receptive field (Fig. 3=).
At the same time, we propose new encoding and decoding framework called Loop-Enc- Enc-Dec (Fig. 2).c). The first step we pre-train the fMRI encoder on the data set to realize the encoding process from images to fMRI. The second step we train the end-to-end encoder-decoder model under the guidance of F-attention, and input the images decoded by the decoder into the fMRI encoder for encoding to add the re-encoding constraint in the loss function, that is called "Loop-Enc-eec", In the training process, natural pictures without fMRI data are also added for end-to-end self-enc-dec training. which increases the stability and generalization of the model. After experimental eval- uation, the performance of mainstream methods. The main contributions of this article follows:
under the guidance of F-attention is better than that of
2

---

Under review as a conference paper at ICLR 2022
As far as know, first introduced the visual attention mechanism to the work of reconstructing visual stimulus images from fMRL. According to neuroscience research, we abstract visual attention the attention distribution of foreground objects (F-attention), and build visual attention decoder to decode the human visual attention distribution from fMRI successfully.
To our best knowledge, we first introduced the self celf-attention module into the work of fMRI decoding to capture global information fMRI data.
We propose new enc-dec framework called Loop-Enc-Dec, guided by F-attention, which adds more loss constraints to the training the image reconstruction model. Evaluating the quality of reconstructed images, our method outperforms previous works.
2 METHODS
In this section, we first introduce the data used in the experiments, then introduce the details of F-attention and Loop-Ene-Dec framework proposed in this article, as well as the step-by-step training strategy for the model. Finally, our evaluation method is given.
2.1 FMRI DATA SET
We use a publicly available benchmark data sets (Horikawa Kamitani 2017). which widely used in the work of visual stimulus image reconstruction. In the image presentation experiment, IMRI signals were measured while subjects viewed sequence object images from ImageNet (Dnng et al. 2009p. The image presentation experiment consisted of two sessions: the training image session and the testing image session. In the training image session, 1200 images from 150 object categories (8 images from each category)
each presented once. In the testing image session, 50 images from 50 object categories (one image from each category) were each presented 35 times. The data collectors performed their analysis for each combination of feature types/layers and brain regions of interest (ROIs: V V1-V4, the lateral occipital complex (LOC), fusiform face area (FFA), parahippocampal place area (PPA), lower visual cortex (LVC; V1-V3), higher visual cortex (HVC; covering regions around LOC, FFA and PPA) and the entire visual cortex (VC: covering all of the visual subareas listed above), the voxel size 3x3x3 mm3) (Horikawa Kamitani 2017).
a
FMRI
F-attention decoder
F-attention maps
Visual stimulus images
Pre-train encoder
FMRI
1MRI encoder
C
Guide
Visual stimulus images
fMRI encoder
Reconstruction images
E
D
Image decoder
Loop
Loop-Enc Dec
Figure 2: The overall model framework of the paper. (a) The overall structure of F-attention decoder. (b)(c) The Loop-Enc-Dec We first pre-train a IMRI encoder to encode images to IMRI, and then use an image decoder guided by F-attention to reconstruct images, during the training, decoded images loop back to fMRI encoder.

---

Under review as a conference paper at ICLR 2022
2.2 PROPOSED MODEL
Summarizing the proposed model. We first trian the F-attention decoder on fMRI data 2h.. then pre-train the image encoder(Fig. PD). next fix the weight of the encoder, train the de- coder under the guidance of F-attention(Fig. 2 When testing, we first use fMRI as the input of F-attention decoder to decode the distribution of F-attention, then input the fMRI into the image decoder to reconstruct the visual stimulus images guided by corresponding F-attention.
2.2.1 F-ATENNTION
In order to get more useful information from fMRI data to guide the work of visual stimulus image reconstruction, introduced the human visual attention mechanism into the work. We abstracted the human visual attention as foreground attention (F-attention) based on neuroscience research PPoggio & Anselmil 2016; Khosla et al. 2020). By constructing a F-attention decoder, decode human visual attention distribution from fMRT data.
a FMRI
Fatttention decoder
F-attention map
foroground
F-attention
F-attention map
Fttteenionn labeling process
self-attention module
[MRI
Conv and upsampled x3
self attention map
1xl conv
softmax
conv
transpase
fully connection
self attention
feature
Conv and upsampled
1x1 corw
x2
Foreground-attention decoder
(a)The overall structure of F-attention decoder, through the F- attention decoder, we can decode the human visual attention distribution from the visual neural representation of the brain. (b)The human visual attention distribution labeling process of training set. (c) detailed structure of F-attention decoder.
Figure 3: Foreground-attentnee
Visual attention labeling. For the purpose of obtaining the human visual attention distribution of the training set, we first label the train data (Fig. (3)). According to the definition of F-attention, we first extract the prominent objects the natural images, and then perform the binary value on the images. Through the transformation operation, the main distribution areas of F-attention obtained. Then we assigned attention weights to different regions the images (the bluer the color, the higher the attention weights, in the train set, the weights of the foreground object areas are the background areas are 0), the final F-attention maps are obtained as the human visual attention labels of the training set.
F-attention decoder. Then we construct F-attention-deccode to realize the decoding process from fMRI data to human visual attention(Fig. ( First, the preprocessed fMRT data integrated into a vector, then the vector sequentially connected to a fully connection module, convolution and upsampling modules, self-attention module, convolution and upsampling modules, and finally get the F-attention map. The self-attention module calculation formula is as follows, dy is 1;
QKT
Self Attention (Q, K,V) softmar(
V
dO
In the training process, we use the mean square error loss (MSE) to constrain. After 400 rounds of iterations, the error decreases and stabilizes. The loss function is follows:
Loss attention (FAr FA7) [(FA FA?)
(2)
4

---

Under review as a conference paper at ICLR 2022
Where FAA, represents the visual attention distribution corresponding to the images in the training set, and FA stands for the visual attention distribution decoded by the F-attention decoder.
2.2.2 LOOP-ENC-DEE
The Soop-Enc-Doe reconstruction model we proposed is divided into two steps for training, pre- training encoder (Fig. 2b) and training Loop-Ene-Decoder guided by F-attention (Fig. 2b).
Pre-train IMRI encoder. The fMRI encoder realizes the encoding process from natural images to fMRI. The structure of the encoder is mainly composed of residual network, convolution and full connection. The front end of the encoder is used as ResNet50 network (He et al.) 2016) pre-trained on the ImageNet dataset, this part extract image features (Fig. 4 The loss function composed of the weighted sum of the mean square error (MSE) and the cosine similarity, as shown below:
y
Σ (1, =z}² Σ cos (2,,
Where a and |3 are hyperparameters, I, is the original fMRI label, and x is the encoding result. In this paper, a is assigned a value of 0.85 and 3 is -0.15, This loss function is also used in the training process of Loop-Enc-Dec.
Train Loop-Enc- Decoder guided by F-attention. We fix the weight pre-trained fMRI encoder, then connect image decoder to form end-to-end enc-dec model. The so-called 'Loop' idea is to re-input the reconstructed images into the encoder for cyclic encoding in the process of training Enc-Dec, increasing the fMRI encoding loss constraint, so that the original end-to-end model is transformed into a loop structure. The image decoder is composed of full connection, convolution and upsampling. At the same time, F-attention is used weight multiply the convolution feature to guide the training process of the model (Fig.
Loop-Enc-Dec
AMRI
FA-D
(MRI Encoder
trained
Image decoder
image
Low Loog
PC
RedNet50 front part
CoNN
conv
0
Pre-trained
& upsampled x3
Loop
Detailed structure of Loop-Enc-Dee We fix the network parameters of the pre-trained encoder, and train the Image decoder under the guidance of F-attention to realize the work of visual stimulus image reconstruction.
Figure 4:
The F-attention (FA) decoded by pre-training F-attention decoder and the convolution features (ConuF) are used as Hadamard product, the output weighted À and added to the original input convolution feature weighted μ. Then complete the guidance of the attention weight, shown in the following formula:
FeatureFA = XFA ConvE + PCoovFF
(4)
In the training process, À is assigned a value of 0.7 and μ 0.3. We also add unlabeled ImageNet images for end-to-end unsupervised self-enc-dec to enhance the generalization ability of the model. The loss function in the training process consists of three parts, as follows:
Loop Losstmage + Loss Noss
(5)
Next, we will introduce the three components of Loss Eoop in detail.
1) Lossimage- During training, as shown in Fig. 4 the image decoder has two inputs, one is the original fMRI data, input it into image decoder generate image imgi, and the other is the
5

---

Under review as a conference paper at ICLR 2022
output of pre-trained fMRI encoder(encoding the visual stimulation image), input it into image decoder to decode image img2. Then as shown in Fig. 4 we use VGG19 (Simonyan & Zisserman, 2014) feature extractor extract the convolution features of visual stimulation images and reconstructed images (img), imgz), then calculate the loss between convolution features as shown in Eq. 6
LosSimage x Σ [im(g)))) - (img))) + (iv(i)))) - (img))]*
(6)
Where V represents the feature extractor composed of pre-trained VGG19.
2) Loop- Then we loop img1, img2 back into the fMRI encoder to encode two different features, which we names fea1. feaz. We use Loss to calculate the loss between encoded features (fea1, fea2) and raw fRMI data, shown in Eq. 7
Loss = Lo ( (fea1, + [oss)arnr((e22.aa
Where x represents the raw fRMII J is defined in Eq.
3) LosSimageNett- In order to enhance the generalization and robustness of the model, introduce the image data without fMRI label from imageNet in the training process. Firstly, the image is input into the IMRI encoder for encoding, and then the encoding result input into the image decoder for decoding to obtain the reconstructed image imgs. We also calculate the loss between VGG19 convolution features.
Los8 Net [i(iing)) V (imggNett)
(8)
Where LOSsyMRI MMRI is Eq. 3 and imgNet et is the imageNet image without fMRI labels.
2.3 EVALUATION METHOD
This paper uses a quantitative method to evaluate the reconstructed visual stimulus images. The evaluation method is called pairwise similarity comparison (Shen et al. [2019b] |ang et al.) (2020). use 2-way method used in Fang et al. (2020). There are total of 50 natural stimulus images in the test set. For each reconstructed image, it is calculated SSIM (Wang et al. 2004) value separately from the 50 natural stimulus images. We define the SSIM between the reconstructed image and the corresponding original image as MSIM and the SSIM with the remaining 49 as Mynndoo then SSIMpair and SSI Mrandoo compare the SSIM values in pairs, total of 50x49=2450 com- parisons, we count the correct of times (SSI Mpair is greater than SSI Mradomm) ) and calculate the correct (Shnn al. 2019b). We will supplement the details of this part the appendix.
RESULTS
In the following, will present the F-attention decoding results, visual stimulus image reconstruc- tion results, some ablation experiments involving the model, and verify some conclusions in neuro- science. All our experiments are performed the data set Horikawa2017 (Horikawa & Kamitani, 2017). There are a total of 1200 images in the training set and a total of 50 images in the test set. There is overlap in image categories between the training set and the test set.
3.1 F-ATTENTION DECODEING RESULTS
We use entire visual cortex (VC) of fMRI data labeled with visual attention distribution for training. A total of 280 rounds of experiments are performed. The learning rate is set by using the SGDR method (Loshchloov & Hutter 2016). According to SGDR, we divide the 280 rounds of training into four small rounds. The learning rate of each small round rises to the maximum value of 0.03 through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing. Each small round repeats this operation realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model, the details are shown in the appendix. The results of decoding the F-attention distribution from IMRI are shown in Fig. 5 the more blue the area, the higher the weight of visual attention. From the results, it can be seen that the F-attention decoder we design can decode the distribution of visual attention successfully, whether the object is an animal, man made object, human, such the fourth image in the second row, the duck in the grass can be accurately found, the seventh image in the fourth row, visual attention distribution area of the person lying the hammock can also be accurately found.

---

Under review as a conference paper at ICLR 2022
Seeing images
F-attention maps
Seeing images
F-attention maps
Figure 5: F-attention map decoding results of test data set. The more blue the area, the higher the weight of visual attention. The distribution of F-attention in the images can be accurately found.
3.2 VISUAL STIMULUS IMAGES RECONSTRUCTION RESULTS
In this part, we present the reconstruction results of our method and compare the results with other
existing works (Shen et al.) 2019a Belly et al.) 2019, Fang et al. 2020; Mozafari et al., 2020), the comparison results are shown in Fig. 6 Through the evaluation method of pairwise similarity comparison mentioned above, quantitatively evaluate the results of the experiment. It can be seen from the experimental results that our method can effectively reconstruct the images seen by humans from fMRI, and better decode the shape and direction of the prominent object. Compared with the previous work (reconstruction results of the visual stimulus images shown in Fang et al.) 2020)), ) some improvements have been obtained. From subjective perspective, our results can better pay attention to the distribution of foreground objects in the images. For example, for the reconstruction of the first goldfish image, the position and color of the goldfish in the image are more accurate. From quantitative comparison point of view, our method experiment results achieve 68.9% by pairwise SSIM comparison, slightly better than 68.4% (Beliy et al. 2019), ) and better than 66.8% (Fang et al. 2020), 63.8% (Shen et al.) 2019a) and 54.3% (Mozafari et al. (0220), indicating the effectiveness of our method and foreground-attenn reconstruction of visual stimulation images successfully.
can guide loop-Enc-Ded complete the
3.3 MODEL ABLATION EXPERIMENTS
re- construction of visual stimulus images, the results are shown in Fig. 7 First, give the results obtained using the complete method this paper (the second row of Fig. (7]. the pairwise SSIM value 68.9%. Then on the basis of the complete method, the F-attention guidance is removed from the training of the decoder, and only the Loop-Enc-Dee structure is used for step-by-step train- ing. First, the fMRI encoder is pre-trained, then the Loop-Ene-Dec is trained. We finally use the image decoder to test and get the reconstruction results (the third row of Fig. 77. the pairwise SSIM value 66.7%. Then we remove the LossLoop on the basis of the complete method, just use the image loss of end-to-end encoding, and at the same time train the image decoder under the guidance of F-attention and obtain the reconstruction results (the fourth row of Fig. 7 The value of pairwise remove the F-attention guid-
Here, we conduct ablation experiments to discuss the role of F-attention and Loop-Enc-De
SSIM is 65.0%. Finally, the basis of the complete method, ance and the LossLoop- and only use the end-to-end enc-dee for step-by-step training. The image reconstruction results are shown in the fifth row of Fig. 7 the pairwise SSIM value is 62.5%.
Comparing the ablation experimental results, it is not difficult to see that our proposed F-attention and LossLoop have played positive role in the image reconstruction work. Comparing the results in the second row with the results the fifth row, it can be seen that under the combined effect of F-attention and Loss Loop, the image reconstruction quality has been greatly improved, including

---

Shen Under review as a conference paper at ICLR 2022
Seeing images
Ours
correct
Fang
et al.
Belly
et al.
et al.
30
ours
Tao Fang al.
Milad Mozafari et al.
Belly al.
Mozafari
Figure 6: (a) Partial reconstruction results compared with other methods. (b) Result comparison with pairwise similarity, As mentioned in Fang et al. (2020). the quantitative evaluation value is the average result five experiments
the shapes and colors of the prominent objects the images, 68.9% is better than 62.5% pairwise SSIM quantification.
b
Seeing imagos
Full method
by coFnect
F-attention ablation
48
Los8
Full method
F-actention ablation
Lossuo ablation
F-attention ablation
LosS ablation
Figure 7: Model ablation experiments. (a) Visual stimulus images reconstructed by full method, full method without F-attention, full method without LosStoop- full method without F-attention and LossLoop- (b) Comparing ablation results by pairwise SSIM.
Comparing the ablation experimental results separately. Comparing the results of the second row and the third row, it can be seen that F-attention plays certain guiding role in the model, making the reconstruction of foreground objects in the images more prominent, such as the goldfish in the water in the first column, and the people lying in a hammock in the second column, pairwise SSIM quantitative comparison shows that 68.9% is better than 66.7%. The guiding effect of F-attention can also be obtained by comparing the results of fourth rows and fifth rows. The reconstruction results of foreground objects of fourth row are more prominent, and pairwise SSIM quantitative comparison shows that 65.0% is better than 62.5%. Comparing the results of the second row and the fourth row, it can be seen that the Loss Loos has made the details of the reconstruction images more clear and accurate. For example, the face of the leopard in the fifth column is clearer, and the shape
8

---

Under review as a conference paper at ICLR 2022
and details of stone pan comparison shows that 68.9% is better than 65.0%. The constraint effect of Loss can also be obtained by comparing the results of the third row and the fifth row. The details and quality of the image reconstruction in the third row are better. The pairwise SSIM quantitative comparison shows that 66.7% is better than 62.5%. According to the above ablation experiments, proved the guiding role of F-attention in image reconstruction and the constraint role of Loss in the training process.
accurate in the seventh column. The pairwise SSIM quantitative
3.4 VALIDATION OF NEUROSCIENCE RESEARCH
We verify some results of neuroscience research through experiments in the following. is generally believed that in the human visual cortex, the low-level visual (LVC) cortex often pays attention to the shape and contour information of objects, and has significant directional sensitivity, while the high-level visual cortex (HVC) may correspond to the category information, movement information and other higher-level information of objects (Carandini et al. 2005 DiCarlo et al., 2012, Bracci jet al., 2017: Zeman et al. 2020). We reconstruct the visual stimulus images with the fMRI data of the low-level visual cortex and the high-level visual cortex respectively. The results are shown in the Fig. 8 the pairwise SSIM value of the images reconstructed from LVC is 64.7%, and HVC 55.7%. By comparing the experimental results,
find that the LVC reconstruction results largely restore the shape and direction information of the foreground objects in the images, while the HVC reconstruction results lose the shape and direction information of the foreground objects. Quantitatively, the pairwise SSIM value of LVC is also better than HVC. This results verify that low-level visual cortex (LVC) pays more attention to the shape and contour information of objects than hige- visual cortex (HVC).
Seeing images
VC resulli
connect
HVC results
LVC results
LVC
Figure 8: Validation of neuroscience research. (a) Visual stimulus images reconstructed from differ- ent visual cortex areas, including VC, HVC and LVC. (b) Result comparison with pairwise SSIM.
4 CONCLUSION
In this paper, inspired by human visual attention, we define the Foreground-attention distribution of people when they view natural images, and successfully decode the visual attention distribution (Foreground-attentin
from the IMRI data of human brain. We achieve the decoding and interpre- tation of the visual attention of the human brain. The results show that areas with higher attention are concentrated on prominent objects in the images, such as animals and animal faces in the im- ages, while lower attention is given to messy background information. This result is in line with the research conclusions on human brain visual attention (Ungerleider & G 2000 Braun et al.) 2001 Poggio & Anselmi 2016 Khosla et al., 2020). At the same time, we propose a new enc-dec train- ing strategy called Enop-Enc-Dec, and apply the decoded F-attention to guide it, then reconstruct the visual stimulus images from the (MRI data successfully. Compared with the previous work, the reconstructed visual images in this paper are more realistic and get higher score on pairwise SSIM. From the perspective of neuroscience, we verify the following conclusions through exper- iments: The low-level visual cortex pays more attention to the shape of objects in the scene and has significant directionality, while the high-level visual cortex is insensitive to shape and direction information.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Roman Beliy, Guy Gaziv, Assaf Hoogi. Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri Advances in Neural Information Processing Systems, 32:6517-6522 17-277 2019.
Stefania Bracci, J Brendan Ritchie, and Hans Op de Beeck. On the partnership between neural representations of object categories and visual features in the ventral visual pathway. Neuropsy- chologia, 105:153-1666 2017.
Jochen Braun, Christof Koch, Joel Davis, al. Visual attention and cortical circuits. MIT Press, 2001.
Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Ol- shausen, Jack Gallant, and Nicole Rust. Do we know what the early visual system does? Journal of Neuroscience, 25(46):00577-1055)
2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 43(3):415-444 2012.
Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape- semantic gan. Advances in Neural Information Processing Systems, 33, 2020.
Yusuke Fujiwara, Yoichi Miyawaki, and Yukiyasu Kamitani. Modular encoding and decoding mod- els derived from bayesian canonical correlation analysis. Neural computation, 25(4):979-1005, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):1-15, 2017.
Meenakshi Khosla, Gia Ngo, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Neural encoding with visual attention. Advances in Neural Information Processing Systems, 33, 2020.
Yunfeng Lin, Jiangbei Li, and Hanjing Wang. Denn-gan: Reconstructing realistic image from fmri. In 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6. IEEE, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 160.008800. 2016.
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki Tan- abe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using combination of multiscale local image decoders. Neuron, 60(5):915-92) 2008.
Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstructing natural scenes from finr pat- terns using bigbigan. In 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2020.
Tomaso Poggio and Fabio Anselmi. Visual cortex and deep networks: learning invariant represen- tations. MIT Press, 2016.
Russell A Poldrack and Martha J Farah. Progress and challenges in probing the human brain. Nature, 526(7573):311-37,, 2015.
Kai Qiao, Jian Chen, Linyuan Wang, Chi Zhang, Li Tong, and Bin Yan. Biggan-based bayesian reconstruction of natural images from human brain activity. Neuroscience, 444:92-105, 2020.
10

---

Under review as a conference paper at ICLR 2022
Katja Seeliger, Umut Güşlüü, Luca Ambrogioni, Yagmur Gügglütürk, and Marcel AJ van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018.
Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End- to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuro- science, 13:21, 2019a.
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruc- tion from human brain activity. PLoS computational biology, 15(1)=e1006633 2019b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Tuomas Tallinen, Jun Young Chung, John S Biggins. and L Mahadevan. Gyrification from con- strained cortical expansion. Proceedings of the National Academy of Sciences, 111635):12667- 12672, 2014.
Sabine Kastner Ungerleider and Leslie G. Mechanisms of visual attention in the human cortex. Annual review of neuroscience, 23(1):315-341, 2000.
Marcel AJ Van Gerven, Floris P De Lange, and Tom Heskes. Neural decoding with hierarchical generative models. Neural computation, 22(12):312--3142, 2010.
Rufin VanRullen and Leila Reddy. Reconstructing faces from fmri patterns using deep generative neural networks. Communications biology, 2(1):1-10, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from eтror visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Siyu Yu and Nanning Zheng- A correlation network framework for revealing brain connections in brain decoding. IEEE Workshop Nonlinear Circuit Networks, 2017.
Siyu Yu, Nanning Zheng, Yongqiang Ma, Hao Wu, and Badong Chen. A novel brain decoding method: A correlation network framework for revealing brain connections. IEEE Transactions on Cognitive and Developmental Systems, 11(1):95-106, 2018.
Astrid Zeman, Brendan Ritchie, Stefania Bracci, and Hans Op de Beeck. Orthogonal repre- sentations of object shape and category in deep convolutional neural networks and human visual cortex. Scientific reports, 10(1):1-12, 2020.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference machine learning, pp. 7354-7363 PMLR, 2019.
Yichen Zhang, Shanshan Jia, Yajing Zheng, Zhaofei Yu, Yonghong Tian, Siwei Ma, Tiejun Huang, and Jian K Liu. Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Networks, 125:19-30 2020.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DETAILS OF MODELS AND TRAINING
In this paper, the details of the F-attention decoder we designed are as follows, the input fMRI data is one-dimensional vector, and the fully connected layer is connected, the output size is 512* 10 10, then reshape it into the feature of size (10, 10, 512). The convolutions has step size of 2 and a kernel size of 4. The dimension of Query and Key is 68*68*8, and the dimension of Value 68*68*64. In Loop-Enc-Dec, the front-end ResNet50 feature extractor in the fMRI decoder selects 'conv3- block4-out' as the feature output, the output feature size is 14 14#12,2 followed by two layers of residual networks, one layer of max- max-pooling and convolution (the size of the convolution kernel is 3, the step size is I, padding connection. In order to eliminate the checkerboard effect that appears in the reconstructed image, in image decoder, we set size the convolution kernel to twice the convolution step length, and linear interpolation for each upsampling to expand the feature to four times the original.
'same'), and finally outputs the encoded results through a layer of fully
A.2 CHANGES IN LEARNING RATE USING SGDR
The linear function and cosine annealing function used in each round are follows:
(Eeur (ober11)je +1)+bmaz Frour < Fry Eutl
ISSGDR
(9)
bmin + 1 (Iman (minn ) * (1 + cosl Emgr Ey
Eper Em }), Eeur 2 Ewarm
Ecur counts how many epochs have been performed since the last restart, En represents the 'warm' epochs of each small round, Eper represents the total epochs of each small round, 4maz represents the maximum learning rate, and Imin represents the termination learning rate.
Changes in learning using SGDR
0.030
0.025
0.010
0.005
0.000
Figure 9: Changes in learning rate using SGDR
total of 280 rounds of experiments were performed. The learning rate set using the SGDR method (Loshchilov & Hutter 2016). According to SGDR, we divide the 280 rounds of training into four small rounds. The learning rate each small round rises to the maximum value 0.03 through a linear function, and then decays to the minimum value of 4e-5 through cosine annealing. Each small round repeats this operation to realize the periodic restart of the learning rate and improve the convergence speed and generalization of the model.
12

---

Under review as a conference paper at ICLR 2022
Table Determining the value of and p Eq
0.1
0.3
0.5
0.7
0.9
0.9
0.7
0.5
0.3
0.1
PairSSIM
0.660
0.676
0.671
0.689
0.681
A.3
THE DETAILS OF SSIM
SSIM is an evaluation index to measure the similarity of two images, the calculation formula as follows:
(2pziyy + c2) + c2)
SSIM (n² + p3 c o 02)
(10)
Where x and are two images, Ma is the mean of a, My is the mean of o² is the variance of x, o is the variance of any is the covariance of x and maintain stability. I. is the dynamic range of pixel values. =0.01, =.033.
(k1 2.) c2 = (k:2 L,),, is constant used
A.4 HOW TO DETERMINE THE VALUE OF À AND IN EQ
We tested the sequential combination parameters of μ 0.1, 0.3, 0.5, 0.7, 0.9 and X > 0.7, 0.5, 0.3 and 0.1 respectively, and then selected the best combination, == and = 0.3, as shown in It may not be the best among all the values of X and jr, but is appropriate.
Table T
A.S CHANGE LOSS FUNCTION DURING TRAINING
In the process of training, the loss function Eq. 5 of each part can converge well. As shown in Fig- TO figure a) shows the downward trend of total loss LossLoop- Enc- Dec the training process, fig- ure b) shows the downward trend of Loss figure c) and d) show the change trend of Lossimages and figure e) shows the change trend Lossemage The overall trend of error decreases and can be stabilized to certain level. The cyclical rise and fall is due to the using of SGDR learning rate change strategy.
d
Figure 10: Change of loss function during training
13


Reviews:
[{"review_id": "zt9YsUE9xPz", "reviewer": "Reviewer_48tB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is that, (if I understand the training process correctly) the label information used for training the attention decoder is from another attention map generator (the F-attention labeling process), which produces artificial attention map. Then what is the meaning of fitting the BOLD signal to the artifical attention map (see Fig.3a&b)?  Obviously this kind of fitting can give an attention map based on the fMRI data, which is similar to the F-attention map given in Fig.3b after optimization. However, this may not reflect what the human attention looks like in this task. \n \nOther concerns:\n1) The description of 'Visual attention labelling' should be made more clearer, e.g., how to extract the prominent objects in the images, is it a DNN-based objected detection algorithm? What does \"perform a binary value\" mean? What is a \"tranform operation\"?\n2) The MSE loss in high dimension may not be easy to optimize (Eq.3, also in Eq.4). Could you please provide some demonstration of the training loss process, and give more illustration of how far the optimization goes?\n3) The authors should make more clearer of the contribution of each part in their proposed model. Even though each part in explained in Sec.2, it is still very hard to know the underlying reason of such design. In addition, the ablation study does not make much sense. The increase of performance can be explained by including more training parameters. \n4) The choose of  $\\alpha$ and $\\beta$ in Eq.4 (also $\\lambda$ and $\\mu$ in Eq.5) seems to be empirical. \n5) The example of the decoded goldfish is not representative, because the decoded position of other images are also very accurate, as showed in Fig.6. \n6) The argument in Sec.3.4 is wrong. Firstly, the authors should define clearly about LVC and HVC. Otherwise it is hard to say which one is more responsible for the shape perception and contour perception. At least, it shouldn't be in primary visual cortex.\n7) What is $d_k$ in Eq.2?\n\nSuggestions:\n1, The F-attention is the key in this work, but also a strong hypothesis assumed by the authors that human will adopt to do image classification in such fMRI task. However, the authors should be careful when use this term, as this kind of simple task may not require attention in the human brain. On the other hand, even the brain activation (at the temporal lobe) may reflect the attention info, it is unclear which kind of attention will be involved, only bottom-up attention, or a mixture of top-down feedback and bottom-up attention [1]? So it is better to use more accurate terms, such as 'visual saliency map [2]', etc.\n2, To get the attention labels, it is better to use the eye tracking data which is a more direct reflection of the attention of participants, rather than relying on the algorithm-based attention labeling. Such kind of data (3T fMRI + eye tracking data) may be available on the website.\n3, The terms in Eq.7&8&9 should be showed in the figure to help understanding. \n\n[1] Gilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350-363.\n\n[2] Li, Z. (2002). A saliency map in primary visual cortex. Trends in cognitive sciences, 6(1), 9-16.\n", "summary_of_the_paper": "The authors proposed a model to decoding the fMRI signal from the human visual cortex by introducing the Foreground-attention. They also proposed a enc-dec training strategy called Loop-Enc-Dec, which is guided by the F-attention, to successfully reconstruct the visual images from the fMRI data. A higher score based on pairwise SSIM is achieved compared to previous works.", "summary_of_the_review": "The idea using F-attention to guide the network training is fine from the view of network training, but the way of using it to do image reconstruction from the FMRI data seems incorrect.  Fitting the F-attention model to the artifical attention map does not make much sense.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i4ZeURLM9No", "reviewer": "Reviewer_Pz1R", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe encoder decoder loop and use of foreground is relatively novel.\n\nWeakness: \nIt is not very convincing to me that the loop-enc-dec model and the use of foreground attention is able to reconstruct stimuli images from brain responses better than past attempts. The one (if not only) quantitive comparison the paper makes with past results (as in Figure 6) shows that this method barely has any improvement over other methods. In terms of qualitative comparison, it is not clear to me the reconstruction provided by this methods is better than those generated by other methods. \n\nOther than that, this paper would benefit a lot from a more thorough editing - some wordings are very confusing and there are a lot of typos. For example, \"deep learing models\" in the second paragraph of the introduction. And in the results section (3.2), 68.4% should be from Beliy et. al, not Fang et. al.\n\n", "summary_of_the_paper": "This paper proposes a novel method incorporating decoded foreground attention and a new training scheme with encoder and decode in a loop to reconstruct image stimuli from fMRI data.", "summary_of_the_review": "This paper has some interesting and innovative ideas about recounting visual stimuli from brain responses. However the paper is not very well written, hard to understand and the results are not very convincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VFIR3YdfFU", "reviewer": "Reviewer_UCCu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Brain decoding is an interesting and still very much under-explored area\n* Proposed a new framework by introducing the F-attention to enhance the image reconstruction further. \n\nWeaknesses:\n\n* The experimental section is weak, and hard to interpret the reconstruction results.\n** Previous works had shown reconstruction results on at least two datasets. Why do authors consider only one fMRI natural image dataset of 1250 images? (Why not the BOLD 5000 fMRI dataset? And Vim-1 dataset)\n** Authors can also report the 2-way, 5-ways, and 10-way results for better comparison with previous methods. \n** From Figure 6, Without any statistical significance test on all the results, it is difficult to compare which method performs better than the remaining methods.\n** Also, the proposed method has marginal improvement (68.9%) in terms of SSIM over the previous method (68.4%). No clear explanation for the comparison of the results.\n** Figure 4 shows that ResNet is used as a pretrained encoder for extracting fMRI from stimuli, and what is the purpose of VGG19 in the Loop-Enc-Dec architecture? Throughout the paper, there is no discussion on VGG19.\n** Any insights on self-attention?\n** What is the dimension of Query, Key, and Value weight matrices?\n** How is SSIM calculated?\n** How do authors obtain λ value as 0.7 and μ as 0.3?\n* Paper: M. Mozafari, L. Reddy and R. VanRullen, Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN, this paper is not cited at all, and no comparison with previous works in terms of evaluation metrics. \n* None of the image reconstruction results look comparable to Ground Truth or with previous methods.\n* Cognitive insights are missing in this paper.\n* Figure 8 illustrates that Lower visual areas have better reconstruction performance than Higher visual areas? Does this indicate the proposed model is reconstructing shapes and corners more? Is it because the number of voxels present in LVC is more than HVC?\n* The total number of voxels present in VC?\n* Among all the brain ROIs, which brain ROI is highly involved in image reconstruction?\n* Using the Loop-Enc-Dec method to reconstruct natural images from fMRI is not new to this field.\n* The mathematical notations used in Equations 5, 6, 7, 8, and 9 are difficult to follow.\n** The naming conversion is bad.", "summary_of_the_paper": "The paper proposed a Loop-Enc-Dec framework to perform an image reconstruction in a neural decoding task. The solution is based on an end-to-end encoder-decoder model under the guidance of Foreground-attention to enhance the perceptual quality of reconstructed images. The experimental results show visible improvements in the quality of reconstructed images without requiring additional fMRI training data. Also, the proposed method shows improvements in the structural similarity of image reconstruction.", "summary_of_the_review": "Although the paper tests all of their contributions, the benefits do not seem to be dramatic. I do not see proper evaluation metrics that are used in previous methods for the model evaluation and also not tested on other existing fMRI datasets. The experimental section is weak, and it is hard to reproduce the results, so the paper does not introduce a significantly better technique.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper present a model for reconstructing images from fMRI recordings, based on an encoder and decoder used in a loop. The reviewers were unanimous in their opinion that this paper is not ready for publication at this stage. They raised concerns ranging from the quality of the result and how to compare them to previous methods, to the justification behind different modeling choices. The authors were gracious in their responses to the reviewers. I do not recommend acceptance at this stage,
Decision: Reject