Foreground-attention in neural decoding: Guiding Loop-Enc-Dec to reconstruct visual stimulus images from fMRI

1 INTRODUCTION . In recent years , reconstructing visual stimulus images from fMRI has gradually gained attention , which provides the possibility of ” mind reading ” in the future ( Fig . 1 ) . Existing work has shown that there is a certain mapping relationship between visual stimuli and brain activity ( Poldrack & Farah , 2015 ) , which provides us with the possibility and basis for reconstructing visual stimuli from fMRI data . The fMRI data which is collected from the human brain records the variations in blood oxygen level dependent ( BOLD ) and reflects the activity of nerves in human brain . Through the analysis of fMRI data , the correlation between brain activity and visual tasks can be explored , helping us to understand human visual mechanism better . Prior studies . In recent years , there has been a lot of research in this field . The main methods can be roughly divided into the following two categories : Linear regression models , which encode fMRI data into image pixel values , and finally achieve image reconstruction . Deep learning models , such as DCNN and GAN model frameworks . For the first type of methods , Yoichi Miyawaki et al . applied a multi-scale linear weighting model to predict the pixel value of each image and obtain the reconstruction results of the black and white simple images ( Miyawaki et al. , 2008 ) . Fujiwara et al . proposed to use Bayesian canonical correlation analysis ( BCCA ) to build a reconstruction model , which extracts image information from measured data and reconstruct images ( Fujiwara et al. , 2013 ) . Marcel AJ van Gerve et al . adopted a hierarchical generative model composed of Boltzmann machines with restricted conditions , which explores the reconstruction of feature hierarchies based on learning ( Van Gerven et al. , 2010 ) . Yu et al . advanced a correlation network framework that could be flexibly combined with diverse pattern representation models . They revealed the effective connectivity of human brain and reconstructed the visual stimulus images ( Yu & Zheng , 2017 ; Yu et al. , 2018 ) . For the second type of methods , Guohua Shen et al . proposed an end-to-end deep convolution model to reconstruct visual stimulus images directly by using fMRI data , and at the same time they used GAN framework to optimize the training of the model ( Shen et al. , 2019a ) . In response to the current problem of the lack of data sets used to reconstruct stimulus images from fMRI data , Roman Beliy et al . suggest an approach for self-supervised training on unlabeled fMRI data ( Beliy et al. , 2019 ) . Yunfeng Lin et al . proposed a model called DCNN-GAN by combining a reconstruction network and a GAN module . They utilized the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images ( Lin et al. , 2019 ) . Tao Fang et al . defined a Shape-Semantic GAN , considering the functional differences of different visual cortex areas , and reconstruct visual stimulus images under the guidance of shape and semantic information ( Fang et al. , 2020 ) . The ideas of DCNN and GAN have also been applied to various other model frameworks to achieve the reconstruction of visual stimulus images ( Seeliger et al. , 2018 ; VanRullen & Reddy , 2019 ; Zhang et al. , 2020 ; Qiao et al. , 2020 ; Mozafari et al. , 2020 ) . Our contributions . Because of the complex neural information in the visual cortex , how to extract valuable information from high-dimensional fMRI data has been a challenging problem . Inspired by human visual attention mechanism , we introduce visual attention to this work for the first time . We first decode the human visual attention from a small amount of fMRI data successfully , and then use visual attention to guide the work of reconstructing visual stimulus images from fMRI . Existing studies have shown that the human visual system is more inclined to focus on prominent objects , and the neural representation of these prominent objects in the brain is more obvious ( Ungerleider & G , 2000 ; Braun et al. , 2001 ) . The distribution of attention leads to an information bottleneck , only the most prominent objects are allowed to appear in the inferior temporal cortex , especially the ventral visual stream that encodes the identity of the object . The visual attention mechanism is crucial for simulating the neural response of the higher visual system ( Poggio & Anselmi , 2016 ; Khosla et al. , 2020 ) . Based on the above conclusions , we propose Foreground-attention ( F-attention ) . We abstract the distribution of human visual attention as the most prominent foreground object distribution area in the images . Through the constructed F-attention decoder , the fMRI corresponding to the natural stimulus images can be decoded into visual attention distribution ( Fig . 2a ) . Because human brain is strongly wound into sulci and gyri , some spatially adjacent voxels are not directly connected in the human brain ( Tallinen et al. , 2014 ) , the correlation between voxels is not only determined by location . Therefore , we need to consider global information when decoding fMRI , so we first introduce the self-attention module ( Vaswani et al. , 2017 ; Wang et al. , 2018 ; Zhang et al. , 2019 ) into the fMRI decoding process , the self-attention module can capture global information of the input data and expand the receptive field ( Fig . 3c ) . At the same time , we propose a new encoding and decoding framework called Loop-Enc-Dec ( Fig . 2b , c ) . The first step we pre-train the fMRI encoder on the data set to realize the encoding process from images to fMRI . The second step we train the end-to-end encoder-decoder model under the guidance of F-attention , and input the images decoded by the decoder into the fMRI encoder for encoding to add the re-encoding constraint in the loss function , that is called “ Loop-Enc-Dec ” . In the training process , natural pictures without fMRI data are also added for end-to-end self-enc-dec training , which increases the stability and generalization of the model . After experimental evaluation , the performance of Loop-Enc-Dec under the guidance of F-attention is better than that of mainstream methods . The main contributions of this article are as follows : • As far as we know , we first introduced the visual attention mechanism to the work of reconstructing visual stimulus images from fMRI . According to neuroscience research , we abstract visual attention as the attention distribution of foreground objects ( F-attention ) , and build a visual attention decoder to decode the human visual attention distribution from fMRI successfully . • To our best knowledge , we first introduced the self-attention module into the work of fMRI decoding to capture global information of fMRI data . • We propose a new enc-dec framework called Loop-Enc-Dec , guided by F-attention , which adds more loss constraints to the training of the image reconstruction model . Evaluating the quality of reconstructed images , our method outperforms previous works . 2 METHODS . In this section , we first introduce the data set used in the experiments , then we introduce the details of F-attention and Loop-Enc-Dec framework proposed in this article , as well as the step-by-step training strategy for the model . Finally , our evaluation method is given . 2.1 FMRI DATA SET . We use a publicly available benchmark data sets ( Horikawa & Kamitani , 2017 ) , which is widely used in the work of visual stimulus image reconstruction . In the image presentation experiment , fMRI signals were measured while subjects viewed a sequence of object images from ImageNet ( Deng et al. , 2009 ) . The image presentation experiment consisted of two sessions : the training image session and the testing image session . In the training image session , 1200 images from 150 object categories ( 8 images from each category ) were each presented once . In the testing image session , 50 images from 50 object categories ( one image from each category ) were each presented 35 times . The data collectors performed their analysis for each combination of feature types/layers and brain regions of interest ( ROIs ; V1–V4 , the lateral occipital complex ( LOC ) , fusiform face area ( FFA ) , parahippocampal place area ( PPA ) , lower visual cortex ( LVC ; V1–V3 ) , higher visual cortex ( HVC ; covering regions around LOC , FFA and PPA ) and the entire visual cortex ( VC ; covering all of the visual subareas listed above ) , the voxel size is 3×3×3 mm3 ) ( Horikawa & Kamitani , 2017 ) . 2.2 PROPOSED MODEL . Summarizing the proposed model . We first trian the F-attention decoder on fMRI data ( Fig . 2a ) , then pre-train the image encoder ( Fig . 2b ) , next we fix the weight of the encoder , train the decoder under the guidance of F-attention ( Fig . 2c ) . When testing , we first use fMRI as the input of F-attention decoder to decode the distribution of F-attention , then input the fMRI into the image decoder to reconstruct the visual stimulus images guided by corresponding F-attention . 2.2.1 F-ATTENTION . In order to get more useful information from fMRI data to guide the work of visual stimulus image reconstruction , we introduced the human visual attention mechanism into the work . We abstracted the human visual attention as foreground attention ( F-attention ) based on neuroscience research ( Poggio & Anselmi , 2016 ; Khosla et al. , 2020 ) . By constructing a F-attention decoder , we decode human visual attention distribution from fMRI data . Visual attention labeling . For the purpose of obtaining the human visual attention distribution of the training set , we first label the train data ( Fig . 3b ) . According to the definition of F-attention , we first extract the prominent objects in the natural images , and then perform the binary value on the images . Through the transformation operation , the main distribution areas of F-attention are obtained . Then we assigned attention weights to different regions in the images ( the bluer the color , the higher the attention weights , in the train set , the weights of the foreground object areas are 1 , the background areas are 0 ) , the final F-attention maps are obtained as the human visual attention labels of the training set . F-attention decoder . Then we construct F-attention-decoder to realize the decoding process from fMRI data to human visual attention ( Fig . 3c ) . First , the preprocessed fMRI data is integrated into a vector , then the vector is sequentially connected to a fully connection module , convolution and upsampling modules , a self-attention module , convolution and upsampling modules , and we finally get the F-attention map . The self-attention module calculation formula is as follows , dk is 1 : Self −Attention ( Q , K , V ) = softmax ( QK T √ dk ) V ( 1 ) In the training process , we use the mean square error loss ( MSE ) to constrain . After 400 rounds of iterations , the error decreases and stabilizes . The loss function is as follows : LossF−attention ( FAi , FA ∗ i ) ∝ ∑ ( FAi − FA∗i ) 2 ( 2 ) Where FAi represents the visual attention distribution corresponding to the images in the training set , and FA∗i stands for the visual attention distribution decoded by the F-attention decoder .

Summary: The authors proposed a model to decoding the fMRI signal from the human visual cortex by introducing the Foreground-attention. They also proposed a enc-dec training strategy called Loop-Enc-Dec, which is guided by the F-attention, to successfully reconstruct the visual images from the fMRI data. A higher score based on pairwise SSIM is achieved compared to previous works.