Foreground-attention in neural decoding: Guiding Loop-Enc-Dec to reconstruct visual stimulus images from fMRI

1 INTRODUCTION . In recent years , reconstructing visual stimulus images from fMRI has gradually gained attention , which provides the possibility of ” mind reading ” in the future ( Fig . 1 ) . Existing work has shown that there is a certain mapping relationship between visual stimuli and brain activity ( Poldrack & Farah , 2015 ) , which provides us with the possibility and basis for reconstructing visual stimuli from fMRI data . The fMRI data which is collected from the human brain records the variations in blood oxygen level dependent ( BOLD ) and reflects the activity of nerves in human brain . Through the analysis of fMRI data , the correlation between brain activity and visual tasks can be explored , helping us to understand human visual mechanism better . Prior studies . In recent years , there has been a lot of research in this field . The main methods can be roughly divided into the following two categories : Linear regression models , which encode fMRI data into image pixel values , and finally achieve image reconstruction . Deep learning models , such as DCNN and GAN model frameworks . For the first type of methods , Yoichi Miyawaki et al . applied a multi-scale linear weighting model to predict the pixel value of each image and obtain the reconstruction results of the black and white simple images ( Miyawaki et al. , 2008 ) . Fujiwara et al . proposed to use Bayesian canonical correlation analysis ( BCCA ) to build a reconstruction model , which extracts image information from measured data and reconstruct images ( Fujiwara et al. , 2013 ) . Marcel AJ van Gerve et al . adopted a hierarchical generative model composed of Boltzmann machines with restricted conditions , which explores the reconstruction of feature hierarchies based on learning ( Van Gerven et al. , 2010 ) . Yu et al . advanced a correlation network framework that could be flexibly combined with diverse pattern representation models . They revealed the effective connectivity of human brain and reconstructed the visual stimulus images ( Yu & Zheng , 2017 ; Yu et al. , 2018 ) . For the second type of methods , Guohua Shen et al . proposed an end-to-end deep convolution model to reconstruct visual stimulus images directly by using fMRI data , and at the same time they used GAN framework to optimize the training of the model ( Shen et al. , 2019a ) . In response to the current problem of the lack of data sets used to reconstruct stimulus images from fMRI data , Roman Beliy et al . suggest an approach for self-supervised training on unlabeled fMRI data ( Beliy et al. , 2019 ) . Yunfeng Lin et al . proposed a model called DCNN-GAN by combining a reconstruction network and a GAN module . They utilized the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images ( Lin et al. , 2019 ) . Tao Fang et al . defined a Shape-Semantic GAN , considering the functional differences of different visual cortex areas , and reconstruct visual stimulus images under the guidance of shape and semantic information ( Fang et al. , 2020 ) . The ideas of DCNN and GAN have also been applied to various other model frameworks to achieve the reconstruction of visual stimulus images ( Seeliger et al. , 2018 ; VanRullen & Reddy , 2019 ; Zhang et al. , 2020 ; Qiao et al. , 2020 ; Mozafari et al. , 2020 ) . Our contributions . Because of the complex neural information in the visual cortex , how to extract valuable information from high-dimensional fMRI data has been a challenging problem . Inspired by human visual attention mechanism , we introduce visual attention to this work for the first time . We first decode the human visual attention from a small amount of fMRI data successfully , and then use visual attention to guide the work of reconstructing visual stimulus images from fMRI . Existing studies have shown that the human visual system is more inclined to focus on prominent objects , and the neural representation of these prominent objects in the brain is more obvious ( Ungerleider & G , 2000 ; Braun et al. , 2001 ) . The distribution of attention leads to an information bottleneck , only the most prominent objects are allowed to appear in the inferior temporal cortex , especially the ventral visual stream that encodes the identity of the object . The visual attention mechanism is crucial for simulating the neural response of the higher visual system ( Poggio & Anselmi , 2016 ; Khosla et al. , 2020 ) . Based on the above conclusions , we propose Foreground-attention ( F-attention ) . We abstract the distribution of human visual attention as the most prominent foreground object distribution area in the images . Through the constructed F-attention decoder , the fMRI corresponding to the natural stimulus images can be decoded into visual attention distribution ( Fig . 2a ) . Because human brain is strongly wound into sulci and gyri , some spatially adjacent voxels are not directly connected in the human brain ( Tallinen et al. , 2014 ) , the correlation between voxels is not only determined by location . Therefore , we need to consider global information when decoding fMRI , so we first introduce the self-attention module ( Vaswani et al. , 2017 ; Wang et al. , 2018 ; Zhang et al. , 2019 ) into the fMRI decoding process , the self-attention module can capture global information of the input data and expand the receptive field ( Fig . 3c ) . At the same time , we propose a new encoding and decoding framework called Loop-Enc-Dec ( Fig . 2b , c ) . The first step we pre-train the fMRI encoder on the data set to realize the encoding process from images to fMRI . The second step we train the end-to-end encoder-decoder model under the guidance of F-attention , and input the images decoded by the decoder into the fMRI encoder for encoding to add the re-encoding constraint in the loss function , that is called “ Loop-Enc-Dec ” . In the training process , natural pictures without fMRI data are also added for end-to-end self-enc-dec training , which increases the stability and generalization of the model . After experimental evaluation , the performance of Loop-Enc-Dec under the guidance of F-attention is better than that of mainstream methods . The main contributions of this article are as follows : • As far as we know , we first introduced the visual attention mechanism to the work of reconstructing visual stimulus images from fMRI . According to neuroscience research , we abstract visual attention as the attention distribution of foreground objects ( F-attention ) , and build a visual attention decoder to decode the human visual attention distribution from fMRI successfully . • To our best knowledge , we first introduced the self-attention module into the work of fMRI decoding to capture global information of fMRI data . • We propose a new enc-dec framework called Loop-Enc-Dec , guided by F-attention , which adds more loss constraints to the training of the image reconstruction model . Evaluating the quality of reconstructed images , our method outperforms previous works . 2 METHODS . In this section , we first introduce the data set used in the experiments , then we introduce the details of F-attention and Loop-Enc-Dec framework proposed in this article , as well as the step-by-step training strategy for the model . Finally , our evaluation method is given . 2.1 FMRI DATA SET . We use a publicly available benchmark data sets ( Horikawa & Kamitani , 2017 ) , which is widely used in the work of visual stimulus image reconstruction . In the image presentation experiment , fMRI signals were measured while subjects viewed a sequence of object images from ImageNet ( Deng et al. , 2009 ) . The image presentation experiment consisted of two sessions : the training image session and the testing image session . In the training image session , 1200 images from 150 object categories ( 8 images from each category ) were each presented once . In the testing image session , 50 images from 50 object categories ( one image from each category ) were each presented 35 times . The data collectors performed their analysis for each combination of feature types/layers and brain regions of interest ( ROIs ; V1–V4 , the lateral occipital complex ( LOC ) , fusiform face area ( FFA ) , parahippocampal place area ( PPA ) , lower visual cortex ( LVC ; V1–V3 ) , higher visual cortex ( HVC ; covering regions around LOC , FFA and PPA ) and the entire visual cortex ( VC ; covering all of the visual subareas listed above ) , the voxel size is 3×3×3 mm3 ) ( Horikawa & Kamitani , 2017 ) . 2.2 PROPOSED MODEL . Summarizing the proposed model . We first trian the F-attention decoder on fMRI data ( Fig . 2a ) , then pre-train the image encoder ( Fig . 2b ) , next we fix the weight of the encoder , train the decoder under the guidance of F-attention ( Fig . 2c ) . When testing , we first use fMRI as the input of F-attention decoder to decode the distribution of F-attention , then input the fMRI into the image decoder to reconstruct the visual stimulus images guided by corresponding F-attention . 2.2.1 F-ATTENTION . In order to get more useful information from fMRI data to guide the work of visual stimulus image reconstruction , we introduced the human visual attention mechanism into the work . We abstracted the human visual attention as foreground attention ( F-attention ) based on neuroscience research ( Poggio & Anselmi , 2016 ; Khosla et al. , 2020 ) . By constructing a F-attention decoder , we decode human visual attention distribution from fMRI data . Visual attention labeling . For the purpose of obtaining the human visual attention distribution of the training set , we first label the train data ( Fig . 3b ) . According to the definition of F-attention , we first extract the prominent objects in the natural images , and then perform the binary value on the images . Through the transformation operation , the main distribution areas of F-attention are obtained . Then we assigned attention weights to different regions in the images ( the bluer the color , the higher the attention weights , in the train set , the weights of the foreground object areas are 1 , the background areas are 0 ) , the final F-attention maps are obtained as the human visual attention labels of the training set . F-attention decoder . Then we construct F-attention-decoder to realize the decoding process from fMRI data to human visual attention ( Fig . 3c ) . First , the preprocessed fMRI data is integrated into a vector , then the vector is sequentially connected to a fully connection module , convolution and upsampling modules , a self-attention module , convolution and upsampling modules , and we finally get the F-attention map . The self-attention module calculation formula is as follows , dk is 1 : Self −Attention ( Q , K , V ) = softmax ( QK T √ dk ) V ( 1 ) In the training process , we use the mean square error loss ( MSE ) to constrain . After 400 rounds of iterations , the error decreases and stabilizes . The loss function is as follows : LossF−attention ( FAi , FA ∗ i ) ∝ ∑ ( FAi − FA∗i ) 2 ( 2 ) Where FAi represents the visual attention distribution corresponding to the images in the training set , and FA∗i stands for the visual attention distribution decoded by the F-attention decoder .

Summary: The paper proposed a Loop-Enc-Dec framework to perform an image reconstruction in a neural decoding task. The solution is based on an end-to-end encoder-decoder model under the guidance of Foreground-attention to enhance the perceptual quality of reconstructed images. The experimental results show visible improvements in the quality of reconstructed images without requiring additional fMRI training data. Also, the proposed method shows improvements in the structural similarity of image reconstruction.