Hyperrealistic neural decoding: Reconstruction of face stimuli from fMRI measurements via the GAN latent space
Authors: Thirza Dado
Yağmur Güçlütürk
Luca Ambrogioni
Gabrielle Ras
Sander E. Bosch
Marcel van Gerven
Umut Güçlü
Venue:  (2021)
Keywords: Deep learning
Face perception
fMRI
Generative Adversarial Networks
Neural decoding

Abstract: We introduce a new framework for hyperrealistic reconstruction of perceived naturalistic stimuli from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring functional magnetic resonance imaging data as subjects perceived face images created by the generator network of a GAN. Subsequently, we used a decoding approach to predict the latent state of the GAN from brain data. Hence, latent representations for stimulus (re-)generation are obtained, leading to state-of-the-art image reconstructions. Altogether, we have developed a highly promising approach for decoding sensory perception from brain activity and systematically analyzing neural information processing in the human brain.

Under review as a conference paper at ICLR 2021
HYPERREALISTIC NEURAL DECODING: RECONSTRUCTION OF FACE STIMULI FROM FMRI MEASUREMENTS VIA THE GAN LATENT SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce new framework for hyperrealistic reconstruction of perceived natu- ralistic stimuli from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) the earliest step of our neural decoding pipeline by acquiring functional magnetic resonance imaging data as subjects perceived face images created by the generator network of GAN. Subsequently, we used decoding approach to predict the latent state the GAN from brain data. Hence, latent representations for stimulus (re- rre))ennerttion are obtained, leading state-of- the-art image reconstructions. Altogether, we have developed a highly promising approach for decoding sensory perception from brain activity and systematically analyzing neural information processing in the human brain.
INTRODUCTION
In recent years, the field of neural decoding has been gaining more and more traction as advanced computational methods became increasingly available for application neural data. This is very welcome development both neuroscience and neurotechnology since reading neural information will not only help understand and explain human brain function but also find applications in brain computer interfaces and neuroprostheties help people with disabilities.
Noural Ercoding
Neural Encoding
Nound Decodingg
Neuna Decoding
The mapping between sensory stimuli (left) and fMRI recordings (right). Neural encoding seeks to find transformation from stimulus to the observed brain response via latent representation (middle). Conversely, neural decoding seeks to find the information present in the observed brain responses by mapping from brain activity back to the original stimulus.
Figure 1:
Neural decoding can be conceptualized as the inverse problem of mapping brain responses back to sensory stimuli via latent space (20). Such a mapping can be idealized composite function of linear and nonlinear transformations (Figure TT. The linear transformation models the mapping from brain responses to the latent space. The latent space should effectively capture the defining properties of the underlying neural representations. The nonlinear transformation models the mapping from the latent space to sensory stimuli.
The systematic correspondences between latent represennations of discriminative convnets and neural representations of sensory cortices are well established (23 14 2 6). As such, exploiting these systematic correspondences in neural decoding of visual experience has pushed the state-of-the-ar forward (20). This includes linear reconstruction of perceived handwritten characters (15), neural decoding of perceived and imagined object categories (10), and reconstruction of natural images (17) and faces 99 (T). Yet, there still much room for improvement since sfate-of-the art results still fall short of providing photorealistic reconstructions.

---

Under review as a conference paper at ICLR 2021
Stimulus
Latent vector
Linear Decoding
PGGAN
fMRI
P512
PGGAN
21
2
Predicted
latent
2512
vector
Reconstruction
Figure 2: Schematic illustration of the HYPER framework. Face images are generated from randomly sampled latent features are then presented as visual stimuli during brain scanning. Next, linear decoding model learns the mapping from brain responses to the original latent representation, after which predicts latent features & for unseen brain responses. Ultimately, these predicted latent features are fed to the GAN for image reconstruction.
by a face-generating GAN, as denoted by the dotted box. These faces
At the same time, generative adversarial networks (GANs) have emerged as perhaps the most powerful generative models to date 11 12 that can potentially bring neural decoding to the next level. However, since the true latent representations of GANs are not readily available for preexisting neural data (unlike those of the aforementioned discriminative convnets), the adoption of GANs in neural decoding has been relatively slow (see (16) for an earlier attempt with GANs and (2 for a related attempt with VAEs).
In this study, we introduce a very powerful yet simple framework for HYperrealistic reconstruction of PERception (HYPER), which elegantly integrates GANs in neural decoding by combining the following components (Figure 2
GAN. used a pretrained GAN, which allows for the generation of meaningful data samples from randomly sampled latent vectors. This model is used both for generating the stimulus set and for the ultimate reconstruction of perceived stimuli. the current study, we used the progressive growing of GANs (PGGAN) model (11). which generates photorealistic faces that resemble celebrities.
ii fMRI. We made use of neural data with known latent representation, obtained by presenting the stimulus set produced using the above-mentioned generative model, and recording the brain responses to these stimuli. In the current study, collected fMRI recordings in response the images produced using the PGGAN. We created a dataset consisting of separate training and test set.
ii Decoding model. We used a decoding model, mapping the neural data to the latent space of the generative model. Using this model, then obtained latent vectors for the neural responses corresponding to the stimulus images in the test set. Feeding these latent vectors back into the generative model resulted the hyperrealistic reconstructions of perception.
METHODS
2.1 TRAINING ON SYNTHETIC IMAGES WITH KNOWN LATENT FEATURES
State-of-the art face reconstruction techniques use deep neural networks to encode vectors of latent features for the images presented during the fMRI experiment 9 (1). These feature vectors have been shown to have a linear relation with measured brain responses. However, this approach entails information loss since the target images need to be reconstructed from the linear prediction using an approximate inversion network such as a variational decoder, leading to a severe bottleneck to the maximum possible reconstruction quality.
2

---

Under review as a conference paper at ICLR 2021
In this paper, we avoid this sub-optimality by presenting the participants photorealistic synthetic images generated using PGGAN. This allows us to store the ground-trute latents corresponding to the generated images which can be perfectly reconstructed using the generative model after predicting them from brain data.
2.2
NEURAL DECODING
2.2.1
PREDICTING LATENT VECTORS FROM BRAIN DATA.
We adapted the deep generative network of PGGAN by adding a dense layer at the beginning to transform brain data into latent vectors. This layer is trained by minimizing the Euclidean distance between true and predicted latent representations (balchsize = 30, Ir = 0.00001, Adam optimization) with weight decay (alpha 0.01) to reduce complexity and multicollinearity of the model. The remainder of the generative network was kept fixed.
The first decoding model is trained with loss function that takes only the latent vectors into account. Yet, the ultimate goal is to reconstruct what our participants were seeing. Technically, this is achieved when all the layer activations up until the output image would be similar between the real and predicted latent vector. Therefore, we created five additional loss functions that include these layer activations to examine how these contribute further optimization of neural decoding. Importantly, we only took the centers of the activation maps to exclude surrounding background noise. In the end, we trained one model on the latent vectors alone, and five models on the latent vectors together with one PGGAN layer activation.
2.2.2 PREDICTING LAYER ACTIVATIONS FROM BRAIN DATA.
Earlier work has found correspondences between artificial neural networks and the brain (7). Based on this knowledge, we trained four decoding models predict PGGAN's layer activations from brain data to explore the correspondence between this deep generative network and the brain. Specifically, we used the following layer outputs of PGGAN: 4, 9, 14, and 19, to which we will refer to as layer activation 1, 2. 3, and 4, respectively, for the remainder of this manuscript. The loss function was the Euclidean distance between true and predicted layer activations (and not the latent vectors). The rest of training proceeded before. Next, we examined the contribution of cach voxel in a predefined mask on model performance using a combination a searchlight mapping approach and occlusion analysis. The searchlight approach takes cubic subset of 7 x 7 voxels, centered each voxel is 2 x 2 x mm3 this results in volumes of 14 x 14 x 14 mm 3 Neighboring voxels are only included when they are also in the mask. Ultimately, this searchlight is excluded from the brain data input to detect the effects of the center voxel on model performance.
voxel. As
2.3 DATASETS
2.3.1 VISUAL STIMULI
High-resolution face images (1024 x 1024 pixels) are generated by the generator network of a Progressive GAN (PGGAN) model (1) from randomly sampled latent vectors. Each generated face image is cropped and resized to 224 x 224 pixels. In total, 1050 unique faces are presented once for the training set, and 36 faces are repeated 14 times for the test set of which the average brain response is taken. This ensured that the training set covers large stimulus space to fit general face model, whereas the voxel responses from the test contain less noise and higher statistical power.
3

---

Under review as a conference paper at ICLR 2021
2.3.2 BRAIN RESPONSES
An IMRI dataset was collected, consisting of BOLD responses that correspond to the per- ceived face stimuli. The BOLD responses (TR 1.5 voxel size = 2 x 2 x 2 mm², whole- brain coverage) of two healthy subjects were measured (S1: 30-year old male; S2: 32-year old male) while they were fixating on a target (0.6 x 0.6 degrees) (19) superimposed
S1
the stimuli (15 x 15 degrees) to minimize involun- tary eye movements.
During preprocessing, the obtained brain vol- umes realigned to the first functional scan and the mean functional scan, respectively, after which the volumes normalized to MNI space. A general linear model is fit to deconvolve task-related neural activation with the canonical hemodynamic response function (HRF). Next, for each voxel, we computed its t-statistic and converted these t-scores to z-statistics to obtain brain map in terms of z per perceived stimulus. Ultimately, most-active 4096 voxels were selected from the training set to define voxel mask (Figure 3) Most of these mask voxels are located in the downstream brain regions. Voxel responses from the test set are not used to create the voxel mask to avoid double-dipping.
Voxel mask: 4096 most active voxels based highest z-statistics within the averaged z z-map from the training set responses, resulting in distributed network of activity.
Figure 3:
The experiment was approved by the local ethics committee (CMO Regio Arnhem-Njimegen) Subjects provided written informed consent in accordance with the Declaration of Helsinki. The fMRI dataset for both subjects and used models openly accessible via Github.
2.4 EVALUATION
Model performance is assessed in terms of three metries: latent similarity, feature similarity, and structural similarity. First, latent similarity the Euclidean similarity between predicted and true latent vectors. Second. feature similarity is the Euclidean similarity between feature extraction layer outputs (n 2048) of the ResNet50 model, pretrained for face recognition, which we feed stimuli and reconstructions. Lastly, structural similarity is used to measure the spatial interdependence between pixels of stimuli and reconstructions (22).
Next, based on the assumption that there exists a hyperplane in latent space for binary semantic attributes (e.g. male vs. female), (18) have iden- tified the decision boundaries for five semantic face attributes in PGGAN's 's latent space: gender, age, the presence of eyeglasses, smile, and pose, by training five independent linear support vec- tor machines (SVMs) (Figure 4 We used these decision boundaries to compute feature scores per image, by taking the dot product between latent representation and decision boundary (re- sulting in scalar). In this way. model perfor- mance with regard to specific visual features could be captured along continuous spectrum, and compared across images.
Figure 4: Semantic face editing of a stimulus im- age from the testing set (number 7) by varying the latent vector along a separation boundary.
4

---

Under review as a conference paper at ICLR 2021
B
SKmm
Lat. (1)
Lot, (1
Lat, L2
Lai, L3
Lat, LA
Lat. Sim
044
Feat. Sim
0.7
036
035
Struct. Sim
040
Modks and layor activatianss
C
Gender
Age
Eyeglasses
Pose
Smiil
C000 coet.
6m
Models trited latett and layer activations
Figure 5: Results of the five models that predict latent vectors from brain data. The highest per- formance is achieved by our first model which is trained on latent representations alone and no intermediate layer activations. A. Four testset stimuli (left) and their five corresponding model reconstructions. B. The average latent similarity, feature similarity, and structural similarity (Y axis) of the five models (X axis). C. Five graphs display the Pearson correlation coefficients (Y axis) between true and predicted semantic feature scores of the five models (X axis) for each visual feature. We found high correlations for gender, pose, and age, but significant correlation for the smile attribute.
RESULTS
Linear decoding of fMRI recordings using PGGAN's latent space has led to unprecedented stimulus reconstructions. The highest performance is achieved by the first model that trained on latent vectors alone (Figure 5).
Figure 6 presents all image reconstructions of this best brain decoding model together with the originally perceived stimuli. To keep the presentation concise, the first half of the images (1-18) are reconstructed from brain activations from Subject and the second half (19-36) from Subject 2. The interpolations visualize the distance between predicted and true latent representations that underlie the (re)generated faces. demonstrates which features are being retained or change. The bar graphs next to the perceived and reconstructed images show the scores of each image in terms of five semantic face attributes in PGGAN's latent space: gender, age, the presence of eyeglasses, smile, and pose. Looking at the similarities and differences in the graphs for perceived and reconstructed images is way to evaluate how well each semantic attribute is captured by our model. For most reconstructions, the two graphs match in terms of directionality. There are a few cases, however, demonstrating that there is still room for improvement, e.g. number 31, 34, and 35. Correlating the feature scores for stimuli and reconstructions resulted in significant (p 0.05: Student's t-test) results for gender, age, eyeglasses, and pose, but not for smile (Figure 5). We would like to point out that using feature scores quantifies model performance as continuous rather than binary, explaining the significant correlation for eyeglasses despite lack of reconstruction in number 1 and 8.
Next, we compared the performance of the HYPER framework to the state-of-the-ar VAE-GAN approach (1I and the traditional eigenface approach (3) which map the brain recordings onto different latent spaces. For a fair comparison, we used the same voxel mask to evaluate all the methods presented in this study without any optimization particular decoding approach. The
5

---

Under review as a conference paper at ICLR 2021
S1
Recenstr
Interp
Percett
Gecendtr
Interp
Percey
2.
11.
12.
5.
15.
7.
16.
17.
S2
Recumstr:
Interp
Penoric
Inter
Percric
19.
28.
20.
29.
21.
30.
20
24.
33.
25.
34.
26.
35.
Figure 6: Results of model O that is trained on only the latent vectors. Here, we display testing set samples 1-18 for Subject and 19-36 for Subject 2. Image reconstructions (left) versus perceived images (right). Interpolations visualize similarity regarding the underlying latent representations. Next to each reconstruction and perceived stimulus, a rotated bar graph displays the corresponding feature scores for gender, age, eyeglasses, pose, and smile.
VAE-GAN approach predicts 1024-dimensioms latent representations which are fed to the VAE's decoder network for stimulus reconstruction (128 x 128 pixels). The eigenface approach predicts first 512 principal components (or "eigenfaces') after which stimulus reconstruction (64 x 64 pixels) achieved by applying an inverse PCA transform. All quantitative and qualitative comparisons showed that the HYPER framework outperformed the baselines and had significantly above-chance latent and feature reconstruction performance (p x 0.001, permutation test), indicating the probability that random latent vector image would be more similar to the original stimulus (Table TD.

---

Under review as a conference paper at ICLR 2021
Table 1: Model performance of our method compared the state-of-the-art VAE-GAN (1 and the eigenface approach (3) is assessed in terms of the feature similarity (column 2) and structural similarity (column 3) between stimuli and reconstructions (mean + std error). The first column displays latent similarity which is only applicable to the HYPER method because the true and predicted latent vectors are known. Because of resolution differences, all images are resized 224 x 224 pixels and smoothed with a Gaussian filter (kernel size 3) for fair comparison. In addition, statistical significance of the HYPER method is evaluated against randomly generated latent vectors and their reconstructions.
Lai. sim.
Feat. sim.
Struct. sim.
81
HYPER
0,4521 + D.022
0,1745 0.0088
019663 0.0155
0.001 feat)
0.001:permm.com (ext)
perantee...eext
VAE-GAN
0.1416 0.0025
0.5598 0.0151
Eipenfaae
0.1319 0.0016
0.5877 0.0115
HYPER
0.4447 D8000
0,1715 0:0019
0,0128
G 0.00 perm perented))
(P 0:00 permmlleereeeeee bext)
( 0.001: permm.fow
VAE-GAN
0.1461 + 0.0022
0.5832 0.0141
Eigenface
0.1261 1: 0.0019
0.5616 0.0097
We also present arbitrarily chosen but representative reconstruction examples from the VAE-GAN and eigenface approach, again demonstrating that the HYPER framework resulted in markedly better reconstructions (Figure
HYPER
VAE-GAN
Eigenface
Stimolus
Subject
Subject
Midel
Subject
Subject
Model
Subject
Subject
Figure 7: Qualitative results of our approach compared 21 and the eigenface approach in reconstructing image 26, 28, and 36 (arbitrarily chosen). The model columns display the best possible results. For (2D. this displays reconstructions directly decoded from the 1024-dimensional latent representation of this method. For the eigenfaces approach, this shows reconstructions directly obtained from the 512 principal components.
Lastly, we looked for any correspondence between the generative network and the brain. For each voxel. the ocarchiight-occlusion analysis identified on which of the four layer activation preedcctnn models it had the largest effect in terms of Euclidean activation similarity. This layer activation was then assigned to that voxel, and mapped to the brain (Figure B). The majority of the voxels was found to correspond to layer activations closest to the latent vector (i.e., L1 and L.2). Other than that, there was no systematic relationship between voxels and layer activations. This is not surprising considering that the voxel mask covered mostly the downstream regions such as FFA which are already specialized for high level representations. Nevertheless, would be interesting to further investigate if GAN layers map onto the visual cortex similar to task-optimized models do when trained and tested on natural images.
7

---

Besides the large scientific potential, this research could also have societal impacts when enabling vari- ous applications in the field of neurotechnology (e.g. brain computer interfacing and neuroprosthetics) to help people with disabilities. While the current work focuses on decoding sensory perception, Under review as a conference paper at ICLR 2021
A
H
C
IM
53
ROGS
Figure 8: A. Layer activations that corresponded most with each voxel are mapped on the brain of two subjects respectively. In pink, borders are indicated between the primary visual cortex, the early visual cortex, the dorsal stream visual cortex, ventral stream visual cortex, and MTT+ complex and neighboring visual areas, as based on (4). B. Distribution of layers assigned to voxels across different regions of interest. C. Activation similarity of different layers.
4 DISCUSSION
We have decoded brain recordings during perception of face photographs using the presented HYPER method, leading to state-of-the-art stimulus reconstructions. The HYPER framework resulted in considerably better reconstructions than the two benchmark approaches. It is important to note that the reconstructions by the VAE-GAN approach appear to be of lower quality than those presented in the original study. A likely explanation for this result could be that the number of training images in our dataset was not sufficient to effectively train their model (8000 vs 1050) and the different voxel selection procedure.
Image reconstructions by HYPER appear to contain biases. That is, the model predicts primarily latent representations corresponding to young, cookenn-looking faces without eyeglasses because predictions tend to follow the image statistics of the (celebrity) training set. PGGAN's generator network is also known to suffer from this problem referred as "feature entanglement" where manipulating one particular feature in latent space affects other features as well (18). For example, editing a latent vector to make the generated face wear eyeglasses simultaneously makes the face look older because of such biases in the training data. Feature entanglement obstructs the generator to map unfamiliar latent elements to their respective visual features. is easy to foresee the complications for reconstructing images of existing faces.
modified version of PGGAN, called StyleGAN 12 13), is designed to overcome the feature entanglement problem. StyleGAN maps the entangled latent vector additional intermediate latent space - thereby reducing feature entanglement which then integrated into the generator network using adaptive instance normalization. This results in superior control over the semantic attributes in the reconstructed images and possibly the generator's competence to reconstruct unfamiliar features. Compared PGGAN, the generated face photographs by StyleGAN have improved considerably in quality and variation, of which the latter is likely to elevate current biases. Replacing the PGGAN with StyleGAN would therefore be logical next step for studies concerned with the neural decoding of faces.
extensions of our framework to imagery could make it preferred means for communication for locked-in in patients.
CONCLUSION
We have presented framework for HYperrealistic reconstruction PERception (HYPER) by neural decoding of brain responses via the GAN latent space, leading to unparalleled state-of-the-art stimulus reconstructions. Considering the speed of progress in the field of generative modeling, we believe that the HYPER framework that we have introduced in this study will likely result in more impressive reconstructions of perception and possibly even imagery in the near future, ultimately also allowing better understand mechanisms of human brain function.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018.
[2] Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS Comput Biol, 10(12):e100396) 2014.
[3] Alan S Cowen, Marvin M Chun, and Brice A Kuhl, Neural portraits of perception: reconstruct- ing face images from evoked brain activity. Neuroimage, 94:1 12-22, 2014.
[4] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):111-178 2016.
[5] Ian Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[6] Umut Güçlü, Jordy Thiclen, Michael Hanke, and Marcel Van Gerven. Brains beats. In Advances in Neural Information Processing Systems, pages 2101-2109, 2016.
[7] Umut Güşlü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 2015.
[8] Umut Güęlü and Marcel van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neurolmage, 145:329-3360 2017.
[9] Y. Güçlütürk, U. Güęlü, K. Seeliger, S. Bosch, van Lier, and M. A. van Gerven. Reconstruct- ing perceived faces from brain activations with deep adversarial neural decoding. Advances in Neural Information Processing Systems, pages 4246 4257, 2017.
[10] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):1-15, 2017.
[11] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 17100/0990.. 2017.
[12] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[13] Tero Karras, Samuli Laine, Mïia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8110-8119, 2020.
[14] Seyed-Mahdi Khaligh-Razs Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsuper- vised, models may explain it cortical representation. PLoS Comput Biol, 10(11), 2014.
[15] Sanne Schoenmakers, Markus Barth, Tom Heskes, and Marcel Van Gerven. Linear reconstruc- tion of perceived images from human brain activity. Neurolmage, 83:951-961,
[16] Katja Seeliger, Umut Güęlü, Luca Ambrogioni, Yagmur Güçlütürk, and Marcel AJ van Ger- ven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage, 181:775-785, 2018.
[17] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLoS Comput Biol. 15())ee1006633, 2019.
[18] Yujun Shen, Jinjin Gu, Xiaoou Tang. and Bolci Zhou. Interpreting the latent space of gans for semantic face editing. arXiv preprint arXiv: 1907.10786. 2019.

---

Under review as a conference paper at ICLR 2021
[19] Lore Thaler, Alexander C Schützz, Melvyn A Goodale, and Karl R Gegenfurtner. What is the best fixation target? the effect of target shape on stability of fixational eye movements. Vision Research, 76:31-42. 2013.
[20] Marcel AJ van Gerven, Katja Seeliger, Umut Güçlü, and Yagmur Güçlütürk. Current advances in neural decoding. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 379-394. Springer, 2019.
[21] Rufin VanRullen and Leila Reddy. Reconstructing faces from fmri patterns using deep generative neural networks. Communications biology, 2(1):193, 2019.
[22] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
[23] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-optinee hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 1 111(2))8619 8624, 2014.
10


Reviews:
[{"review_id": "up7nqqwF0o", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript entitled \"HYPERREALISTIC NEURAL DECODING :\nRECONSTRUCTION OF FACE STIMULI FROM MEASUREMENTS VIA THE GAN LATENT\nSPACE\" presents a novel percept reconstruction method based on\nfunctional neuroimaging. The approach consists in decoding brain\nresponses via the GAN latent space, and leads to state-of-the-art\nstimulus reconstructions.  More precisely, it is shown to outperform\nVAE-GAN and eigenfaces. Experiments focus on metrics well suited to\nface space analysis.\n\nA nice feature of the paper is that it rests on a relatively straightforward\nidea: simply learn to predict latent representations and then let the\nmagic of GAN-based face generation work. The main novelty is to use\ngenerated samples as fMRI space, leading to a better anchoring of the\ndecoder in latent space.  While the main concept of the paper is quite\ninteresting, the overall quality of the paper is not great, due to\nbasic statistical (sample size, simple validation) and conceptual\n(absolute lack of neuroscientific perspective) issues. To summarize,\nthe paper presents a clever engineering of the decoding problem,\nbut does not bring novel insights.\n\n\nMajor comments\n==============\n\n* My first concern is about statistics The test set is reduced to 36\n  samples, which is strikingly insufficient to obtain reliable\n  estimates. Indeed, the accuracy of the results will be with an\n  uncertainty of about 1/sqrt(36) = 1/6. Even worse, the study design\n  prevents cross-validation and is based on simple validation.  This\n  means that essentially nothing -beyond the fact that the model is\n  \"significantly better than chance\"- could be concluded from the\n  results. In particular, it is claimed that \"the HYPER framework\n  outperformed the baselines\" but is the difference significant ?\n  Experiments are carried out on 2 subjects only, which is below the\n  standards of the field. This means that the results may simply\n  represent overfit.\n\n* My second concern is that the model assumes that the brain is, so to\n  say, constantly viewing faces: it says nothing about whether the\n  subject is involved in face perception or not. While this sounds\n  like a minor detail, it illustrates how narrow the perspective taken\n  in the paper is: it is always possible to input brain activity to\n  whatever-generator, and produce nice \"hyperrealistic\"\n  hallucinations.  More deeply, this means that all this framework boils down to\n   an engineering trick that does not inform us about the\n  biological organization of perception.\n  \n* The embarrassing lack of neurophysiological relevance of the\n  framework appears clearly in the following problematic statement\n  \"These feature vectors have been shown to have a linear relation\n  with measured brain responses.\" this is problematic, because the\n  latent space used by Gans or VAEs is somewhat arbitrary /\n  unconstrained (even in terms of dimensionality). How comes\n  that it can fit neural structures ? Nothing in the paper backs up\n  the statement --- that is kept as vague as possible. Not\n  surprisingly, biological relevance or impact are deferred to future\n  work...\n\n* The bibliography of the document is biased toward a certain\n  lab, and thus unfair with respect to concurrent efforts, e.g. \n\nR. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, and M. Irani,\n“From voxels to pixels and back: Self-supervision in natural-image\nreconstruction from fMRI,” in Advances in Neural Information\nProcessing Systems, 2019.\n\nReconstructing faces from fMRI patterns using deep generative neural networks\nR VanRullen, L Reddy. Communications biology 2 (1), 193\n\nTransfer learning of deep neural network representations for fMRI decoding\nM Svanera, M Savardi, S Benini, A Signoroni, G Raz, T Hendler, L Muckli, ...\nJournal of neuroscience methods 328, 108319\n\nIn the discussion section at least, it seems necessary to report about other efforts on handwritten digits decoding, e.g.\n\nReconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception\nM Senden, TC Emmerling, R Van Hoof, MA Frost, R Goebel\nBrain Structure and Function 224 (3), 1167-1183\n\n  \nMinor comments\n==============\n\n• Method presentation is not extremely clear: from the methods section\n  I thought that a searchlight approach was used, which begs the\n  question of how a global decoding performance is obtained. My\n  understanding is that both searchlight- and whole brain analyses are\n  carried out, but this is not clear.\n  \n• The authors should indicate which tool is used to perform their\n  analyses (data processing; visualization etc.)\n\n• \"We found high correlations for gender, pose, and age, but no\n  significant correlation for the smile attribute.\" is weird to me\n  because pose has the lowest correlations. Maybe I am missing\n  something here.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "i5hriLVTKNE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work concerns reconstruction of visual stimulus from brain state as measured by fmri.\n\nThis is a quite well-explored field with an already impressive state of the art based on deep network methods. State of the art is well described in the paper and is represented e.g. by architectures, similar to the pipeline developed for the present study.\nThe main motivation of the present paper is to invoke a more complex image reconstruction network (GAN based).\nThe improved visual quality is quantified in several metrics (including the ability to reconstruct / represent face relevant features).\n\nReservations: The novelty is limited compared to the existing baselines mentioned in the submission.  \n While visual fidelity / quality is a basic important dimension to \"brain read out\", the quality presented for the competing baseline seems lesser that in the original baseline paper(s), hence questioning these papers. However, I can not rule out that the present study doesn't get the baseline correct (one potential reason being data size as mention in the submission).\n \nConclusion: The novelty is limited and the author miss the opportunity to ask novel research questions. The comparison with a baseline is hampered by lower visual performance of the present instantiation than found in the original paper. \nIn a sense the author have missed the opportunity to ask novel research questions in this interesting domain. It is unclear (to me) if the improved visual quality is simple due to the improved performance of the GAN or the ability to read out more relevant features from the fmri.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gp0g02uMlZA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript entitled “Hyperrealistic neural decoding: Linear reconstruction of face stimuli from fMRI measurements via the GAN latent space” utilizes a GAN-based network structure for generating faces that are presented to the subjects during fMRI acquisition. The acquired fMRI signals are then used to predict latent vector in GAN, where the predicted vectors (rather than the original, trained vectors) are used to generate “fMRI-derived” faces in turn. This work proposed a novel perspective to the field of cognitive neuroscience and human brain mapping in functional neuroimaging studies. It is an indirect yet highly effective way for modeling brain functional signals, and representing how the visual encoding-decoding actually works in human brain. Similar schemes of investigation can be easily developed based on the proposed work (e.g. analyzing correspondence between EMG and fMRI during motor tasks), which will bring significant value to the community.\n\nThis work has great potentials, yet several interesting and important points of investigation were skipped in the study design, which lowered the scientific value of it and the validity of the model. Specifically,\n1)\tThe framework needs to include more semantic information into face generation and analysis: currently only simple features including age, gender, eyeglasses, smile, and pose are considered. Even for these features, evaluation of the predicted latent features and corresponding reconstructed face images w.r.t. their semantic information, as covered in Fig. 5, is conflicting, insufficient, and difficult to interpret: Firstly Fig. 5(c) implies low correlation between the true/predicted semantic scores for “Pose”, but in the caption and text it is stated that “but no significant correlation for the smile attribute”. Secondly, group-wise Pearson correlation score for the semantic features can only support the accuracy of the generated latent features, which has already been illustrated in Fig. 5(b) and Table 1. The reviewer suggests that individual semantic response (in the form of reconstructed faces) to the face stimulus might be more interesting to investigate. Finally, it is counter-intuitive to find that the first (trained on latent representations alone) can achieve the best performance even for semantic feature prediction, as image semantics such as pose and glasses are well-represented on the image itself.\n2)\tAnalysis of brain activation map based on the learned model, as shown in in Fig. 8, is difficult to read and interpret. ROI1~5 in Fig. 8 (b) cannot find the corresponding definitions for their names (is ROI1 visual cortex?). Implication of Fig. 8 (c) is difficult to understand and not provided in the text. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "a4bOFmQ8WM4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to reconstruct images of faces from fMRI measurements, using GANs. The authors collected a new dataset, showing static faces generated by a GAN model to human subjects, and recording their brain BOLD responses with MRI. Then, they learned a model to reconstruct the stimulus based on the brain responses. The authors demonstrate the high quality of the reconstructed faces, comparing with other recent methods.\n\nMain limitations:\n\n1. The reconstructed faces have a higher quality than previous work. However, the method is not conceptually different from e.g. [21], and the authors mostly used a better generative model, i.e. a better prior. In other words, since the generative model used (PGGAN) creates high-quality faces, they cannot reconstruct anything but high-quality faces. The quality of the faces is thus not a characteristic of the proposed method, but of the GAN model used. The authors should discuss this limitation in their paper.\n\n2. More importantly, the authors generated all stimuli using the same GAN model as used in reconstruction. The space of faces is much larger than the space of faces generated by this particular GAN. This choice of stimulus makes the prior perfect, and other methods cannot compete fairly. To the extreme, one could use a GAN that always generates the same face, and decoding with this same GAN would then be perfect. This design choice makes any comparison with other methods unfair and thus impossible. To alleviate this major limitation, the authors could use a different GAN for reconstruction, specifically a GAN that was not trained on the same celebA dataset. However, it is not clear what training loss would be best in this case.\n\nFor these important reasons, I would rather recommend this paper for rejection.\n\nOther proposed improvements:\n\n3. In figure 6, I fail to understand how \"interpolations visualize similarity regarding the underlying latent representations\". Indeed, one can interpolate between any couple of latent representations, leading to a smooth interpolation between the faces. However, it does not give any clear insight about the similarity of latent representations. In fact, a lot of couples of faces seem rather far perceptually, such as 3, 5, 7, 12, 13, 16, 17, 21, 25, 27, 29, 31, 34, 35. If anything, the interpolations make the comparison more difficult. I would suggest removing them.\n\n3. For reproducibility, the authors should mention how the latent space was sampled during the creation of the stimulus.\n\n5. \"elegantly\" is not adding much to the paper.\n\n6. \"searchlight mapping approach and occlusion analysis\" is not described in detail, nor linked to academic references.\n\n7. In Figure 4, rows are not labeled.\n\n8. In Figure 8, the legend should be ordered with the \"smile\" label at the top and the \"gender\" label at the bottom.\n\n9. In the discussion, the mention of biases due to the particular distribution of PGGAN is a plus, though the authors could discuss more in detail the potential social impact of having such biases.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XqINH1hRp3-", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Short summary\n---------------------\nThe authors propose a framework combining a GAN and linear encoder and decoders to reconstruct perceived face stimuli from fMRI data. They compare their framework to two baselines in the field and display a higher similarity (in different spaces) between the reconstructed images of their method and the stimuli, compared to the baselines.\n\nStrengths\n--------------\nThe framework is simple and allows for using different models to generate the stimuli, encode brain activity, decode brain activity and decode the stimuli. Appropriate baselines are selected and the authors quantify the similarity between generated and reconstructed images in different spaces.\n\nWeaknesses\n-------------------\nI am uncertain of the impact of the proposed approach, as it does not propose techniques to investigate brain functioning, nor does it provide a means of communication with disabled patients (as claimed by the authors). I see Ethical concerns with this type of model, which, in my opinion, are not counterbalanced by its usefulness.\n\nNovelty\n-------------\nThe approach combines well established models and techniques into a novel framework. While I found some creativity in the setup, the novelty is overall low and I am not convinced that some of the techniques used are not acting as bottlenecks (see detailed comments).\n\nClarity\n----------- \nThe paper is relatively clear. I appreciated the presence of multiple figures and of various examples. I believe that the methods could be clearer (see detailed comments).\n\nSignificance\n----------------\nTo me this was a major concern with this work as I found some claims bold and not substantiated by the experimental setup or the results. For instance, the authors claim to decode naturalistic stimuli. However, they can decode GAN generated images, which is substantially different, especially given the approach to average the fMRI signals over 14 repetitions in the test set to increase SNR.\n\nRigor\n--------\nOverall, I found that the work was relatively well performed, although not excellent. I wished that the comparison with the VAE approach was fair and would suggest that the authors work towards achieving SOTA in their baselines for a fair comparison.\n\nDetailed comments\n---------------------------\n- Faces generated by a GAN cannot be deemed naturalistic\n- Isn’t the setup circular? How would that model help understand face processing in naturalistic settings?\n- Ethics concerns: how is the application helping people with disabilities or understanding brain function? The authors mention this as a potential communication means for locked-in patients. However, (1) the results are not strong enough to suggest a potential communication tool, (2) the reliance on fMRI signals makes this impractical and expensive. There are no conclusions regarding brain function that this technique brings, especially given the voxel selection based on a linear regression model.\n- Novelty and technical sophistication is rather low: combining existing techniques in a novel system.\n- A better test would have been to reconstruct stimuli that were not generated by the GAN\n- “Importantly, we only took the centers of the activation maps to exclude surrounding background noise”. I believe the authors refer to “activation maps” as the fMRI z-score maps. This can however be confusing for an ML reader. Please clarify the language.\n- It is unclear to me what the goal of the “five additional loss functions” is, or how they are formulated.\n- why is the test set not sampled the same way as the training set? While taking the average of 14 repetitions increases signal-to-noise ratio, this setting further departs from any naturalistic decoding.\n- An fMRI study with 2 subjects is not representative, given inter-subject variability. This is further compounded by the low number of test images (36 per subject after averaging).\n- There is a clear imbalance in the generated stimuli in terms of age, race or whether they wear glasses. This limits the impact of the scores for the 5 different attributes. It is also a reason why neuroscience experimental stimuli are thoroughly controlled for. While this is touched upon in the discussion, this could be an indication that the proposed approach would fail on naturalistic stimuli.\n- Some reconstructed stimuli are highly similar, despite different generated images (e.g. 23 and 24). What could explain this phenomenon? Could there be some type of mode collapse in the reconstruction? It would be interesting to compute the pair-wise similarity between reconstructed images compared to the pairwise similarity of the generated images (in the different spaces mentioned, i.e. latent, feature, attributes).\n- Isn’t the linear regression model a bottleneck here? Why not use the “raw” BOLD signal instead of z-scores? This reflects an assumption that the encoding between stimulus and activation map is linear. Couldn’t there be a non-linear mapping between stimulus perception and the latent space? Overall, the proposed framework relies on established techniques without questioning or reflecting on their assumption.\n\nMinor\n-------\n- Given the linear model used, why limit to 4096 voxels? This seems like an arbitrary number. Is it related to the z-scores or a specific p-value threshold on the activation map?\n- The relationship between the trained models and the ResNet mentioned in section 2.4 is unclear. Is it used for evaluation as another way to estimate the similarity between the reconstruction and the GAN generated images? Is there a justification/reference for this technique?\n- Figure 5C, could the authors use the same y-scale? The legend mentions “We found high correlations for gender, pose, and age, but no significant correlation for the smile attribute.” Pose however had the lowest correlation values. How about eyeglasses?\n- “[…] permutation test), indicating the probability that a random latent vector or image would be more similar to the original stimulus”. This sentence is unclear to me: is the permutation test assessing whether HYPER has significantly higher similarity between the reconstructed image and the generated image than if using a random latent vector? Please provide more details on the hypothesis tested by the permutation test in each space, as well as how these tests compare the different techniques. Couldn’t the permutation tests be applied to the baselines techniques as well?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The approach proposed here have raised major concerns from multiple reviewers especially concerning the novelty and the experimental validation procedure.
Decision: Reject