Brain2GAN; Reconstructing perceived faces from the primate brain via StyleGAN3
Authors: Thirza Dado
Paolo Papale
Antonio Lozano
Lynn Le
Feng Wang
Marcel van Gerven
Pieter R. Roelfsema
Yağmur Güçlütürk
Umut Güçlü
Venue: Submitted to ICLR 2023 (2023)
Keywords: Face reconstruction
generative adversarial networks
neural decoding

Abstract: Neural coding characterizes the relationship between stimuli and their corresponding neural responses. The usage of synthesized yet photorealistic reality by generative adversarial networks (GANs) allows for superior control over these data: the underlying feature representations that account for the semantics in synthesized data are known a priori and their relationship is perfect rather than approximated post-hoc by feature extraction models. We exploit this property in neural decoding of multi-unit activity responses that we recorded from the primate brain upon presentation with synthesized face images in a passive fixation experiment. The face reconstructions we acquired from brain activity were astonishingly similar to the originally perceived face stimuli. This provides strong evidence that the neural face manifold and the disentangled w-latent space conditioned on StyleGAN3 (rather than the z-latent space of arbitrary GANs or other feature representations we encountered so far) share how they represent the high-level semantics of the high-dimensional space of faces.

Under review as a conference paper at ICLR 2023
BRAIN2GAN; RECONSTRUCTING PERCEIVED FACES FROM THE PRIMATE BRAIN VIA STYLEGAN3
Anonymous authors Paper under double-blind review
ABSTRACT
Neural coding characterizes the relationship between stimuli and their correspond- ing neural responses. The usage of synthesized yet photorealistic reality by gener- ative adversarial networks (GANs) allows for superior control over these data: the underlying feature representations that account for the semantics in synthesized data are known priori and their relationship is perfect rather than approximated post-hoc by feature extraction models. We exploit this property in neural decoding of multi-unit activity (MUA) responses that we recorded from the primate brain upon presentation with synthesized face images in a passive fixation experiment. First, the face reconstructions we acquired from brain activity were remarkably similar to the originally perceived face stimuli. Second, our findings show that responses from the inferior temporal (IT) cortex (i.e., the recording site furthest downstream) contributed most to the decoding performance among the three brain areas. Third, applying Fuclidean vector arithmetic to neural data (in combination with neural decoding) yielded similar results as on w-latents. Together, this pro- vides strong evidence that the neural face manifold and the feature-disentangled w-latent space conditioned on StyleGAN3 (rather than the z-latent space of ar- bitrary GANs or other feature representations we encountered far) share how they represent the high-level semantics of the nigh-dimensional space of faces.
INTRODUCTION
The field of neural coding aims at deciphering the neural code characterize how the brain rec- ognizes the statistical invariances of structured yet complex naturalistic environments. Neural en- coding seeks to find how properties of external phenomena are stored in the brain by modeling the stimulus-response transformation (van Gerven 2017). Vice versa, neural decoding aims to find what information about the original stimulus is present in and can be retrieved from the measured brain activity by modeling the response- stimuuus transformation (Haynes & Rees 2006 Kamitani Tong 2005). particular, reconstruction is concerned with re-creating the literal stimulus image from brain activity. In both cases, it is common factorize the direct transformation into two by
Stimulus
Feature
Response
Analysis
Encoding
GU
Synthesis
Decoding
Figure 1: Neural coding. The transformation between sensory stimuli and brain responses via an intermediate feature space. Neural encoding is factorized into nonlinear "analysis" and linear "encoding" mapping. Neural decoding is factorized into a linear "decoding" and nonlinear "syn- thesis" mapping.

---

Under review as a conference paper at ICLR 2023
invoking an in-between feature space (Figure D.. Not only does this favor data efficiency as neural data scarce but it also allows one to test alternative hypotheses about the relevant stimulus features that are stored in and can be retrieved from the brain.
The brain can effectively represent an infinite amount of visual phenomena to interpret and act upon the environment. Although such neural representations are constructed from experience, novel yet plausible situations that respect the statistics of the natural environment can also be mentally simu- lated or imagined (Dijkstra et al.) 2019). From machine learning perspective, generative models achieve the same objective: they capture the probability density that underlies (very large) set of observations and can be used to synthesize new instances which appear be from the original data distribution yet are suitably different from the observed instances. Particularly, generative adversar- ial networks (GANs) (Goodfellow et al. 2014) are among the most impressive generative models to date which can synthesize novel yet realistic-looking images (e.g.. natural images and images of human faces, bedrooms, cars and cats (Brock et al. 2018 Karras et al. 2017 2019 2021 from randomly sampled latent vectors. A GAN consists of two neural networks: generator network that synthesizes images from randomly-sampled latent vectors and a discriminator network that dis- tinguishes synthesized from real images. During training, these networks are pitted against each other until the generated data are indistigguishable from the real data. The bijective relationship of the generator can be exploited in neural decoding to disambiguate the synthesized im- ages as visual content is specified by their underlying latent code (Kriegeskorte) 2015) and perform analysis by synthesis (Yuille & Kersten 2006).
Deep convnets have been used to explain neural responses during visual perception, imagery and dreaming (Horikawa Kamitani] 2017b St-Yves & Naselaris 2018 Shen et al. 2019b a
Güęlüüüükk et al.) 2017]] VanRullen & Reddy 2019 Dado et al. 20222) To our knowledge, the latter three are the most similar studies that also attempted to decode perceived faces from brain activity. GGüçlütürk et al. (017)) used the feature representations from VGG16 pretrained on face recognition (i.e., trained supervised setting). Although more biologically plausible, unsupervised learning paradigms seemed to appear less successful in modeling neural representations in the primate brain than their supervised counterparts (Khaligh- -aagvii & Kriegeskorte| 2014) with the exception of VanRullen & Reddy 2019) and (Dado et al., 2022) who used adversarially learned latent repre- (VAE-GAN) and a GAN, respectively. Importantly, (Dado al. 2022) used synthesized stimuli to have direct access to the ground-truth latents instead of using post-hoc approximate inference, as VAE-GANs do by design.
sentations of a variational autoencoder-GiN
The current work improves the experimental paradigm of (Dado al. 2022 and provides several novel contributions: face stimuli were synthesized by a feature-disentangled GAN and presented to macaque with cortical implants in passive fixation task. A decoder model was fit on the recorded brain activity and the tround-truth latents. Reconstructions were created by feeding the predicted latents from brain activity from held-out test set to the GAN, Previous neural decoding studies
Conv 3x3 or lx]
Conv 3x3 or 1xl
Fourier Feat.
Conv lxl
Upsomple
Downnsomple
Leoky RetU
Upsomple
Downsommle
Leaky ReLU
Conv 1xl
Crap
Crop
L1-11
2=(512.) W=(512,)
L0
L13
(1024, 1024, 3)
Fixed
Learned
A
Affine
Figure 2: StyleGAN3 generator architecture. The generator takes a 512-dim. latent vector as input and transforms it into a 1 resolution RGB image. We collected dataset of 4000 training set images and 100 test set images.
2

---

Under review as a conference paper at ICLR 2023
used noninvasive fMRI signals that have low signal-to-noise ratio and poor temporal resolution leading to a reconstruction bottleneck and precluding detailed spatio-temporal analysis. This work is the first to decode photorealistic faces from intracramial recordings which resulted in state-of-the- art reconstructions as well as new opportunities to study the brain. First, the high performance of explain neural repre-
decoding via w-latent space indicates the importance of disentanglement sentations upon perception, offering a new way forward for the previously limited yet biologically more plausible unsupervised models of brain function. Second, we show how decoding performance evolves over time and observe that the largest contribution is explained by the inferior temporal (IT) cortex which is located at the end of the visual ventral pathway. Third, the application of Euclidean vector arithmetic to w-latents and brain activity yielded similar results which further suggests func- tional overlap between these representational spaces. Taken together, the high quality of the neural recordings and feature representations resulted in novel and unprecedented experimental findings that not only demonstrate how advances in machine learning extend to neuroscience but also will serve an important benchmark for future research.
2
METHODS
2.1
DATA
2.1.1 STIMULI
We synthesized photorealistic face images of 10242 resolution from (512-dim.) z-latent vectors with the generator network of StyleGAN3 (Karras et al. 2020) (Figure 2 which pretrained on the high- quality Flickr-Faces-HQ (FFHQ) dataset (Karras et al. 2019). The z-latents were randomly sampled from the standard Gaussian. First, StyleGAN3 maps the z-latent space via an 8-layer MLP to an intermediate (512-dim.) w-latent space in favor of feature disentanglement. That is, the original z- latent space is restricted to follow the data distribution that it is trained on (e.g., older but not younger people wear eyeglasses in the training set images) and such biases are entangled in the 2- z-latents. The less entangled w-latent space overcomes this such that unfamiliar latent elements can be mapped to their respective visual features (Karras et al. 2019). Second, we specified truncation of 0.7 that the sampled values are ensured to fall within this range to benefit image quality. During synthesis, learned affine transformations integrate w-latents into the generator network with adaptive instance normalization (like style transfer (Huang & Belongie, (017)) as illustrated in Figure [ Finally, we synthesized a training set of 4000 face images that were each presented once to cover a large stimulus space to fit a general model. The test set consisted of 100 synthesized faces that were averaged over twenty repetitions.
2.1.2 FEATURES
For the main analysis, the z- and w-latent space of StyleGAN3 were both used as the in-between feature space. In addition, we also extracted the intermediate layer activations to our face stimuli from alexnet for object recognition (Krizhevsky 2014). VGG16 for face recognition (Parkhi al. 2015) and the discriminator network of StyleGAN3. We fit multiple encoding models to see how well their feature representations can explain the recorded responses. Because the features from VGGFace and the discriminator were very large (~ 10%), we performed downsampling, in (Eickenberg et al. 2017). L.e., for each channel the activation, the feature map was spatially smoothed with Gaussian filter and subsampled such that the total number of output features was lower than 50,000 per image. The kernel size was set to be equal to the downsampling factor.
done
2.1.3 RESPONSES
We recorded multi-unit activity (Super & Roelfsema 2005) with chronically implanted electrode arrays (64 channels each) in one macaque (male, 7 years old) upon presentation with the synthesized face images in a passive fixation experiment (Figure 3. Neural responses were recorded VI (7 arrays), V4 (4 arrays) and IT (4 arrays) leading to total of 960 channels (see electrode placings in Figure T). For each trial, we averaged the early response of each channel using the following time- windows: 125 ms for V1, 55- ms for V4 and 75-175 ms for IT. The data was normalized as in BBahhiva et al. 2019) such that for each channel, the mean was subtracted from all the responses which were then divided by the standard deviation. All procedures complied with the NIH Guide
3

---

Under review as a conference paper at ICLR 2023
G
300 ms 200 ms
200 ms
200 ms
200 ms
200 ms
200 ms
200 ms
200 ms
Figure 3: Passive fixation task. The monkey was fixating a red dot with gray background for 300 ms followed by fast sequence four face images (500² pixels): 200 ms stimulus presentation and 200 ms inter-trial interval. The stimuli were slightly shifted to the lower right such that the fovea corresponded with pixel (150, 150). The monkey was rewarded with juice fixation was kept for the whole sequence.
for Care and Use of Laboratory Animals and were approved by the local institutional animal care and use committee.
2.2 MODELS
We used linear mapping to evaluate our claim that the feature- and neural representation effectively encode the same stimulus properties, as is standard in the field. A more complex nonlinear trans- formation would not be valid to support this claim since this could theoretically map anything to anything. We used regularization for encoding due to the high dimensionality of the feature space.
2.2.1 DECODING
Multiple linear regression was used to model how the individual units within feature representations y (c.g., w-latents) are linearly dependent on brain activity x per electrode:
N Σ 1=1
L=
yy w
(1)
where ranges over samples. This was implemented by prepending dense layer to the generator architecture to transform brain responses into feature representations which were then run through the generator as usual. This response-feature layer was fit with ordinary least squares while the remainder of the network was kept fixed. Note that truncation was applied for the reconstruction from predicted features/latents.
2.2.2 ENCODING
Kernel ridge regression was used to model how every recording site in the visual cortex linearly dependent the stimulus features. That is, an encoding model is defined for each electrode. In contrast to decoding, encoding required regularization to avoid overfitting since we predicted from feature space x; (xx,) where d() is the feature extraction model. Hence used ridge regression where the norm of w penalized to define encoding models by a weighted sum of oxx):
N
C =
Σ ==1
(1:
w T ((xx))) + >//lw/12
(2)
where (x1, X2, ......,,''' )m Nxd y (V1, 02. T E Nx1, N the number of stimulus- response pairs, d the number of pixels, and A; > 0 the regularization parameter. We then solved for w by applying the "kernel trick" (Welling 2013):
Wj (AyI.. + कT by
where - (x(x)) ((x2), )(xx)) IR Nx. (i.e., the design matrix) where g is the number of feature elements and (V1, ....... UN Nx1 This means that must lie in the space induced by the training data even when > N. The optimal X, is determined with grid search, as in GGüçlü & van Gerven 2014). The grid obtained by dividing the domain of X in M values and evaluate
4

---

Under review as a conference paper at ICLR 2023
model performance every value. This hyperparameter domain is controlled by the capacity of the model, i.e,, the effective degrees of freedom dof of the ridge regression fit from [1, N]:
N
dof(A,)
(4)
where s are the non-zero singular values of the design matrix & as obtained by singular value de- composition. We can solve for each AÚ with Newton's 's method. Now that the grid of lambda values defined, we can search for the optimal A that minimizes the 10-fold cross validation error.
2.3 EVALUATION
Decoding performance was evaluated by six metrics that compared the stimuli from the held-out test set with their reconstructions from brain activity: latent similarity, alexnet perceptual similar- ity (object recognition), VGG16 perceptual similarity (face recognition), latent correlation, pixel correlation and structural similarity index measure (SSIM). For latent similarity, we considered the cosine similarity per latent dimension between predicted and ground-truth latent vectors:
2, ...24 2
Latent cos similarity
Di (2 Σ (2))
where 2 and are the 512-dimensional predicted and ground-truth latent vectors, respectively. For perceptual similarity, we computed the cosine similarity between deeper layer activations (rather than pixel space which is the model input) extracted by deep neural networks. Specifically, fed the stimuli and their reconstructions to alexnet pretrained on object recognition (Krizhevsky) (014)) and VGG16 pretrained on face recognition (Parkhi al. (2055) and extracted the activations their last convolutional layer. We then considered the cosine similarity per activation unit:
f(i); f(x):) (())
Perceptual cos similarity
where and f are the 224 x 224 RGB (image dimensionality that the models expects) visual stimuli and their reconstructions, respectively, n the number of activation elements, and f() the image-activation transformation. Latent- and pixel correlation measure the standard linear (Pearson product-momen moment) correlation coefficient between the latent dimensions the predicted and ground- truth latent vectors and the luminance pixel values of stimuli and their reconstructions, respectively. SSIM looked at similarities terms of luminance, contrast and structure (Wang et al.) 2004).
Furthermore, we introduce a new SeFa attribute similarity metric between stimuli and their recon- structions using the intrinsic semantic vectors of the generator which we accessed using closed-fone factorization (" ("SeFa") (Shen & Zhou 2021). In short, the unsupervised SeFa algorithm decomposes the pre-trained weights of the generator into 512 different latent semantics (of 512 dimensions each) which can be used for editing the synthesized images in w-space. This is also means understand what each latent semantic encodes: if face becomes younger or older when traversing the latent conclude post-hoc that encodes the attribute "age". In our case, we used it to score each stimulus and reconstruction by taking the inner product between their w-latent and latent semantic and check for similarity.
in the negative positive direction of the latent semantic,
2.4 IMPLEMENTATION DETAILS
We used the original py Torch implementation of StyleGAN3 Karras et al. 2021). the Py PyTorch implementation of alexnet and the keras implementation of VGGI6. All analyses were carried out in Python 3.8 on the internal cluster.
3 RESULTS
3.1 NEURAL DECODING
We performed neural decoding from the primate brain via the StyleGAN3; see Figure and Table [ for qualitative and quantitative results, respectively. Percep- tually, it is obvious that the stimuli and their reconstructions share significant degree of similarity
w-latent space of
5

---

Under review as a conference paper at ICLR 2023
Stimulus
Reconstruction
Figure 4: Qualitative results: The 100 test set stimuli (top row) and their reconstructions from brain activity (bottom row).
(e.g.. gender, age, pose, haircut, lighting, hair color, skin tone, smile and eyeglasses). The impor- tance of feature disentanglement for neural decoding is highlighted when compared to the decoding performance via the original z-latent space. The qualitative results from z-latent space can be found in Appendix A.1. We provided visual guide for the six evaluation metries in Appendix A.4 by showing the stimulus-recoonstruttin pairs with the five highest- and the five lowest similarity for each metric. In addition, we also repeated the experiment with another macaque that had silicon- based electrodes in VI, V2, V3 and V4; see Appendix A.3.
The contribution of each brain region to the overall reconstruction performance was determined by occluding the recordings from the other two sites. Concretely, the two site responses replaced with the average response of all but the corresponding response such that only the site interest remained. Alternatively, one could evaluate region contribution by training three different decoders
Table 1: Quantitative results. The upper block decoded the z-latent whereas the lower block decoded the w-llatet from brain activity. Decoding performance is quantified terms of six metries: latent cosine similarity, latent correlation (Student's t-test), perceptual cosine similarity using alexnet and VGGFace, pixel-wise correlation (Student's t-test) and structural similarity index (SSIM) in pixel space between stimuli and their reconstructions (mean + std.error). To make the comparison more fair, the predicted z-latents were transformed to w-space and truncated at 0.7 for comparison with the grouhdd-truth w-llatens. The from "all" recording sites or only from a specific brain area. The latter is achieved by occluding the recordings from the other two brain regions in the test set. Neural decoding from all brain regions into the w-latent resulted in the overall highest reconstruction performance.
display performance when using either the recordings
Lat. sim.
Lat. corr.
Alexnet sim.
VGG16 sim.
Pixel core.
SSIM
All
0.3909 + 0.0070
0.2824 = 0.0054
0.2416 + 0.0023
0.1789 = 0.0013
0.4331 + 0.0001
0.3811
VI
0.2785 + 0.0079
0.1398 0.0051
0.1871 1 0.0022
0.1430 0.0012
0.2887 1: 0.0001
0.2640
V/4
0.2763
0.0079
0.1362 = 0.0050
0.1956 + 0.0022
0.1485
0.0012
0.2430 + 0.0001
0.2211
IF
0.3012 0.0076
0.1747 0.0055
0.2054
0.0022
0.0012 0.1498
0.3105 1 0.0001
0.2794
All
0.4579 0.0076
0,2908
0.0047
0.2740 + 0.0029
0.2391
0.0017
0.6055 + 0.0001
0.5547
V7
0.3792 0.0089
0.1478
0.0047
0.1447
0.0023
0.1151
0.0012
0.2958 + 0.0001
0.2256
V4
0.3783 0.0023
0.1450
0.0049
0.1856 + 0.0020
0.1315
0.0011
0.1816 + 0.0001
0.1684
0.4009 IT
0.0085
0.0051 0.1828
0.1861 1: 0.0026
0.0014 0.1451
0.4119 1 0.0001
0.3275

---

Under review as a conference paper at ICLR 2023
0.8217 p= 1.le-25
0.7992 p 2.26-22
== 0.7904 p= 1242222
0.7061 p 2.36-16
r= 0.6987
p= -31e--99
6.4e-16
True
Pred
-Dosse age
-I -Lighting age
-Pose, ago
-Hair, agn
-Agc, teeth, race
~Gender, hair
SCIm
Recon
3
Figure 5: SeFa attribute similarity. The SeFa attribute scores were computed for stimuli and their reconstructions and evaluated for similarity in terms of correlation (Student's 's t-test). The six plots display the attribute scores of the 100 stimuli (True; Y-axis) and the predictions (Pred; X-axis) for the six latent semantics with the highest similarity. We travel in the semantic direction and edit an arbitrary face to reveal what facial attributes a latent semantic encodes. That is, a latent semantic was subtracted of or added to a w-latent which was fed to the generator to create the corresponding face image. For instance, when the semantic boundary with the highest similarity (r = 0.8217) is subtracted from w-latent, the corresponding face changes pose to the left and becomes younger whereas its pose changes to the right and gets older when this semantic is added.
on neural data subsets, but the current occlusion method made it more interpretable how region contributed to the same decoder's performance. As such, we found that this is for the largest part determined by responses from IT which is the most downstream site we recorded from.
We validated our results with a permutation test as follows: for a thousand times, we sampled hundred random latents from the same distribution our original test set and generated their corresponding face images. Per iteration, we checked whether these random latents and faces were closer the ground-truth latent and faces than our predicted latents and faces. We found that our predicted latents from brain activity and corresponding faces were always closer to the original stimuli for the w-latents and all six metrics, yielding statistical significance (p < 0.001). This indicates that our high decoding performance is not just a consequence of the high-quality images that the generator synthesizes. The charts showing the six similarity metrics over iterations for the random samples and our predictions based on brain activity can be found in Appendix A.2.
Next, we quantified how well facial attributes were predicted from brain activity (Figure The 512 latent semantics are known to be hierarchically organized (Shen & Zhou 2021) and we also find this back our predictive performance where the highest and lowest correlations are found at the earlier
Lat. sim.
Lat. corr.
Alexnet sim.
VGCFace sim.
Pixel carr.
SSIM
0.42
0.
0.20
0.8
0.4
0.4
0.39
0.1
0.16
0.33
02
G2
0.32
0.22
0.08
02
03
0.1
0.2
03
01
02
03
0.3
01
02
03
V1
time [sec]
Figure 6: Neural decoding performance in time. The development of decoding performance (Y- axis) in terms of all six metrics based on the recorded response data in V1, V4, IT and all brain regions and over the full time course of 300 ms (X-axis). Stimulus onset happened at 100 ms. For visualization purposes, sigmoid was fit to these data points and the shaded areas denote the predefined time windows we used for our original analysis.
7

---

Under review as a conference paper at ICLR 2023
and later latent semanties, respectively. Face editing revealed that the earlier latent semantics encode clear and well-known facial attributes (e.g., gender, age, skin color, lighting and pose) whereas those in later latent semantics remain unclear since editing did not result in perceptual changes.
Finally, Figure shows how decoding performance evolved time. Rather than taking the average response the predefined time windows for V1, V4 and IT, took the average response over a ms window that was slided without overlap over the full time course of 300 ms. This resulted in thirty responses over time per stimulus. As expected, performance climbed and peaked first for (early) VI, then (intermediate) V4 and lastly (deep) IT. We can see that IT outperformed the other two regions terms of all six metrics.
3.2 NEURAL ENCODING
Neural encoding predicted the brain activity from the eight (feature) layers from alexnet for ob- ject recognition, VGG16 for face recognition and the StyleGAN3 discriminator (i.e., 3 x = 24 encoding models in total). Encoding performance was assessed by correlating the predicted and recorded responses (Student's t-test) after which the model layer with the highest performance was assigned to each recording site the brain (Figure 7 Our results show gradient from early to deeper brain areas for all three models. That is, visual experience is partially determined by the selective responses of neuronal populations along the visual ventral "what" pathway (Ungerleider & Mishkin 1982) such that the receptive fields of neurons in early cortical regions are selective for simple features (e.g., local edge orientations (Hubel & Wiesel [962)) whereas those of neurons in deeper regions respond to more complex patterns of features (Gross et al. 1972: Hung et al., 2005]. Previous work has shown how the features extracted by deep convolutional neural networks predict neural responses in the ventral visual stream to perceived naturalistic stimuli in the human brain (Yamins al. 2014 Cadieu et al. 2014, K Razavi Kriegeskorte 2014] Güiçlü & van Gerven 2015 Yamins & DiCarlo, 2016 Cichy et 20. Eickenberg al. 2017) well as in the primate brain (Freiwald & Tsao 2010; Chang & (217). line with the literature, our results
show that (carly) VI encodes earlier model layers whereas (deeper) IT encodes deeper model layers.
3.3 WALKING THE NEURAL FACE MANIFOLD VIA W-LATENT SPACE
Manifold hypothesis states that real-world data instances can be viewed as points high- dimensional space that concentrated manifolds which (locally) resemble Fuclidean space. Linear changes in the (low-dimensional) GAN latent landscape directly translate to the correspond- ing (high-dimensional) pixel space and thus approximate local manifolds (Shao et 2018). That is, visual data that look perceptually similar in terms of certain features are also closely positioned in latent space. As such, interpolation between two distinct latent vectors resulted in an ordered set
Aloonet
A.
Layer assignment (86)
100
B.
0.8
0
VOG16
100
0.2
Alexnot (obj. recog)
Dicciminator
VGG16 (face recog.)
100
Discriminator
50
-02
LB
V4
All
8888 Layers
Visual
Figure 7: A. Layer assignment across all electrodes to visual areas VI, V4 and IT. We observe gradient for Alexnet, VGG16 and the discriminator where early layers are mostly assigned to V1, intermediate layers to V4 and deep layers to IT. B. Performance all 24 encoding models (x-axis) in terms of correlation (Student's t-test, y-axis) between all predicted and recorded brain responses shows that encoding performance was good overall.
8

---

Under review as a conference paper at ICLR 2023
A.
NIPE
Inter Interoclation
Interpolatie intepolation
Interpolation
B.
8ay
Woman
Girl
Smilng
Neutral
Meusral
Smillig
Blonde
Brunette
Blonde
Figure 8: Linear operations were applied to ground-truth w-latents (row 1), w-latents decoded from brain activity (row 2), recorded responses which we then decoded into w-latents (row 3) and encoded responses from ground-truth w-latents which we then decoded w-latents. A. Linear interpolation between the w-latents of two test set stimuli. B. Vector arithmetic with the average w-latent of specific facial attributes. To obtain the average w-latent we averaged as many relevant samples as possible from the test set.
of latents where contained semantics in the corresponding images vary gradually with latent code. We used spherical rather than regular linear interpolation to account for the spherical geometry of the latent space (i.e., multimodal Gaussian distribution). The latent space also obeys simple arith- metic operations (Mikolov al.] 2013). The generated faces from interpolation (Figure (8IA) and vector arithmetic (Figure 3) in neural space were perceptually similar as when applied directly to the w-llatent space. This indicates that the functional neural face manifold and w-llaentt space are organized similarly such that responses can be linearly modified to obtain responses unseen faces.
4 DISCUSSION
Neural decoding of brain activity during visual perception via the feature-diseetanglled w-latent space conditioned StyleGAN3 resulted in image reconstructions that strongly resemble the originally-perceivee stimuli, making it the state-of-the-aat in the field. Although it is safe to as- sume that the brain represents the visual stimuli it is presented with, it has been largely unsolved how it represents them as there are virtually infinite candidate representations possible to encode the same image. The goal of this study was find the correct representation. Our results demon- strate that StyleGAN3 features/latents are linearly related to brain responses such that latent and response must encode the same real-world phenomena similarly. This indicates that StyleGAN3 successfully disentangled the neural face manifold (DiCarlo & Cox (007) rather than the conven- tional z-latent space of arbitrary GANs or any other feature representation we encountered so far. Note that StyleGAN3 has never been optimized on neural data. We also found that the features of the discriminator are predictive of neural responses. The StyleGAN3-brain correspondence can shed light on what drives the organization of (neural) information processing in vision. For instance, the analogy between adversarial training of StyleGAN3 and predictive coding where the brain is continuously generating and updating its mental model of the world to minimize prediction errors. To conclude, unsupervised generative modeling can be used to study biological vision which in turn supports the development of better computational models thereof and other (clinical) applications.

---

Under review as a conference paper at ICLR 2023
5 ETHICS STATEMENT
In conjunction with the evolving field of neural decoding grows the concern regarding mental pri- vacy. Because think it likely that access subjective experience will be possible in the foreseeable future, we want to emphasize that it is important to at all times strictly follow the ethical rules and regulations regarding data extraction, storage and protection. should never be possible to invade subjective contents of the mind.
6 REPRODUCIBILITY STATEMENT
Upon publication, we will make our data publicly available together with the code and detailed description on how to recreate our results to ensure transparency and reproducibility.
REFERENCES
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6439), 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.1 100::9999911, 2018.
Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James J DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS computational biology. 10(12):e1003963, 2014.
Le Chang and Doris Y Tsao. The code for facial identity in the primate brain. Cell, 169(6):0013- 1028, 2017.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific reports, 6(1):1-13, 2016.
Thirza Dado, Yagmur Güęlüäürk, Luca Ambrogioni, Gabriëlle Ras, Sander Bosch, Marcel Ger- ven, and Umut Güçlü. Hyperrealistic neural decoding for reconstructing faces from fmri activa- tions via the gan latent space. Scientific reports, 12(1):1-9, 2022.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 1 11(8):333-33 341, 2007.
Nadine Dijkstra, S.E. Sander E. Bosch, and Marcel A.J. M.A.J. van Gerven. Shared neural mech- anisms of visual perception and imagery. Trends in Cognitive Sciences, 23(5):423-434, 2019. ISSN 1879307X. doi: w...0066..iic..211.22.... tics. 2019.011100 02. 004
URL https: org. 1016/
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184 194, 2017.
Winrich A Freiwald and Doris Tsao. Functional compartienntalzzation and viewpoint generaliza- tion within the macaque face-processing system. Science, 330(6005):845--0 -488851, 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Charles G Gross, CE de Rocha-Mirande, and DB Bender. Visual properties of neurons in inferotem- poral cortex of the macaque. Journal of neurophysiology. 35(1):96-11 111,, 1972.
Umut Güçlü and Marcel AJ Gerven. Unsupervised feature learning improves prediction of human brain activity in response to natural images. PLoS computational biology, 100037443 2014.
10

---

Under review as a conference paper at ICLR 2023
Umut Gügllü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005-00014 2015.
Yagmur Güçlürürk, Umut Güiçlü, Katja Seeliger, Sander Bosch, Rob van Lier, and Marcel van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural de- coding. Advances in neural information processing systems. 30, 2017.
John-Dylan Haynes and Geraint Rees. Decoding mental states from brain activity in humans. Nature reviews neuroscience, 7(7):523-555 2006.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):1-15, 2017a.
Tomoyasu Horikawa and Yukiyasu Kamitani. Hierarchical neural representation dreamed ob- jects revealed by brain decoding with deep neural network features. Fronfiers in computational neuroscience, 11:4, 2017b.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture the cat's visual cortex. The Journal of physiology,
1962.
Chou Hung, Gabriel Kreiman, Tomaso Poggio, and James DiCarlo. Fast readout of object identity from macaque inferior temporal cortex. Science, 310(5749):863 866--666,, 2005.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8()):679-685, 2005.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4414 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. I Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-81 19, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34, 2021.
Seyed-Mahdi thaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised. models may explain it cortical representation. PLoS computational biology, 10( 1000031953 2014.
Nikolaus Kriegeskorte. Deep neural networks: new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint 49X4v 2014.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 41.1-41.11. BMVA Press, September 2015. ISBN 1-901725-53-7, doi: 10.5244/C. 29.41. URL httpt:tt:///ttt /tttts://dx..o doi www.oooorr////
29../22.9.. 29.41

---

Under review as a conference paper at ICLR 2023
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 315-323, 2018.
Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End- to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuro- science, pp. 21, 2019a.
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruc- tion from human brain activity. PLoS computational biology, 15(1)e 2019b.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1532-1540, 2021.
Ghislain St-Yves and Thomas Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. In 2018 IEEE international conference on systems. man, and cybernetics (SMC), pp. 1054-1061. IEEE, 2018.
Hans Super and Pieter R Roelfsema. Chronic multiunit recordings in behaving animals: advantages and limitations. Progress in bruin research, 147:263-282, 2005.
L.G. Ungerleider and M. Mishkin. Two cortical visual systems. In Analysis of visual behavior, pp- 549-586-. MIT Press, Cambridge, MA, 1982.
Marcel AJ van Gerven. A primer on encoding models in sensory neuroscience. Journal of Mathe- matical Psychology, 76: 72-183, 2017.
Rufin VanRullen and Leila Reddy. Reconstructing faces from fmri patterns using deep generative neural networks. Communications biology. 2(1):1-10, 2019.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4))600- 612, 2004.
Max Welling. Kernel ridge regression. Max Welling's classnotes in machine learning, pp. 1-3, 2013.
Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-36,, 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optmmizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy sciences, 11 111(23):8619 8624, 2014.
Alan Yuille and Daniel Kersten. Vision bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006.
12


Reviews:
[{"review_id": "kU0QsgirW8", "reviewer": "Reviewer_1pMd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author found that the neural responses of macaques to facial visual stimuli are very similar to the w-space of styleGAN3. Specifically, the neural response of the macaques' brain can be decoded into w-space using linear mapping, and the corresponding generation faces are similar to the perceived stimuli. Also, the reconstructing looks similar when applying the same linear operations on the brain activity and the latent.", "summary_of_the_review": "This paper provides evidence that the neural face manifold and the disentangled w-latent space conditioned on StyleGAN3 are similar in how they represent the high-level semantics of the high-dimensional space of faces. But more comparison are suggest to be added between the reconstruction quality of z-latent and w-latent.", "strength_and_weaknesses": "Strength:The author finds evidence that the neural face manifold and the disentangled w-latent space conditioned on StyleGAN3 are similar in how they represent the high-level semantics of the high-dimensional space of faces. Especially, the StyleGAN3 has never been optimized on neural data.\n\nWeaknesses: \n1) Why the author use a AlexNet pretrained on object recognition task rather than on the face recognition task? What is the different between the cosine similarity from the AlexNet and the VGGFace?\n2) For the conclusions in the summary, “This provides strong evidence that the neural face manifold and the disentangled w-latent space conditioned on StyleGAN3 (rather than the z-latent space of arbitrary GANs or other feature representations we encountered so far) share how they represent the high-level semantics of the high-dimensional space of faces”, the author should add some qualitative comparison to make a visual support. Also, the qualitative comparison can give the reader a visual feeling of the difference in image space caused by the difference in six decoding performance evaluation metrics.\n3) The gap between the z-latent and w-latent in the “Alexnet sim” and “VGGFace sim” are much more lower than in the “Lat. sim.” and the “Lat. corr.”, how does the author explain this phenomenon? How much does the difference in “Alexnet sim” and “VGGFace sim” affect the reconstruction image quality? If the visualized difference is not obvious, it will weaken the conclustion that “This provides strong evidence that the neural face manifold and the disentangled w-latent space conditioned on StyleGAN3 (rather than the z-latent space of arbitrary GANs or other feature representations we encountered so far) share how they represent the high-level semantics of the high-dimensional space of faces” in the abstract. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VUgdajX6Uew", "reviewer": "Reviewer_1cBk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the encoding and decoding of faces in the (macaque) brain. To study decoding, they generate faces by sampling a random vector from the latent space of a GAN. They show the faces to a macaque monkey and record the neural response. A linear model is trained to map the neural response to the latent vector that produced the face image. On test data, they find that the recovered vector can be used to produce a high quality image of a face that matches the original face qualitatively and quantitatively. To study encoding, they map the activations of a trained vision model (AlexNet, VGG16) to the neural response. They find that correlation with the neural response increases with depth in the machine model.", "summary_of_the_review": "The reconstruction results are impressive, but have already been accomplished in many previous works. The decoding techniques are not significantly different than what was introduced in prior work, although the data is new. But even then, the data consists only of neural recordings for a single subject.", "strength_and_weaknesses": "# strengths\n- New macaque monkey data.\n- This seems like the first attempt at reconstructing faces from intracranial data. (Is it?) If so, that should be mentioned somewhere\n- The reconstructed images are impressive\n\n# weaknesses\n- I have concerns about the originality of this work. Decoding high quality faces from brain signals has been previously accomplished (Dado et al. 2021, VanRullen and Reddy 2019, Güçlütürk and Güçlü 2017). The decoding and encoding techniques, namely the use of GAN latent spaces, have already been discussed in previous works. In this work, the authors use a slightly different model (StyleGAN3 vs StyleGAN), but the overall methods remain the same.\n- The analysis of reconstructed faces in terms of attributes is also introduced in these previous works. (The authors do acknowledge this)\n- I think the paper could be much improved if the relationship with previous works were more clearly explained.\n- The novel contribution of this work seems mainly to be the newly collected data. But even this is only done for one subject.\n- The difference between the $w$ latent space and $z$ latent space could be explained more clearly. It's mentioned that the $w$ latent space is obtained by passing the $z$ latent space through an MLP, but since it results in much better reproductions (relative to $z$), could the authors explain a little bit more about what makes the two spaces different?\n\n## references\n- Dado et al. 2021 Hyperrealistic neural decoding: Reconstructing faces from fMRI activations via the GAN latent space (https://www.biorxiv.org/content/10.1101/2020.07.01.168849v3.full)\n- VanRullen and Reddy 2019 Reconstructing faces from fMRI patterns using deep generative neural networks https://www.nature.com/articles/s42003-019-0438-y\n- Güçlütürk and Güçlü et al. 2017 Reconstructing perceived faces from brain activations with deep adversarial neural decoding", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HFgUHMKVCU", "reviewer": "Reviewer_4n7A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about \"Neural coding\", that is to characterize the biological brain recognization. They used a StyleGAN3 to synthesis several high solution images, and show those images to a 7 years old macaque , and recorded its multi-unit brain activity. Finally, they use several latent similarity metric to compute distant z and w space. Finally, they conclude that StyleGAN3 W space is better than Z space in decoding performance. ", "summary_of_the_review": "It is my first time to  know \"neural coding\" field. \nThe experiments that to explore brain actively with neural representation is interesting. \nAlthough the there is a face that StyleGAN3 W space is more disentangled than Z space, the experimental shows its closer relationship to brain perceived stimuli is exciting.  \n", "strength_and_weaknesses": "Pro: \nThe idea to evaluate StyleGAN3 W space and Z space by using a macaque is interesting. \nThe experiments are extensive and solid. \nCon:\nI believe this paper is more experimental. There is not much novelty. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject