Mind Reader: Reconstructing complex images from brain activities
Authors: Sikun Lin
Thomas Christopher Sprague
Ambuj Singh
Venue: NeurIPS 2022 Accept (2022)

Abstract: Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects or simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and can reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applying state-of-the-art deep learning models to this problem. We find that incorporating an additional text modality is beneficial for the reconstruction problem compared to directly translating brain signals to images. Therefore, the modalities involved in our method are: (i) voxel-level fMRI signals, (ii) observed images that trigger the brain signals, and (iii) textual description of the images. To further address data scarcity, we leverage an aligned vision-language latent space pre-trained on massive datasets. Instead of training models from scratch to find a latent space shared by the three modalities, we encode fMRI signals into this pre-aligned latent space. Then, conditioned on embeddings in this space, we reconstruct images with a generative model. The reconstructed images from our pipeline balance both naturalness and fidelity: they are photo-realistic and capture the ground truth image contents well.

Mind Reader: Reconstructing complex images from brain activities
Sikun Lin
Thomas Sprague Ambuj K Singh
UC Santa Barbara {sikun, tsprague, ambuj)oucsb.
Abstract
Understanding how the brain encodes external stimuli and how these stimuli can be decoded from the measured brain activities are long-standing and challenging questions in neuroscience. In this paper, we focus on reconstructing the complex image stimuli from fMRI (functional magnetic resonance imaging) signals. Unlike previous works that reconstruct images with single objects simple shapes, our work aims to reconstruct image stimuli that are rich in semantics, closer to everyday scenes, and reveal more perspectives. However, data scarcity of fMRI datasets is the main obstacle to applying state-of-the-aar deep learning models to this problem. We find that incorporating an additional text modality beneficial for the reconstruction problem compared to directly translating brain signals to images. Therefore, the modalities involved in our method are: voxel-level IMRI signals, (ii) observed images that trigger the brain signals, and (iii) textual description of the images. To further address data scarcity, we leverage an aligned languaee latent space pre-trained on massive datasets. Instead of training models from scratch to find a latent space shared by the three modalities, we encode fMRI signals into this pre-aligned latent space. Then, conditioned on embeddings in this space, we reconstruct images with a generative model. The reconstructed images from our pipeline balance both naturalness and fidelity: they are photo-realistic and capture the ground truth image contents well.
Introduction
In effort to understand visual encoding and decoding processes, researchers in recent years have curated multiple datasets recording fMRI signals while the subjects are viewing natural images Н 34 ETT. In particular, the Natural Scenes Dataset (NSD (3) was built to meet the needs of data-hungr deep learning models, sampling at an unprecedented scale compared to all prior works while having the highest resolution and signal-to-nois ratio (SNR). addition, all the images used in NSD are sampled from MS-COCO [22]. which has far richer contextual information and more detailed annotations compared to datasets that are commonly used in other fMRI studies (e.g., Celeb A face dataset 23 ImageNet (10), self-curated symbols, grayscale datasets). This dataset, therefore, offers the opportunity explore the decoding of complex images that are closer to real-lift scenes.
Human visual decoding can be categorized into stimuli category classification M. stimuli identi- fication [ and reconstruction. We focus on stimuli reconstruction in this study. Different from previous efforts in reconstructing images from fMRI
we approach the problem with one more modality, that of text. The benefits of adding the text modality are threefold:
5.
24
35
first, the brain is naturally multimodal. Research 7 1 255 indicates that the brain is not only capable of learning multisensory representations, but larger portion of the cortex is engaged in multisensory processing: for example, both visual and tactile recognition of objects activate the same part of the rejontvre cortex (26]. Visual-linguistic pathways along the border of the occipital lobe [27 also bring a more intertwined view of the brain's 's representation of these two modalities. Second, multimodal deep models tend to explain the brain better (having higher representation correlations)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Inputs/Oiiguts
Frozen Weights
Generation
COCO
Intermediaaes
Trainned Weights
API
Losses
IMRI activily
Xinri
Xtang
Mayping
NOE
fm
mapping moddlt
Aftice
Oqals
Paale
cambeddingg CLIP space
CLJPans
conditional StyloGAN2
Reater
Git
Mare
Dasie
reconstructed
Image
Foxi: FoloRRel
(a)
(b)
Figure I: The pipeline for reconstructing seen images from fMRI signals. details different components, from collected data to the reconstructed image. The pipeline trained two stages: during the first stage, mapping models fmi Jm are trained to encode IMRI activities into the CLIP embedding space. In the second stage.
conditional generator G and contrastive discriminator D are finetuned while both fmis J' are kept frozen. shows the image generation process once models are trained.
lb
than the visual-only models, even when compared with activities in the visual cortex ( Lastly, our goal is to reconstruct complex images that have multiple objects in different categories with intricate interactions: it is natural to incorporate contextual information as an additional modality.
Instead of training a model to map all three modalities (fMRI, image, text) to unified latent space, we propose to map fMRI to a well-aligned space shared by image and text, and use conditional generative models to reconstruct seen images from representations in that space. This design addresses the data scarcity issue of brain datasets by separating fMRI from the other two modalities. In this way, large amount of data is readily available to learn the shared visual- language representation and to train generative model conditioned on this representation. Furthermore, pre-trained models can be utilized to make the whole reconstruction pipeline more efficient and flexible.
Our contributions are as follows: (1) to the best of our knowledge, this is the first work on recon- structing complex images from human brain signals. It provides opportunity to study the brain's s visual decoding in more natural setting than using object-centered images. Compared to previous works, it also decodes signals from more voxels and regions, including those outside the visual cortex, that are responsive to the experiment. This inclusion allows us to study the behavior and functionality of more brain areas. (2) We address the data scarcity issue by incorporating additional text modality and leveraging pre-trained latent space and models. For the reconstruction, we focus on semantic representations of the images while taking low-level visual features into account. (3) Our results show we can decode complex images from fMRI signals relatively faithfully. We also perform microstimulation on different brain regions to study their properties and showcase the potential usages of the pipeline.
2 Method
Recent developments in contrastive models allow more accurate embeddings of images and their semantic meanings the same latent space. This performance is realized using massive datasets: models such as CL.IP (28) and ALIGN (17] utilize thousands of millions of image-text pairs for representation alignment.
comparison, brain imaging datasets that record pairs of images and fMRI range from 1.2k to 73k samples, making it difficult to Iearn brain encoding and decoding models from scratch. However, we can utilize aligned embeddings obtained from pre-trained contrastive models as the intermediary and generate images conditioned these embeddings. In pipeline, as shown in fig. I we first map fMRI signals to CLIP embeddings of the observed image and its captions, then pass these embeddings conditional generative model for image reconstruction.
2.1 Caption screening
Each image Xing in the COCO dataset has five captions (Xeap, .... Xeup,) collected through Ama- zon's Mechanical Turk (AMT), and in nature, these captions vary in their descriptive ability. Fig.

---

Captions
probubilities
(1) A group of people sitting and standing top of sandy beach.
0,0376
(2) A surtboard rests on the beach while people play in the waves.
0.519
(3) A surfboard on the sand and people on the beach behind.
0.4302
people hanging out and appreciating taeir time.
0,001286
(5) group of people are sitting on the beach shore.
0.0122
Figure 2: Image caption sereening through CLIP encoders. For this sample, threshold is put half of the largest probability; 0.5 0.519. Therefore, captions (2) and (3) of the image are kept.
shows a sample image with its five captions, and we can tell captions (2) and (3) are more objective and informative than caption (4) when it comes to describing the content of that image, thus are more helpful to serve as the image generation condition. We utilize pre-trained CLIP encoders to screen the high-quality captions since representations in the CLIP space are trained to be image-text aligned. A caption with an embedding more aligned to the image embedding is more descriptive than less aligned one: also less general and more specific this particular image because of the contrastive loss in CLIP. For the screening, we pass each image together with its five captions to the CLIP model, which outputs corresponding probabilities that the captions and image are proper pairs. We keep captions with probabilities larger than half of the highest probability. After screening out less informative captions, we have one to three high-quality captions per image.
2.2 Mapping fMRI signals to CLIP space
Each fMRI signal that reflects a specific image is 3D data volume, and the value on position (i, j, k) the relative brain activation on this voxel triggered by the image. We apply an ROI (region of interest) mask on this 3D volume to extract signals of cortical voxels that are task-related and have good SNRs. The signal is then flattened into a ID vector and voxel-wise standardized within each scan session. The end results Xfmri used by our image reconstruction pipeline. We choose to use the ROI with the widest region coverage, and the length N of Ximii ranges from 12682 to 17907 for different brains in the NSD dataset.
Our goal is to train two mapping models, fmi and fm in fig. T (collectively denoted fm), that
encodes Ximii E IR N to bing = Cimg (Xmmg) E IR and heap = Crse E R212 respectively. Here Cimg, Cixa are CLIP image and text encoders, and Xeap is one of the image captions chosen randomly from the vetted caption pool. We construct both f' as CNN with one Conv ID layer followed by four residual blocks and three linear layers. The training objective is a combination of MSE loss, cosine similarity loss, and contrastive loss on cosine similarity. We use the infoNCE definition of contrastive loss, for the th sample in batch of size B:
39
expícos(a (8) (())
Contra(a ( (0)0 = E, log B 21=1 exp(cos(a b(1)TT)
For the mapping model fox that encodes fMRI to image embeddings, we have h hing = fmi (xmmi) The training objective is:
L.n E.
| 1 '9' 1 + az(1 - ch (9' m) + as Contra(h ming
cos(h
(8)
(4)
(2)
where T, T, 022, 03 are non-negative hyperparameters selected through sweeps. The loss Lmc for caption embedding mapping model fme is defined similarly. Although CLIP embeddings are trained to be aligned, there are still systematic differences between image and text embeddings, with embeddings under each modality showing outlier values at few fixed positions. In addition, we also notice the generated images emphasize either image content (object proximity, shape, etc.) or semantic features depending on which condition we use. Therefore, including both embeddings as the conditions for a generator can cover both ends, and that is why we train two mapping models for the two modalities. Since the outlier indices fixed for each modality across images, clipping the value should not affect image-specific information. Therefore, before normalizing the ground truth embeddings into unit vectors, we set h clamp(h, -1.5, 1.5). This can greatly improve the mapping performance during training.
2.3 Image reconstruction with CLIP embedding conditioning
The mapping models output fMRI-mapped CLIP embeddings hi'ng and h'ap that serve as conditions for the generative model. We aim to generate images that have both naturalness (being photo-
3

---

realistic) and high fidelity (can faithfully reflect objects and relationships in the observed image). Our generation model is built upon Lafite 42), text-to- sex--o--mmage generation model: adapts unconditional StyleGAN2 20 19
to conditional image generation contexted on CLIP text embeddings.
In our generator G, both conditions mng and hap are injected into the StyleSpace: each of them goes through two fully connected (FC) layers and is transformed into condition codes Cimg and Ceap- These condition codes are max- max--ooled and then concatenated with the intermediate latent code w E W, which is obtained from passing the noise vector 2 Z through mapping network (see fig. I Using a mapping network to transform z into an intermediate latent space W is the key of StyleGAN as W is shown to be much less entangled than 2 [36]_ The conditioned style is then passed to different layers of G as in StyleGAN2, generating image Ximg".
s w/l max(Cimg: Ceap), Ximg = G()).
We align the semantics of generated Ximg' and condition vectors by passing Ximg through pre-trained CLIP encoders and apply contrastive loss (eq. (5) Lez) between them. For further alignment of the lower-level visual features, such prominent edges, corners and shapes, we also pass the image through resnet50 and align the position- wiie averaged representation obtained from Layer2 (eq. C.a).
The discriminator D has three heads that share a common backbone: the first head Da classifies images to be real/fake, the second and the third semantic projection heads D.i. Dec map Ximg' to h mg and h/ The latter two ensure the generated images are faithful to the conditions. It is also shown that contrastive discriminators are useful for preventing discriminator overfitting and improving the final model performance [8 16. Applying contrastive loss (eq. 5 La between the outputs from discriminator semantic projection heads and the condition vectors fed to G can therefore help stabilize the training. To summarize the objective function, the standard GAN loss is used to ensure the naturalness of generated Ximg':
LGANG E, (Da (9'
()]
(4)
LCAND -E, [log (Da(x))) log(1 o (Da
where a denotes the Sigmoid function. Meanwhile, contrastive losses used to align the semanties of generated images and the fMRI-mapped condition vectors that supposedly residing in the CLIP space:
(x)) ). heip + Contra (D., (x (6) Le = Contra (D (x ing
hing
L2 Contra (x iing h())) Contra(Cing (t)') (()))
(5)
Les Contra(ResNNet(x!!) (5) ResNet(x(mn))
The overall training objectives are: LG = LGANG + AlLet + AaLea + Aahes, LD = LGAND + AiCelt, where Å1, A2, As, non-negative hyperparameters.
The whole generation pipeline, consisting of mapping models and GAN, is trained in two stages. First, mapping models fmi and Smc are trained on fMRI-CLIP embedding pairs. Next, starting from the trained mapping model weights and Lafite language-free model weights, we modify the losses and model structure and finetune the conditional generator. For the additional condition vector projection layers in G and semantic head in D. we duplicate the weights in the existing parallel layers to make the model converge faster. Note that Lafite is pre-trained on the Google Conceptual Captions 3M dataset {3 then finetuned on the MS-COCO dataset, both of which are much larger than NSD. Finetuning from it allows to exploit the natural relationships between semantics and images with sparse fMRI data. We can still utilize a two-stage training to compensate for data scarcity even if pre-trained conditional GAN like Lafite is available, for example, when using a different generator architecture. Only this time, we should firstly train the conditional GAN on a large image dataset with noise perturbed hing and heap as the pseudo input condition vectors.
3 Results
3.1 Data and experimental setup
The NSD data is collected from eight subjects. We focus on reconstructing observed scenes from a single subject's brain signals. The reasons are twofold: first, it is more accurate to utilize individual

---

brain coordinates instead of mapping voxels into shared space, which can result in information loss during the process, More importantly, brain encoding and perception are different among individuals. This project aims to get the best reconstruction for single individual, thus training models on one subject's s data. Nevertheless, the commonality of this encoding process among the population an exciting topic for future explorations.
We use subject one from NSD: the available data contains 27750 IMRI-image sample pairs on 9841 images. Each image repeats up to three times during the same or different scan sessions. Note that brain responses to the same image can differ drastically during the repeats (fig. Z The dataset is split image-wisn wise:: 23715 samples corresponding to 8364 images are used as the train set, and 4035 samples corresponding to the remaining 1477 images are used as the validation set. Therefore. our pipeline never sees the image it will be tested on during the training. We use 1085mm-resolution scans and only consider fMRI signals from voxels in the nsdgeneral ROI provided by the dataset. This ROI covers voxels responsive to the NSD experiment (voxels with high SNR) in the posterior aspect of the cortex, and contains 15724 voxels for subject one (Ximn E R 167244). Images are all scaled to 256 x 256. Additional experiment settings, including hyperparameters of two training phases, are provided in appendices A.I and A.2 Our experiments are conducted on one Tesla V100 GPU and one Tesla T4 GPU. The code is publicly available
3.2 Mapping models from IMRI to CLIP embeddings
Evaluation criteria In the first training stage, mapping models fmi and fm are trained to encode fMRI signals to CLIP embeddings. We use two criteria to evaluate the mapper performance to decide which one to use in the next stage. The first criterion is FID (Fréchet Inception Distance) [14] between generated image and ground truth using the trained mapper and pre- pre- vre-trained generator. Given a Lafite model pre-trained on MS-COCO (language-fie setting), we can replace its conditional vector with the outputs of our mapping models to generate images conditionally. These FIDs can indicate the starting points of the finetuning processes: the lower the FID, the better the candidate model. Secondly, we use the success rate of image "retrieval" in batch of size 300. For the ith sample in the batch, if the cosine similarity between h())' and h is larger compared to between h6 and h . * i, then it counts as one successful forward retrieval. For backward retrieval, we count the number of correct matches of h()) to all h()),
Configuration comparisons We tested different configurations on the mapping models, including: (1) Whether to place the threshold at 1.5 as mentioned in section 2.2 (2) When training smi. whether perform image augmentations before passing images through the CLIP encoder; (3) When training fmo. whether to use the CLIP text embedding of a fixed caption, a random valid caption, or use the average embedding of all valid captions: (4) Which loss function use: MSE only, cos (cosine similarity) only, Contra only, MSE + cos, MSE + cos - Contra; (5) Whether auxiliary networks help. We tested adding an auxiliary discriminator with GAN loss, as well as adding auxiliary expander networks with VICReg loss 4..
We found: (1) Clamping ground truth embeddings significantly increase performance; (2) Using image augmentations increase fms performance. This further indicates CLIP embeddings are more semantic related; (3) For fmc: selecting random caption from the valid caption pool each time is better than using a fixed one or using the average embedding of all valid captions: (4) Using MSE + cos as the loss gives the best base models, but then finetune these base models with MSE + cos + Contra can further lower the starting FID for pipeline finetuning, making the training in the next stage converge faster; (5) Adding auxiliary networks and objectives will not improve the performance. In general. although heap and himg are already relatively well aligned, fmc can still map Ximri closer to heap than hung. whereas fnni maps Xmmii to an embedding that is equally close to both, while being able to capture few extreme values in himg (see appendix A.3 for numerical details and mapped embedding visualizations). We think this difference reflects that it easier to map fMRI signals to a more semantic representation (from the text space) than to a visual one.
To verify fMRI-mapped embeddings h' semantically well aligned with ground truth CLIP embeddings, we examined the mismatches during the image retrieval. For four incorrect retrievals, fig. 3 shows which images' h(''' are closer to the ground truth images' h (1) than h (9' Notably, these mismatches are semantically close to the ground truth images. This indicates that the mapping models
https//githubbccm///kiii////miii
5

---

Figure 3: Mismatches are semantically close to the ground truth. Figure shows examples of incorrect matches j (red frame) in a batch of 300 in the validation set. For each ground truth image i (green frame), we pass through CLIP encoder to get (9) and through fme to get (0"", The shown incorrect ones are those images with h6),, j # that is closer to h ( than (()).
Table 1: FID of the pipeline under different seltings.
FID
fms
fme
fma fm
from supervised from LF
without Les
fm frozen
37.75
41.51
without Les
fm frozen
30.83
33.78
50.59
from LF
with Les
fm frozen
29.74
33.35
49.47
from LF
with Les
end to end
45.02
48.54
50.96
can successfully map IMRI signals into a semantically disentangled space. Embeddings in this space are suitable for providing contexts to conditional generative model. We also tested another mapping model fmr that maps fMRI signals to representations obtained from resnet50 Layer2. Unlike the CLIP embedding space, the resnet vector encodes more lower-level visual features. We see a jump in the image retrieval rate when we combine the representations obtained from fmi Jm with fer (table However, the generative model is difficult to train when taking in two conditions from distinct embedding spaces. Therefore, we add the low-level vision constraint into the contrastive loss f.s instead.
3.3 Conditional image generation
Quantitative results In the second training stage, we finetune the conditional StyleGAN2 There is no standard metric to measure image reconstruction quality from fMRI signals for complex images. Since previous works focused on reconstructing simpler images, the metrics typically involve pixel- wise MSE or correlation measures. However, when comes to complex images, it seems more reasonable
perceptual metric, such as FID, which is based Inception V3 [rrl activations and is widely used in GAN. We also detail another metric, n-way identification accuracy, that reflects more of the fidelity and uniqueness of the generated images, in appendix A.4 We perform the ablation studies on the pipeline to answer the following questions: (1) Which mapping model trained in stage one leads to the best final performance? J or ʃmi or using both? (2) Which pre-trained GAN leads to the best final performance? For this, we compare using Lafite pre-trained on either the langue-free (LF) setting or the fully supervised setting. (3) Whether including the contrastive loss
Les between lower-level visual features can further improve the performance of semantic- generative model? Finally, we tested (4) whether finetune the whole pipeline end-to-end or freezing the mapping models is better? The new mapping model losses are set to L',, = L. + CaCaAN if trained end-to-end.
Results are reported in table We observed the following: (1) in terms of FID, using himg obtained from fmi as the generator condition is better than using the h' from fm or using two conditional heads. On the other hand, fnee and the two-head setting achieve as good or even better performance as fmi does in terms of n-way identification accuracy. In addition, if training time or resource is the concern, using two heads and pre-trained LF-Lefete with only condition feeding interface changes and cloned weights in the new branches can already give reasonably good results. (2) Training the pipeline on LF-Lafite is much better than on the fully supervised Lafite. This result expected for the generator conditioned on bing since the supervised version is conditioned on CLIP text embeddings.
?ooes are adapted from httpp://gghuu..oom////lasssssggs
https://gitubb.oom//bbopgg

---

Ground truth stimuli (top row) and generated images conditioned IMRI (bottom row).
(b) Generated images from three different fMRI scans responding to the same stimulus (green frames).
Figure 4: Images generated by our pipeline given input fMRI signals.
ground truth
DNN
DNN+DGN
GAN 35
EncDec 6
Dec
SSGAN
D-VAF/GAN 32
(a) Image reconstruction results from [MRI in previous works.
(b) Image reconstruction results from fMRI by our pipeline. Four ground truth images are green framed.
Figure 5: Comparisons between previous works and our pipeline. We are using the recent NSD dataset that involves more complex scenes. However, for comparison purposes, we choose four similar images from NSD, each containing a single object "plane", and show our reconstructions from IMRI signals in fig. 56
However, the same discrepancy exists for the generator conditioned on h' This may reflect the flexibility of pre-trained generators adapt the slight changes in the embedding space. It also shows the crucial impact of pre-trained model on final performance when training data is limited. (3) The addition of low-level visual feature constraint Ces is beneficial for the model performance, especially faithfulness. It also seems to have more effects on single-head models than the two-head one. (4) For the end-to-end pipeline training, we test performance with A [0.1, 1, 10), all of which give worse performance than keeping the mapping model weights frozen (reported values are from A 1). particular, we found that h'' tends collapse having nonzero values only a few positions if the mappers are finetuned together with GAN.
Qualitative results We show several generated images in fig. Although the generator takes in both the noise vector and IMRI-mapped embeddings, the results vary much more with the latter condition, while only contributes to variations on some minor details. In general, the generated images capture both semanties and visual features relatively well, even complex images containing interactions of multiple objects. Since each stimulus is repeated up to three times to the subject, we have multiple fMRI scans corresponding to the same image. The semantic differences the generated images conditioned on these multiple scans could potentially reveal brain processing discrepancies of the same stimulus. For example, the three generations for the second image fig. eblemsais respectively: (1) the overall scene and the fence, (2) people with green suits, and (3) overhead flags and the fence; these might reflect the variations in the subject's attention or interpretations of that image. Eyetracking data can be further examined to study attention's 's effect generated images.

---

It is challenging to perform one-to-one comparisons with previous deep image reconstruction works since the images in the MS-COCO dataset have much higher complexities than artificial shapes, faces, or images containing single centered object (like in ImageNet). We show results from a few best models for reconstructing images from fMRI in fig. Sa There also a recent survey [ covering more models and results if readers are interested. As our dataset is different, we search for similar images in the NSD validation set and show our generations in fig. 5b Compared to other methods, our pipeline can generate more photo-realistic images that reflect objects' shapes and backgrounds well. also utilizes more voxel activities than previous works (15724 voxels versus a few hundred). More importantly. it is able to reconstruct the relationships of different components when the images more complex. As natural scenes around us are rarely isolated objects and always information- we think reconstructing images through semantic alignment and conditioning is more beneficial and realistic than focusing on lower-level visual features.
Microstimulation In neuroscience, microstimulation refers to the electrical current- excita- tion of neurons and is used to identify the functional significance of a population of neurons. Here, we "microstimuulte" " the input fMRI signals of voxels in different brain ROIs, aiming to identify the roles of individual regions. In the NSD) dataset, there are four floc (functional localizer) experiments targeting regions responsible for faces, bodies, places, and words. A typical standardized fMRI signal has value range around [-4, 4). For the experiment, we locate the corresponding task-specific voxels based on ROI masks and increase the voxel activities to 10 while keeping the activities in unrelated voxels unchanged (see appendix A.3 for visual results). We observe the emergence of bodies or words when we increase the voxel activities in "bodies" or "words" ROIs. For voxels in "places" ROIs, elevating the signals will result in mesh-like patterns in the background, and this is true across different images. For "faces" ROIs, the generated images under elevated facial area signals seem to contain many small repeated pattems//ertuurattions. Interestingly, this appears to result from FFA (fusiform face area) signal changes since increasing only OFA (occipital face area) regions' activity does not result in similar patterns. Overall, increasing a specific task ROI's signal across fMRI samples results in CLIP embedding changes in similar positions. This means the disentangled space of CLIP embedding aligns well with how the human brain processes visual cues.
Apart from task-specific ROIs, we also changed brain region activities based on their roles in the visual processing hierarchy. We use the streams mask in the dataset to identify early visual cortex ROIs, intermediate ROIs, and higher-level ROIs. We then zero out voxels at each level. Our observations are: (1) when silencing the early visual cortex, objects and the whole scene prone to be in dull colors, and objects tend to have sharp shapes. Mcanwhile, the mapped embedding in the CLIP space will constantly have a lower value at almost all positions compared to mapped from unchanged signals (2) Silencing the higher-level ROIs has the opposite effect: more colors, more shapes, and crowded scenes. This is reasonable since the lower-level visual regions will bring up all the details when they lack high-level control. This time, the embeddings in the CLIP space have values consistently higher than normal. Finally, (3) silencing the intermediate ROIS seems have the least visual impact or CLIP embedding changes among the three. When compared to random controls (masking the same number of voxels random locations), we found that masking random voxels has minimal effect on mapped CLIP vectors and reconstructed images, but masking early/higher-level visual cortex levels change the results drastically. We performed the above microstimulation experiments on our pipeline with existing ROIs; however, it is potentially helpful for testing new ROI definitions and hypotheses.
3.4 CLIP space as the intermediary
In this section, we show that multimodal embedding space, particularly the CLIP space, is beneficial for brain signal decoding. To this end, we trained set of multi-label category classifiers to classify if a certain object category exists in the image based on the following inputs: (1) image-triggens fMRI Xtmai E 115724 (2) image CLIP embeddings hing E 5122 (3) CLIP embeddings mapped from image-triggered IMRI Kingg E 512 (4) image ResNet embeddings ResNet (Ximg) E 20 (obtained from Layer4, the final block before fully connected layers). All classifier models consist of 3 linear layers with ReLU activations between, and finish with a Sigmoid activation. For fMRI signals, we use (2048, 512) as the hidden dimension; for CLIP embeddings (setting (2) and (3)), we use (384, 256) as hidden dimensions; and for the ResNet embedding, we use (512, 256) as hidden dimensions. The final output covers 171 classes, including 80 things categories (bounded objects, like "person".
8

---

NJC-R00
from CUP
NHRI
from THBI mapped
from ResNen 50 embodding
from frrn macped messer embeeding
Category Index
Figure 6: Category-wise AUC-ROC of multi-lahel classifiers that predicts from five different signal embedding sources. The first categories are "things categories" the are "stuff categories" in COCO.
Table 2: Numerical AUC-ROC values of the classifiers presented in fig.
Performance w.r.t.
AUC-ROC
"things" categories
"stuff" categories
Overall
IMRI (%)
CLIP [MRI
0.9718 + 0.0266
0.8973 0.0639
0,9318
0.0624
112.36
0,8704 1 0.0557
0.7937 0.0824
0.8293 0.0807
100.00
IMRI-mapped CLIP
0.8468 1 0.0604
0.7817 1 0.0733
0.8119 0.0748
97.90
ResNet-50
0.7061 + 0.0736
0.7032 + 0.0719
0.7044 + 0.0725
84.94
IMRI-mapped ResNet-50
0,5410 = 0.1
0.5520 = 0.0941
0.5469 + 0.1020
65.95
"car"), and 91 stuff categories (mostly unbounded objects, like "tree", "snow") Binary cross-entropy loss is used for each class predict its existence in the input image.
Fig shows the category- wise AUC-ROC. The result demonstrates that CLIP embeddings contain the most object-level information about the image out of all the input sources. Following it, fMRI signals are also surprisingly very predictive, considering they carry
noise. The performance discrepancy between settings (2) and (3) is minimal, meaning mapping fMRI signals into the CLIP space retains most of the fMRI signals' information: this provides strong support for the validity of design. Lastly, ResNet embeddings perform poorly compared with other input sources. Therefore, even with a perfect mapping model, projecting IMRI signals into this space will lose information about the image since the expressiveness of the embedding is bounded by the lower performer. In addition, we note that both CLIP embeddings and fMRI have poorer performance on stuff categories than things categories, whereas ResNet embeddings do not. This indicate brain signals align better with the multimodal CLIP space than with single- modality ResNet space. Previous brain signal decoding work utilizing pre-trained generators all relied on image-only embedding spaces (ResNet-50 24. VGG19 351), ) and we believe moving to a multimodal latent space is a crucial step towards better brain signal decodings.
Further Discussions
Prior to our current pipeline design, we experimented with a DALL-E-like structure 61 since we can view the image reconstruction problem as oggnalto-signall translation. In particular, we applied VQVAE 400 on both IMRI and image to represent them as discrete latent codes and train Transformer model autoregressively generate text and image tokens from fMRI tokens. However, it was challenging to train the Transformer basedd model to converge with limited fMRI-image data. Incorporating the caption as text tokens to serve as the bottleneck between fMRI and image tokens while utilizing pre-trained models on the text and image modality did not help either. We think this suggests the need introduce a semantic medium to avoid direct translations between fMRI and image, as well as a solution to data scarcity.
We address both issues with the semantic space of CLIP embeddings. First, CLIP space is semantically informative and visually descriptive: for example, we can use image-text CLIP embedding alignment
Please refer to htwp://thh.ccaaiiiooccoooooooooo the full category list.

---

probabilities to screen captions. Mapping fMRI signals to representations in this latent space will retain rich information about the image that needs to be reconstructed. Second, the pre-training of the generative model can be separated entirely from fMRI data, meaning it can utilize much larger datasets than the one we use. However, there is a trade-off between generating a semantically similar scene and faithfully reconstructing each pixel. Although trained with additional contrastive loss targeting low-level visual features, the generated images by our pipeline are still leaning towards the former. We consider this reasonable choice since brains are more likely to perceive the image as whole rather than identifying each pixel, especially with multiple objects in the scene, Nevertheless, this results in worse reconstructions for images with fine details but less semantie, such single faces. The reconstruction of complex images with better aligned low-level visual features is worth further studies.
There are many more areas to explore. First, our study focuses on reconstructing a single subject's s brain signals. Applying the model to different subjects and observing the differences when generating the same image would be interesting. Since the data contains behavioral measures like valence and arousal towards each image, one can test if the generated images reflect personalized attention and perceptions. Second, other latent spaces can be examined. Although CLIP is one of the best-aligned computation models for the brain, other multimodal models like TSM 2 seem to have better alignment
[ with the visual cortex. In addition, other conditional generative models, such as diffusion models, can be explored. In particular, DALL-E2 2 [00 generates images conditioned on CLIP embeddings through diffusion, and it also provides an alternative solution to the differences exhibited in the image the text CLIP embeddings by learning a Prior model. Third, given the additional text modality, our pipeline opens up new opportunities to study visual imagery even without ground truth images. For example, one can either use mapping models trained given fMRI-image pairs and pre-trained generators to reconstruct imagined scenes, or study the mapping between brain signals and the text embeddings of the mental images' descriptions. Lastly, we focus on the decoding (brain-to-image) process, but the encoding (image-to-brain) process of complex images is equally important and exciting (we provide initial results on encoding in appendix A.9 additional future directions are discussed in appendix A.
With current brain signal recording devices, the negative social impact of this work is minimal: portable devices like EEG have poor spatial resolutions, making them unlikely to provide enough image-related details; On the other hand, fMRI scanners are used under highly controlled settings with designed procedures, therefore unlikely to have subject-unapproved privacy violations. However, when new devices that can address these issues become readily available, regulations would be needed on collecting and inspecting user data, since they potentially reveal sensitive information that users are unwilling to share through neural decoding. With pre-trained components, the pipeline may also misinterpret brain signals or be hacked to generate from manipulated inputs (no matter how unlikely it is) and produce over-confident false reconstructions because of the training data distributions. Several tricks may alleviate this issue, for example, training an input discriminator and placing before the entire pipeline to filter out suspicious inputs. Or, using a parallel pipeline targeting pixcl-level reconstruction as a check: if the two systems agree with each other above a certain thresholddconffidence the reconstruction results are accepted, otherwise discarded. Future pipeline improvements should also focus on exploring high- figh-performing models pre-trained on large (thus more generalizable) and unbiased datasets.
Conclusion
The paper proposes a pipeline to reconstruct complex images observed by subjects from their brain signals. With more objects and relationships presented in the image, we bring in an additional text modality to better capture the semantics. To achieve high performance with limited data, we utilize pre-trained semantic space that aligns visual and text modalities. We first encode fMRI signals to this visual- languag latent space and use a generative model conditioned on the mapped embeddings to reconstruct the images. We also introduce additional contrastive loss to incorporate low-level visual features into this semantic-basse based pipeline. As a result, the reconstructed images by our method are both photo-realistic and, most of the time, can faithfully reflect the image content. This brain signal to image decoding pipeline opens new opportunities to study human brain functions through strategic input alterations and can even potentially be helpful for numan-brain interfaces.
10

---

Acknowledgments and Disclosure of Funding
This project was partially supported by National Science Foundation under I1S-1817046 and HDR DSC 1924205.
References
[1] Y. Akamatsu, R. Harakawa, T. Ogawa, and M. Haseyama. Perceived image decoding from brain activity using shared information of multi multi-subject fmri data. IEEE Access, 9:26593 26666,, 2021.
[2] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelović, J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[3] E. J. Allen, G. St-Yves, Y. Wu, J. L. Breedlove, S. Prince, L. T. Dowdle, M. Nau. B. Caron, F. Pestilli,
I. Charest, et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-122 2022.
[4] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variance- caaamc--iiaaraanc--ovaraaa regularization for self- supervised learning. arXiv preprint arXiv:2105.0440 2021.
[5] P. Bashivan, K. Kar, and J. J. DiCarlo. Neural population control via deep image synthesis. Science, xea(6439) 2019.
[6] R. Beliy, G. Gaziv, A. Hoogi, F. Strappini, T. Golan, and M. Irani. From voxels to pixels and back: Self supervuuerv in natural-image reconstruction from fimri. Advances in Neural Information Processing Systems, 32, 2019.
[7] Y. Cao, C. Summerfield, H. Park, B. L. Giordano, and C. Kayser. Causal inference in the multisensory brain. Neuron, 10265)1077-1087, 2019.
[8] N. Chang, J. A. Pyles. A. Gupta, M. J. Tarr, and E. M. Aminoff. BOLD5000: A public fMRI dataset of 5000 images. arXiv preprint 1rXiv: 0088008992...//8 2018.
[9] B. Choksi, M. Mozafari, R. Vanrullen, and L. Reddy. Multimodal neural networks better explain multivoxel patterns in the hippocampus. In Neural Information Processing Systems (NeurIPS) conference: 3rd Workshop on Shared Visual Representations Human and Machine Intelligence (SVRHM 2021), 2021.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[11] T. Fang, Y. Qi, and G. Pan. Reconstructing perceptive images from brain activity by shape-semantic gan. Advances Neural Information Processing Systems, 33:13038 -3048, 2020.
[12] G. Gaziv, R. Beliy, N. Granot, A. Hoogi, Strappini, T. Golan, and M. Irani. Self supervisdd natural image reconstruction and rich semantic classification from brain activity. bioRxiv, 2020.
[13] A. A. Ghazanfar and C. F. Schroeder. Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10(6))278-285, 2006.
[14] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[15] T. Horikawa and Y. Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):1-15, 2017.
[16] J. Jeong and J. Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint 1xXiv22003.0942. 2021.
[17] C. Jia, Y. Yang. Y. Xia. Y.-T. Chen, Z. Parekh, H. Pham, Q Le, Y.H.. Sung, Z. Li, and T. Duerig, Scaling up visual and vision- language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[18] M. Kang and J. Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems, 33:21357-21169, 2020.
[19] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020.
[20] T. Karras, S. Laine, M. Aittala, J. Hellsten. J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
11

---

[21] S. Lin, T. C. Sprague, and A. Singh. Redundancy and dependency in brain activities. In SVRHM 2022 Workshop@ NeurIPS.
[22] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755 755. Springer, 2014.
[23] Z. Liu, P. Luo, X. Wang. and X. Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August. 15,2018):1,, 2018.
[24] M. Mozafari, L. Reddy, and R. VanRullen. Reconstructing natural scenes from fMRI patterns using bigbigan. In 2020 International joint conference on neural networks (IJCNN), pages 1-8, IEEE, 2020.
[25] A. Pasqualotto, M. L. Dumitru, and A. Myachykov. Multisensory integration: Brain, body, and world. Frontiers in Psychology, 6:2046, 2016.
[26] P. Pictrini, M. L. Furey, F. Ricciardi, M. I. Gobbini, W.-H. C. Wu, L. Cohen, M. Guazzelli, and J. V. Haxby. Beyond sensory images: Object-baset representation in the human ventral pathway. Proceedings of the National Academy of Sciences, 1115555558--566, 2004.
[27] S. F. Popham, A. G. Huth, N. Y. Bilenko, F. Deniz, J. S. Gao, A. O. Nunez-Elizalde, and J. L.. Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24(1 (4(11))1622) 1):1628 1636, 2021.
[28] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8766 PMLR, 2021.
[29] Z. Rakhimberdina, Q. Jodelet, X. Liu, and T. Murata. Natural image reconstruction from fMRI using deep learning: A survey. Frontiers in neuroscience, 15:795488 2021.
[30] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[31] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[32] Z. Ren, J. X. Xue, Li, F. Yang. Z. Jiao, and X. Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228:1 17602, 2021.
[33] P. Sharma, N. Ding, S. Gondman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt- text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018
[34] G. Shen, K. Dwivedi, K. Majima, T. Horikawa, and Y. Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience, page 21, 2019.
[35] G. Shen, Horikawa, K. Majima, and Y. Kamitani. Deep image reconstruction from human brain activity. PLoS computational biology, 15(1)=e10066)) 2019.
[36] Y. Shen, J. Gu, X. Tang, and B. Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243 224-9225,, 2020.
[37] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[38] S. Takada, R. Togo, T. Ogawa, and M. Hascyama. Question answering for estimation of seen image contents from multi-subject firi responses. In 2020 IEEE 9th Global Conference on Consumer Electronics (GCCE), pages 712-713. IEEE, 2020.
[39] A. Van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints. pages arXiv 2018.
[40] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[41] R. VanRullen and L. Reddy. Reconstructing faces from IMRI patterns using deep generative neural networks. Communications biology, 2(1):1110, 2019.
[42] Y. Zhou, R. Zhang. C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun. Lafite: Towards language-fiv free training for text-to-image generation. arXiv preprint arXiv:2111.1399. 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See section 4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix A.2
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See tables and 6
in appendix A.4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section 3.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See section 3.3
(b) Did you mention the license of the assets? [No] The code and the data are proprietary.
(c) Did you include any new assets either the supplemental material or as URL? Yes]] Additional codes are provided in the supplemental material.
(d) Did you discuss whether and how consent
obtained from people whose data you're using/curating? [No] The dataset paper containing the details is cited.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] The dataset paper containing the details is cited.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "mMnp-A6rtzi", "reviewer": "Reviewer_X8Bs", "summary": "This paper proposed a method for reconstructing real-world complex images with both naturalness and fidelity using multimodalities: fMRI, images, and text. The authors encode the fMRI signal into a pre-trained visual language latent space and reconstruct the image using a generative model conditioned on the embeddings. A microstimulation study is performed on different brain ROIs for interpreting human brain functions.", "questions": "Please see Limitations.", "limitations": "More ablation studies should be listed, for example with/without ROI extraction, and different architectures of mapping model fm.\n\nThe authors state 'the generated images emphasize either image content or semantic features depending on which condition we use', more explanation and examples should be done here.\n\nThe regularity and variability of using different individual fMRI signals for image reconstruction should be validated and explained.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe evaluation and analysis are relatively comprehensive,  covering different proposed design components and several hyperparameter settings in both training stages.\nA good implementation of reconstructing real-world complex images with both naturalness and fidelity from brain activities via fMRI.\n\nWeakness: The models and methods used in this paper are reasonable, but each part of them is known in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Iv8lzAIVvxo", "reviewer": "Reviewer_as89", "summary": "The authors propose a new method to reconstruct image stimuli from fMRI data. They propose to rely on the CLIP model, and map fMRI signal to the CLIP latent space. Subsequently, they fine-tune a generative model using mapped embeddings so as to reconstruct the seen image. \nExperiments show that the approach is able to reconstruct high quality images with high semantic similarity with the target image. \nThe paper has a several limitations and presentation issues, but overall proposes an original approach that could be of interest to the community. ", "questions": "-Have you considered the DALL-E 2 model? It combines CLIP with diffusion models and could be a good extension to the proposed approach. \n\n-Have you considered adding a pixel-level loss function to generate images closer to the target image? \n\n-What is the impact of using only the CLIP text or image embedding? Why maxpool values, instead of e.g concatenating? ", "limitations": "Authors provide a good discussion on limitation. Considering the topic studied in this work, I would recommend to add a discussion on societal impact, and the implications of being able to predict what the brain sees from fMRI data. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n-The idea to rely on large pre-trained models to address the data scarcity inherent to fMRI data is a very interesting idea. The CLIP model is very powerful, and leveraging its high representation power for fMRI is a clever solution. \n-The generated images are of high quality, with high semantic similarity with the target image. \n-Authors provide an honest and detailed discussion on limitations, including discussion on previous negative results. \n\n**Weaknesses**\n\n-My first issue relates to presentation of the work. Firstly, the paper leverages a lot of diverse concepts and methods (fMRI, CLIP, Lafite model), and assumes a lot of prior knowledge on the reader regarding how these methods work. It would make it difficult for a reader not familiar with the models to understand the methodology.  \nOne key presentation issue is the excessive reliance on supplementary materials. Most of the quantitative results are in the appendix, and some results are directly referred to in the main manuscript. The manuscript in its current form is not self contained. I strongly recommend to rearrange the paper so that this is not an issue anymore. \nThe paper doesn't include a related work section, which is essential to properly relate the work to the literature. \n\n-The main limitation, which authors are openly discussing, is that the approach does not aim to reconstruct the image, but generate a semantically similar image. While this is interesting and the results on the few examples provided look quite good, this might not be sufficient if certain image details matter (e.g. colours). \n\n-Another issue is the fact that the work is heavily engineered: it comprises a large set of tunable parameters and losses (e.g. 7-8 loss parameters for clip mapping, 6 losses and dedicated tuning parameters for the GAN part), CLIP embedding values are clipped to an arbitrary value, captions are selected based on a threshold, etc..Providing evidence that these parameters do not crucially affect performance would be appreciated. \n\n-The evaluation is mainly qualitative, with limited examples and comparisons to pre-existing approaches. It would have been good to provide additional qualitative examples in the supplementary materials to demonstrate that the shown examples were not cherry picked. For quantitative results, authors propose a new evaluation metric because of a limitation of their approach (ie focus on semantics rather than reproducing the original image). \nSeveral conclusions are made on the basis on a couple qualitative examples (e.g. semantic closeness between mismatches, microstimulation experiments), again, it would be beneficial to provide more evidence to support these claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DogrN4o4-RO", "reviewer": "Reviewer_cvSx", "summary": "In this paper, the authors propose a new method to reconstruct the images from the corresponding fMRI signals by leveraging the text descriptions of the input images and using that as a training signal for the model. The fMRI data for this task is limited and therefore this paper addresses the data scarcity problem by leveraging an aligned vision-language latent space that is pretrained on large-scale datasets. fMRI signals are then aligned to this latent space thus leading to reconstructions that are photo-realistic and capture the semantic contents of the groundtruth image.", "questions": "My main concern about this work is its comparison with previous works. \n1. It lacks quantitative comparison with any of the proposed framework \n2. Qualitative comparison is not fair", "limitations": "Limitations and Negative impact section is missing. I have addressed key limitations in the weaknesses section. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The proposed idea to leverage the latent space of a multimodal model (CLIP) in reconstruction from fMRI signal is new and interesting.\n2.  The method is evaluated with clear ablations and relevant metrics thus making the experimental validation of the proposed framework very strong.\n3. The paper is well written with a clear description of the method, caption selection and different components of the model used. \n\n### Weakness\n1. The comparison to previous work is not fair. Previous methods were trained using a smaller different dataset while the model in the present work was trained using a higher SNR dataset. Therefore, whether the proposed method is better than any of the previous works can not be inferred based on this work and only on qualitative results in Figure 5\n2. The photorealism comes from the pretrained generator and the previous work[22] which uses BigBiGAN's latent space can also generate photorealistic results. Therefore, a key claim that it is possible to generate photorealistic images using the proposed framework is misleading.\n3. There is no quantitative comparison with any of the previous methods. At least one or two strong baselines are easy to implement and should be compared with e.g. [22] or an end-to-end GAN[32] to figure out how competitive is the proposed approach against these baselines.\n4. Microstimulation results are not convincing.The behavior of images generated is interesting but there is no quantifiable result to interpret what exactly is happening there.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5HnLld8P0Ct", "reviewer": "Reviewer_WHmz", "summary": "The authors develop a pipeline to reconstruct natural scene images from fMRI data by aligning neural representations to the CLIP embedding space and then using a generative model conditioned on the CLIP space to reconstruct photo-realistic images.", "questions": "By utilizing the pretrained CLIP embedding space as an intermediary, and the pretrained GAN for image generation, brain signals are coerced to adhere to the space of natural statistics learned by these models from their original training sets. This makes it completely unsurprising that photo-realistic images are generated for natural scenes, but leaves the reader wondering what would happen if participants were presented with images that differed from the types of images the pretrained pipeline components have been trained on. How adversarial might these images be? One might expect that the pipeline would continue to generate highly photo-realistic images from whichever part of the training set is most aligned, but these will bear little truth to what the participants actually saw. Do the authors care to comment on this? The image reconstruction examples that the authors choose to show on planes highlight this concern, where spurious details are added to or removed from the reconstructed images, altering the veracity of the images, while assigning unrealistically high confidence to the reconstructions due to their high resolution. In many aspects, on a pixel-level, prior methods are more accurate reconstructions, and don’t create this hyper-realistic false reality. The microstimulation experiments section also highlights this concern, where images are highly degraded from ground-truth in many dimensions that are completely unrelated to the expected region, but still coerced into an overly confident final image. The veracity of resulting images and their proximity to ground truth might be best evaluated using human raters, but I understand that the time involved to crowd-source these ratings is out of scope for the review period.", "limitations": "As noted in the previous section, there are considerable concerns to this approach, and the authors neglect discussion of these topics entirely. The authors also fail to address any negative societal implications. By using pretrained models, highly photo-realistic reconstructed images are coerced to fit the natural statistics of the original training sets in ways that are poorly understood, with little regard for the true content of images, and could likely be manipulated to produce offensive or biased images, while attributing those images to the neural signals of the human participant.\n\nOverall, this paper is incredibly creative and well-designed from an engineering perspective, and has incredible potential to be accepted, but needs to be placed into perspective, the limitations section needs to be drastically expanded, and examples should be included of model failure modes, not just best-case scenario reconstructions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very creative idea, and an interesting approach to generating photo-realistic images from fMRI data despite data scarcity. The proto stimulation and ablation experiment provides particularly fascinating insights into the strengths and weaknesses of this approach, and is both a welcome and exciting addition. In addition, the engineering aspects of this paper are excellent and well-designed. The level of detail is appreciated.\n\nThat being said, there are some pretty major fundamental concerns with this approach, in particular the constraints placed on images by utilizing pretrained model components, which the authors entirely neglect to address. Considerable more discussion is required by the authors on this topic, and the associated limitations, and I raise a few specific questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept