Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a new method to reconstruct the images from the corresponding fMRI signals by leveraging the text descriptions of the input images and using that as a training signal for the model. The fMRI data for this task is limited and therefore this paper addresses the data scarcity problem by leveraging an aligned vision-language latent space that is pretrained on large-scale datasets. fMRI signals are then aligned to this latent space thus leading to reconstructions that are photo-realistic and capture the semantic contents of the groundtruth image.
strengths_and_weaknesses: ### Strength
1. The proposed idea to leverage the latent space of a multimodal model (CLIP) in reconstruction from fMRI signal is new and interesting.
2.  The method is evaluated with clear ablations and relevant metrics thus making the experimental validation of the proposed framework very strong.
3. The paper is well written with a clear description of the method, caption selection and different components of the model used. 

### Weakness
1. The comparison to previous work is not fair. Previous methods were trained using a smaller different dataset while the model in the present work was trained using a higher SNR dataset. Therefore, whether the proposed method is better than any of the previous works can not be inferred based on this work and only on qualitative results in Figure 5
2. The photorealism comes from the pretrained generator and the previous work[22] which uses BigBiGAN's latent space can also generate photorealistic results. Therefore, a key claim that it is possible to generate photorealistic images using the proposed framework is misleading.
3. There is no quantitative comparison with any of the previous methods. At least one or two strong baselines are easy to implement and should be compared with e.g. [22] or an end-to-end GAN[32] to figure out how competitive is the proposed approach against these baselines.
4. Microstimulation results are not convincing.The behavior of images generated is interesting but there is no quantifiable result to interpret what exactly is happening there.
questions: My main concern about this work is its comparison with previous works. 
1. It lacks quantitative comparison with any of the proposed framework 
2. Qualitative comparison is not fair
limitations: Limitations and Negative impact section is missing. I have addressed key limitations in the weaknesses section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a method for reconstructing real-world complex images with both naturalness and fidelity using multimodalities: fMRI, images, and text. The authors encode the fMRI signal into a pre-trained visual language latent space and reconstruct the image using a generative model conditioned on the embeddings. A microstimulation study is performed on different brain ROIs for interpreting human brain functions.
strengths_and_weaknesses: Strengths: 
The evaluation and analysis are relatively comprehensive,  covering different proposed design components and several hyperparameter settings in both training stages.
A good implementation of reconstructing real-world complex images with both naturalness and fidelity from brain activities via fMRI.

Weakness: The models and methods used in this paper are reasonable, but each part of them is known in the field.
questions: Please see Limitations.
limitations: More ablation studies should be listed, for example with/without ROI extraction, and different architectures of mapping model fm.

The authors state 'the generated images emphasize either image content or semantic features depending on which condition we use', more explanation and examples should be done here.

The regularity and variability of using different individual fMRI signals for image reconstruction should be validated and explained.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new method to reconstruct image stimuli from fMRI data. They propose to rely on the CLIP model, and map fMRI signal to the CLIP latent space. Subsequently, they fine-tune a generative model using mapped embeddings so as to reconstruct the seen image. 
Experiments show that the approach is able to reconstruct high quality images with high semantic similarity with the target image. 
The paper has a several limitations and presentation issues, but overall proposes an original approach that could be of interest to the community. 
strengths_and_weaknesses: **Strengths**

-The idea to rely on large pre-trained models to address the data scarcity inherent to fMRI data is a very interesting idea. The CLIP model is very powerful, and leveraging its high representation power for fMRI is a clever solution. 
-The generated images are of high quality, with high semantic similarity with the target image. 
-Authors provide an honest and detailed discussion on limitations, including discussion on previous negative results. 

**Weaknesses**

-My first issue relates to presentation of the work. Firstly, the paper leverages a lot of diverse concepts and methods (fMRI, CLIP, Lafite model), and assumes a lot of prior knowledge on the reader regarding how these methods work. It would make it difficult for a reader not familiar with the models to understand the methodology.  
One key presentation issue is the excessive reliance on supplementary materials. Most of the quantitative results are in the appendix, and some results are directly referred to in the main manuscript. The manuscript in its current form is not self contained. I strongly recommend to rearrange the paper so that this is not an issue anymore. 
The paper doesn't include a related work section, which is essential to properly relate the work to the literature. 

-The main limitation, which authors are openly discussing, is that the approach does not aim to reconstruct the image, but generate a semantically similar image. While this is interesting and the results on the few examples provided look quite good, this might not be sufficient if certain image details matter (e.g. colours). 

-Another issue is the fact that the work is heavily engineered: it comprises a large set of tunable parameters and losses (e.g. 7-8 loss parameters for clip mapping, 6 losses and dedicated tuning parameters for the GAN part), CLIP embedding values are clipped to an arbitrary value, captions are selected based on a threshold, etc..Providing evidence that these parameters do not crucially affect performance would be appreciated. 

-The evaluation is mainly qualitative, with limited examples and comparisons to pre-existing approaches. It would have been good to provide additional qualitative examples in the supplementary materials to demonstrate that the shown examples were not cherry picked. For quantitative results, authors propose a new evaluation metric because of a limitation of their approach (ie focus on semantics rather than reproducing the original image). 
Several conclusions are made on the basis on a couple qualitative examples (e.g. semantic closeness between mismatches, microstimulation experiments), again, it would be beneficial to provide more evidence to support these claims.
questions: -Have you considered the DALL-E 2 model? It combines CLIP with diffusion models and could be a good extension to the proposed approach. 

-Have you considered adding a pixel-level loss function to generate images closer to the target image? 

-What is the impact of using only the CLIP text or image embedding? Why maxpool values, instead of e.g concatenating? 
limitations: Authors provide a good discussion on limitation. Considering the topic studied in this work, I would recommend to add a discussion on societal impact, and the implications of being able to predict what the brain sees from fMRI data.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors develop a pipeline to reconstruct natural scene images from fMRI data by aligning neural representations to the CLIP embedding space and then using a generative model conditioned on the CLIP space to reconstruct photo-realistic images.
strengths_and_weaknesses: This is a very creative idea, and an interesting approach to generating photo-realistic images from fMRI data despite data scarcity. The proto stimulation and ablation experiment provides particularly fascinating insights into the strengths and weaknesses of this approach, and is both a welcome and exciting addition. In addition, the engineering aspects of this paper are excellent and well-designed. The level of detail is appreciated.

That being said, there are some pretty major fundamental concerns with this approach, in particular the constraints placed on images by utilizing pretrained model components, which the authors entirely neglect to address. Considerable more discussion is required by the authors on this topic, and the associated limitations, and I raise a few specific questions below.

questions: By utilizing the pretrained CLIP embedding space as an intermediary, and the pretrained GAN for image generation, brain signals are coerced to adhere to the space of natural statistics learned by these models from their original training sets. This makes it completely unsurprising that photo-realistic images are generated for natural scenes, but leaves the reader wondering what would happen if participants were presented with images that differed from the types of images the pretrained pipeline components have been trained on. How adversarial might these images be? One might expect that the pipeline would continue to generate highly photo-realistic images from whichever part of the training set is most aligned, but these will bear little truth to what the participants actually saw. Do the authors care to comment on this? The image reconstruction examples that the authors choose to show on planes highlight this concern, where spurious details are added to or removed from the reconstructed images, altering the veracity of the images, while assigning unrealistically high confidence to the reconstructions due to their high resolution. In many aspects, on a pixel-level, prior methods are more accurate reconstructions, and don’t create this hyper-realistic false reality. The microstimulation experiments section also highlights this concern, where images are highly degraded from ground-truth in many dimensions that are completely unrelated to the expected region, but still coerced into an overly confident final image. The veracity of resulting images and their proximity to ground truth might be best evaluated using human raters, but I understand that the time involved to crowd-source these ratings is out of scope for the review period.
limitations: As noted in the previous section, there are considerable concerns to this approach, and the authors neglect discussion of these topics entirely. The authors also fail to address any negative societal implications. By using pretrained models, highly photo-realistic reconstructed images are coerced to fit the natural statistics of the original training sets in ways that are poorly understood, with little regard for the true content of images, and could likely be manipulated to produce offensive or biased images, while attributing those images to the neural signals of the human participant.

Overall, this paper is incredibly creative and well-designed from an engineering perspective, and has incredible potential to be accepted, but needs to be placed into perspective, the limitations section needs to be drastically expanded, and examples should be included of model failure modes, not just best-case scenario reconstructions.
review_title: null

Meta-review: metareview: In this paper, the authors propose a new method to reconstruct the images from fMRI signals by leveraging the text descriptions of the input images and using that as a training signal for the model. The fMRI data for this task is limited and therefore this paper addresses the data scarcity problem by leveraging an aligned vision-language latent space that is pretrained on large-scale datasets. fMRI signals are then aligned to this latent space, leading to reconstructions that are photo-realistic and capture the semantic content of the ground truth image.

The reviewers agreed that the proposed idea to leverage the latent space of a multimodal model (CLIP) in reconstruction from fMRI signal is new and interesting. At the same time, they voiced concerns about some of the evaluation metrics and baselines. Over the course of the rebuttal, many of the reviewer concerns were addressed (at least mostly) and reviewers increased their scores. In the end, all of the reviewers agreed in favor of acceptance.

confidence: Certain
Decision: Accept