Title: Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural decoding

Mind Reader

Visual Reconstruction

Multimodal Large Model

Concept Localization
Abstract: Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a *Vision Transformer 3D*. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LLMs). Additionally, we have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language-based concepts within brain signals, enhancing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding in neuroscience and human-computer interaction, setting the stage for advanced brain-computer interfaces and cognitive models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework that integrates 3D brain structures with visual semantics using a Vision Transformer 3D. By aligning fMRI features with multiple levels of visual embeddings, it eliminates the need for subject-specific models and allows extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with LLMs. The topic is intriguing, and the proposed method offers practical applications.\n\nMajor concerns and minor comments include:\n\n1. Benchmarking with Real-World Datasets: As a novel machine learning approach, the performance of the proposed method should be benchmarked with more real-world fMRI datasets to evaluate the generalizability of the results. The current study may be limited in scope or sample size, and using a wider variety of datasets will help demonstrate the robustness and scalability of the approach across different subjects and conditions. It would be beneficial to include datasets with diverse characteristics, such as different brain regions, tasks, and populations, to ensure comprehensive evaluation.\n\n2. Discussion on Advantages and Disadvantages: The author(s) should discuss the advantages and disadvantages of the proposed method in the field of neuroscience and brain decoding. This discussion should include a comparison with existing approaches, highlighting the unique contributions and potential limitations of the new method. Additionally, insights into the interpretability and explainability of the proposed method would be valuable. For instance, how does this method enhance our understanding of brain activity patterns? Are there any trade-offs between model complexity and interpretability? Addressing these questions will provide a clearer picture of the method’s potential impact and areas for improvement.\n\n3. Details on Cross-Validation: Please provide more details about the cross-validation used in the empirical studies, perhaps in Section 4. It is essential to specify the type of cross-validation technique employed (e.g., k-fold, leave-one-out) and the rationale behind its selection. Detailed information on the partitioning of the data, the number of folds, and any stratification strategies used will help in understanding the robustness of the validation process. Additionally, discussing the metrics used for evaluation and how they were computed across different folds will add clarity to the reported results.\n\n4. Pseudocode for the Proposed Method: The proposed method can be summarized in the form of pseudocode (algorithm). Providing a step-by-step algorithmic representation will make the methodology more transparent and easier to reproduce. The pseudocode should outline the key steps involved in data preprocessing, feature extraction, model training, and inference. Including comments within the pseudocode to explain the purpose of each step and any critical hyperparameters or configurations will further enhance understanding.\n\n5. Minor Typos and Grammar Mistakes: There are some minor typos and grammatical mistakes throughout the paper. A thorough proofreading and editing process is recommended to improve the overall readability and professionalism of the manuscript.\n\nSTRENGTHS:\nPlease see the Summary section\n\nWEAKNESSES:\nPlease see the Summary section\n\nQUESTIONS:\nPlease see the Summary section\n\nLIMITATIONS:\nPlease see the Summary section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an innovative framework that leverages Vision Transformer 3D (ViT3D) to integrate 3D brain structures with visual semantics, enhancing visual reconstruction and language interaction from fMRI data. By aligning fMRI features with visual embeddings through a unified feature extractor and integrating with LLMs, the framework enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. The framework demonstrates exceptional performance and holds significant potential for applications in neuroscience and human-computer interaction.\n\nSTRENGTHS:\nProvides a multi-task visual neural decoding framework that is simpler and more elegant compared to previous methods, offering a robust example for integrating visual and neural signals with unified modeling. Introduces a method for precise localization of language-based concepts within brain signals simply using the GradCAM method.\n\nWEAKNESSES:\nThe Dual-Stream fMRI Feature Extractor is already a common method in the field, but the combination of the two features appears suboptimal. The authors thoroughly discuss the trade-off between low-level and high-level features. However, based on the results in fig 4, the reconstructed images’ low-level features are inferior to previous methods.\n\nQUESTIONS:\nI am curious if the method of combining low-level and high-level features in eq4 is the most effective. While other works also use this approach, it seems to force the model to trade off between the two types of features. Are there potentially better methods?\nIn visual reconstruction, the method with LLM appears effective only when the beta value is low. Does it overlap in function with the CLIP embedding?\n\nLIMITATIONS:\nThe authors discuss the limitations well in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new, subject-agnostic visual reconstruction pipeline. They introduce a way to integrate across fMRI readings from different subjects and enhance their integration using LLMs. Through this integration they see a consistent improvement of high level semantic feature baselines in their reconstruction.\n\nSTRENGTHS:\n* An interesting new approach to align fMRI feature extraction with ViTs and LLMs\n* Brain captioning with LLMs is highly novel. The results are promising in Table 1.\n* Promising results on NSD reconstruction.\n* Interesting set of analyses using GradCAM.\n\nWEAKNESSES:\n* Concerns about preprocessing and claims about cross-subject analysis\n    * The authors propose a trilinear interpolation to combine BOLD signals across patches across subjects. To me, this makes many assumptions that may not be true or might miss several problems. The authors need to address these questions to substantiate the claim of removing subject-specific modeling or using some kind of mixed subject pretraining to combine subjects. \n    * For example, this method ignores inter-subject variability such as anatomical differences or functional differences, which can vary among individual subjects. \n    * The interpolation makes a linear assumption on BOLD responses which may not be true. I realize this will follow with alignment to a ViT later but this imposes strong assumptions about BOLD signal space. \n    * My main concern here is whether this kind of preprocessing will generalize to modeling over new subjects. Is this approach applicable to general subject populations? Could the authors add some clarification and expand on this point?\n* LLM Interaction details\n    * My understanding is that there is a finetuning objective when using the LLM interaction. I think section 3.4 was cut off somehow and this wasn’t very clear to me.\n    * Some details were missing making Table 1 a bit difficult to understand. See questions\n\nQUESTIONS:\n* What does fine-tuned model refer to in line 204? Is it the LLM (just making sure)\n* When you use BLIP2 generated captions, are you passing the original NSD image and generating a caption? Is this in comparison to your LLM that uses an fMRI embedding in place of [image]? I think this was confusing. \n* Although GradCAM is easily interpretable, I do wonder if this would benefit from something more recent for interpretability like [1]. \n* Could MindEye2 (https://medarc-ai.github.io/mindeye2/) be used as a baseline? \n\n[1] Materzynska et. al. Disentangling visual and written concepts in CLIP. CVPR 2022.\n\nLIMITATIONS:\n* Limitations are addressed but may want to consider toning down some of the claims about subject variability (see weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This model focuses on the task of reconstructing image stimuli from fMRI readings\n- Instead of training a subject specific model, a subject-generic model is trained\n- The model is built around a pre-trained LLM core, which is finetuned to take the fMRI as input and then:\n  - engage in dialogue about the image stimulus\n  - reconstruct the image stimuli by generating a language prompt which is then passed to UnCLIP\n  - create a heatmap of activations in the brain associated with a given concept\n\nSTRENGTHS:\n- The multi-subject aspect of the training appears novel\n- The use of an LLM for image reconstruction and dialogue about the stimulus also appears novel\n- The image reconstructions are of comparable or better quality than existing approaches\n- Ultimately, I think the evaluation could be improved (see Weaknesses), but given the novelty of the approach (using an LLM to do decoding and reconstruction), I argue for acceptance.\n\nWEAKNESSES:\n- It's unclear how much the image reconstructions reflect visual processing in the brain. It would be good to do an ablation to see how much the reconstruction depends on $\\hat{z}_c$. What if $\\hat{z}_c$ were replaced with random noise in the same way that an ablation was done on $\\hat{z}_v$? Basically, I am curious about how much decoding can be done from the prompt $a_r$ only.\n- In the same vein, it would be interesting to do an experiment where the generated prompt $a_r$ is given to a generative model that does not require $z_c$ and $z_v$, i.e., a model that can take a text prompt and generate an image.\n- It's claimed that the model can engage in complex dialogue about the semantic content of the stimuli, but from the example prompts, it seems that many of the answers to the dialogue questions may be available in the prompt. For example, figure 11 seems to suggest that many of the complex reasoning prompts contain enough information to answer the question in the prompt. E.g. ``What is the distribution of the cows in the field based on the description and their positions in the image.'' A good ablation test would be to perform the complex reasoning task without the fMRI inputs.\n- Some technical details are hard to follow, especially with regards to concept localization (see questions).\n\nQUESTIONS:\n- line 293 --- how are concepts \"extracted\"?\n- line \"Finally, patches containing non-task-relevant information are filtered out\" -- how are the masked portions chosen?\n- line 256-258: I think there is a typo somewhere. I'm not sure what the sentence is supposed to say.\n- Fig 1: From the figure, it seems that one can query the LM for concept localization in the brain? But where is that described? What do annotations for that dataset look like?\n- A related question: line 533: what does it mean to localize the results using GradCAM? Can you describe how that was done?\n- Fig 5 and 6: Equation (4) says that $\\beta=1$ corresponds with all noise input. But increasing noise seems to correspond with better image quality?\n- Line 484: Typo: \"Language Extension\"\n- Table 2: what is `C_Subj`?\n\nLIMITATIONS:\nLimitations and potential negative societal impact are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}