Language decoding from human brain activity via contrastive learning

Authors: matteo  ferrante
nicola  toschi
alexander  huth
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-20 11:02:11

Abstract: We propose a novel contrastive learning approach to decode brain activity into sentences by mapping fMRI recordings and text embeddings into a shared representational space. Using data from three subjects, we trained a cross-subject fMRI encoder and demonstrated effective sentence identification with a retrieval module. Our model shows strong alignment between brain activity and linguistic features, with top-1 accuracy up to 49.2\% and top-10 accuracy up to 84\%, significantly outperforming chance levels. These results highlight the potential of contrastive learning for cross-subject language decoding,

Main paper: 
 introduction: 
 Language is one of the most ubiquitous ways we experience the flow of information in our daily lives. We read, speak, communicate, and even think through language. It is a complex phenomenon that allows us to understand and convey information to others. To do this, our brain generates semantic representations of everything we encounter, taking context into account. Recent research has demonstrated a convergence between brain activity [3, 2, 9] during language tasks, such as listening or reading, and the way large language models process sentences. This has been shown through brain recordings using both non-invasive methods, such as fMRI, EEG, and MEG, as well as invasive techniques like LFP and ECoG. Powerful encoding and decoding models have been developed to link external stimuli, such as acoustically perceived sentences, images, videos, music with neural representations recorded during these tasks [19, 1, 6, 17, 4, 18, 13, 15, 7, 8, 14, 16] . These models have shown remarkable results across both invasive and non-invasive brain recording techniques. Here, we propose a novel method based on contrastive learning to learn a cross-subject fMRI encoder that projects fMRI recordings and text embeddings into a shared space, enabling sentence identification with a retrieval module. Our model takes as input fMRI activity and computes fMRI embeddings that can be compared with pre-computed sentence embeddings belonging to a set of candidates. By selecting the closest sentences in this space we can effectively decode the brain activity and obtain clues about semantic representation in the brain. Fig. 1 show a scheme of the pipeline proposed here. This appendix describes the architecture of the neural networks used in this work. The models consist of an Encoder for processing the input data, an Embedding Pooler for projecting embeddings to a common latent space, and a Contrastive Model for learning via contrastive losses. 
data: 
 In our analysis, we used the publicly available dataset from [12] . We focused on subjects S1, S2, and S3, each of whom underwent fMRI recordings for approximately 16 hours while listening to 83 stories from The Moth and Modern Love podcasts. The fMRI data was collected using a 3T Siemens Skyra scanner with a repetition time (TR) of 2.00 seconds and an isotropic voxel size of 2.6 mm. This allowed for the measurement of BOLD signals, which reflect neural activity with an inherent delay due to the hemodynamic response. Preprocessing steps included motion correction, cross-run alignment, standardization and detrending of low-frequency. For more details, we refer the reader to the original paper [12] . Data from the first 70 stories of each subject were used as training set, while the other 12 stories were used as validation set. Also, the story "wheretheressmoke", which was listened to 10 times to ensure better signal to noise ratio at test set, aligning with recent literature on language encoding and decoding [19, 1] .The first step of our pipeline involves reducing the complexity of the signal we need to process. While much of the brain is active during language and semantic tasks [10, 11] , certain regions are more directly involved in language processing. Therefore, we begin by identifying cortical regions that, at the voxels level, are more easily modeled by a language model, allowing us to focus our analysis on these regions. We trained an encoding model of brain activity using a large language model (LLM) as the foundation. For each word in the training stories, we computed embeddings from LLama3-8B [5] , specifically from the 13th layer, using a context window of the previous five words. This choice of layer is supported by previous studies [1] that found early layers in LLama models exhibit stronger correlations with brain activity. To align the word embeddings with the fMRI temporal resolution, we downsampled them using a Lanczos filter, creating matching sentence-fMRI pairs. Finally, we calculated the Pearson correlation between the predicted and true brain activity on held-out validation data, selecting the top 10,000 cortical voxels as our target regions. The activity in these voxels are the targets we aim to embed and decode.The core contribution of this work lies in the proposed representation learning pipeline. Since a single TR of fMRI recording can be influenced by several preceding words due to the hemodynamic response (HRF), we must account for this effect. To address the unique properties of fMRI data, we developed a model consisting of two neural networks: an fMRI encoder and a text embedding pooler. The fMRI model is a cross-subject neural network. The first layer is subject-specific and projects the top 10,000 acivity corresponding to the top 10000 voxels of each subject into a shared representation space of dimensionality common_dim. The remaining layers of the network are shared across subjects, consisting of a multi-layer perceptron (MLP) that transforms the data from common_dim to the final latent dimension latent_dim. The text embedding pooler is an MLP that processes four downsampled word embeddings corresponding to the TRs that might affect the measured brain activity (i.e. those from 1, 2, 3, and 4 timepoints before the corresponding BOLD response). Each word embedding is projected into the final latent space by a linear layer that reduces the dimensionality. These projections are then concatenated, resulting in a final sentence representation of dimensionality latent_dim. In appendix, we detail all the hyperparameter search and architectures. Let x represent the fMRI activity, y the downsampled word embeddings, f the fMRI model, and g the text pooler. We define the projections as: z x = f (x), z y = g(y). Let z x ∈ R n×d represent the encoded fMRI features and z y ∈ R n×d represent the encoded text embeddings, where n is the batch size and d is the latent dimensionality. The logits matrix, which contains the similarity scores between each pair of fMRI and text embeddings, is computed as:logits ij = z xi • z ⊤ yj τwhere τ is the temperature parameter that controls the sharpness of the distribution. The pairwise similarities are computed using the dot product between the fMRI and text embeddings:similarities ij = z xi • z ⊤ yj .Let the target labels be the identity mapping, where each input is matched with itself in the contrastive learning task. The targets vector t ∈ N n is defined as: t = {0, 1, . . . , n -1} This implies that t i = i, ensuring each fMRI sample is matched with the corresponding text sample. The contrastive loss L is computed as a combination of two cross-entropy losses, one for the alignment of fMRI to text and the other for text to fMRI:L = 1 2 L CE (logits, t) + L CE (logits ⊤ , t)where L CE denotes the cross-entropy loss function. This formulation optimizes both directions in the contrastive learning objective, ensuring that fMRI features are closely aligned with the corresponding text embeddings and vice versa. This contrastive learning approach ensures that both fMRI activity and the corresponding text embeddings are projected into the same latent space, aligning brain activity with linguistic features.The retrieval module compares the L2 distance between fMRI and text embeddings across the test set to find the closest sentences to each fMRI sample. We evaluate the decoded sentences using three metrics: identification accuracy, top-1 accuracy, and top-10 accuracy. For top-10 retrieval, we select the 10 closest sentences to each fMRI TR based on L2 distance, with the target sentence defined as the previous 4 TRs (8 seconds) plus 5 preceding context words. Identification accuracy, adapted from vision and music decoding literature, measures how well the model identifies the correct sentence by comparing self-correlations in the latent space with other correlations. We compute Pearson correlations between the predicted vectors and targets, storing the results in a correlation matrix, and successively calculate identification accuracy by comparing the self-correlation with others in the same row and normalizing the result. 
results: 
 The results from our contrastive learning-based language decoding model are shown in Figure 2 and Table 3 . The top panel of Figure 2 illustrates the model's encoding performance, with Pearson  
discussion and conclusions: 
 Contrastive learning has proven to be a robust method for learning cross-subject mappings between brain activity and sentence-level embeddings. However, a key limitation of our approach is that decoding is performed through a retrieval module (i.e., sentence identification). This requires access to candidate sentences beforehand, limiting the model's ability to generalize to brain activity related to sentences that differ significantly from those in the training dataset. Another important limitation pertains to future applications of this work. The learned fMRI embeddings could potentially be used in conjunction with Bayesian decoding techniques or as inputs to modified large language models (LLMs) for open vocabulary decoding. However, these approaches raise concerns about privacy and bias. It will be crucial to address how to distinguish between actual thoughts and brain representations, and how to prevent biases in both models and human interpretations from influencing the results. Future research should explore the concept of neural privacy and develop strategies to disentangle model biases from genuine cognitive processes. In conclusion, this work presents a cross-subject architecture that decodes brain activity into sentences using contrastive learning and sentence identification, laying the groundwork for future advancements in brain-to-language decoding.A Appendix / supplemental material A. 
 
 Appendix: The Encoder network is designed to map the input data to a lower-dimensional latent space. The architecture is summarized in Table 2 . input_dim is set to be the number of cortical voxels (10000) while common_dim is chose to be 4096. The alignment layers apply different transformations to subsets of the data depending on key values provided at runtime. These layers are followed by sequential linear layers with ReLU activation.The Embedding Pooler projects embeddings, such as fMRI data, into a lower-dimensional latent space. The architecture is summarized in Table 3 . The input is first normalized, followed by a sequence of linear layers and GELU activation. The final step reshapes the output into a pooled embedding vector.The Contrastive Model combines the Encoder and Embedding Pooler for multimodal learning using contrastive loss functions. The model supports various loss functions, such as contrastive loss, mean squared error (MSE), and cosine similarity loss. The overall architecture is shown in Table 4 . During training, the model minimizes the distance between fMRI projections and embeddings from other modalities. The training process is logged and monitored using the mean squared error, cosine similarity, and the primary loss function.The models are trained using an AdamW optimizer with a learning rate of 1e -4 and weight decay of 1e -4. The learning rate is adjusted dynamically using a scheduler that reduces the rate by a factor of 0.1 when the validation loss plateaus for 50 epochs. The models are trained for a maximum of 3 epochs using PyTorch Lightning's Trainer, with all computations performed on a single GPU device. The data are publicly available and can be requested at https://openneuro.org/datasets/ds003020/. All experiments and models were trained on a server equipped with four NVIDIA A100 GPU cards (80GB RAM each connected through NVLINK) and 2 TB of System RAM.In this section, we detail the connections and differences between the generative decoding approach described in [19] and our proposed contrastive learning decoder. The aforementioned work employs a Bayesian encoding technique, learning subject-specific encoding and noise models from data, and estimates likelihood probabilities using a large language model (LLM) as a generator of candidate sentences. This enables open-vocabulary text generation guided by fMRI data. In contrast, ourIn our contrastive model, we employ two learned functions, approximated by neural networks, to map x and y into a shared latent space z of dimensionality d:z x = f (x), z y = g(y)where f : R f → R d and g : R v → R d .To simplify the mathematical analysis and highlight the differences and similarities between the two methods, let's assume that both f and g are linear functions:z x = xA, z y = yBwith A being a matrix of shape (f, d) and B a matrix of shape (v, d), so that z x and z y both have shape (t, d).The objective of the contrastive loss is to make the cosine similarity matrix between z x and z y as close as possible to the identity matrix I. This can be achieved by computing:S = z x ∥z x ∥ z y ∥z y ∥ ⊤where S has shape (t, t). We can then use a cross-entropy loss for each element along the diagonal or compute the mean squared error between S and I.By expanding the calculation and ignoring the normalization terms for simplicity, we obtain:S = z x z ⊤ y = xAB ⊤ y ⊤where the matrix multiplications involve shapes:(t, f ) × (f , d) × (d, v) × (v, t)We observe a key connection between the two models: in both cases, we have a similarity (or dissimilarity) matrix of shape (t, t) where the interaction between x and y plays a crucial role.In the Bayesian model, the encoding projects text features into the brain space and re-weights them based on the noise model, whereas in the contrastive model, this process occurs implicitly through the interaction of the functions f and g. In the linear case, the product AB ⊤ takes on a role analogous to W Σ -1 . However, in the contrastive approach, text features are projected into a latent space (typically with d ≪ v), resulting in a less descriptive model. This acts as an implicit regularization, but if d is less than the rank of Σ -1 , some information might be lost. This is a suggestion that higher dimensionality plays an important role in brain decoding of language, guiding us in our hyperparameter search. Thus, while the contrastive model offers greater flexibility-such as training cross-subject models, incorporating nonlinearities, and utilizing compressed latent spaces-it may sacrifice some information about the relationship between x and y. A potential future direction could involve modeling z y directly as a multivariate Gaussian:z y ∼ N (f (x), Z σ )where Z σ could capture the noise properties of the latent space, possibly modeled by another neural network.To optimize the performance of our contrastive learning-based model for decoding brain activity, we conducted a hyperparameter search using a random sampling methodology. Our search focused on minimizing the validation loss across a set of 100 randomly sampled configurations from a predefined search space. The hyperparameter sweep was configured as follows:• Batch Size (BS): We experimented with batch sizes of {512, 1024, 2048}.• Learning Rate (lr): We explored two learning rates: {1e-4, 1e-5}.• Alpha (α): The weight parameter for the contrastive loss was sampled from {0.5, 0.8}. The metric used to evaluate the model's performance was the validation loss, which we aimed to minimize. The random sampling method allowed us to explore a diverse set of hyperparameter combinations without performing an exhaustive grid search, which would be computationally expensive.By systematically varying these key hyperparameters and evaluating each sampled configuration, we were able to identify an optimal combination that balanced both accuracy and generalization across subjects. This hyperparameter search played a crucial role in achieving the strong performance metrics reported in our results.Accordingly to insights given from comparative analysis we found that linear models with high common and latent dim (equal to the input dimensionality) performed better.The development of models that decode brain activity into language raises important ethical concerns, particularly regarding bias, privacy, and the responsible use of such technology. One significant concern is the potential for bias in both the models and the data. fMRI datasets, as well as the pre-trained language models used in this study, can inherit biases from the populations they are trained on, which may lead to biased or inaccurate decoding, especially across diverse groups of individuals. This could have serious implications when applying these models in clinical or social contexts.Another critical issue is privacy. Brain-to-text decoding systems pose a unique risk to neural privacy, as they may allow for the reconstruction of internal thoughts and mental states. This raises questions about consent, data security, and the misuse of brain data in contexts where individuals may not have full control over how their neural activity is used or interpreted. It is essential to develop safeguards to ensure that brain data cannot be used to decode private thoughts without explicit consent, and to explore ways to mitigate any unintended consequences of decoding technologies, such as the risk of surveillance or the exploitation of individuals' cognitive data.As the field progresses, it will be crucial to establish ethical guidelines that prioritize transparency, fairness, and respect for individual autonomy. Future research should also focus on developing methods to disentangle model biases from genuine cognitive processes and explore the concept of "neural privacy" as a framework for protecting individuals in this emerging area of brain-computer interface technology.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: We would like to begin by thanking the authors and reviewers for their valuable contributions and efforts. In the following, we will provide the rationale behind the decision.

The paper presents a contrastive learning approach for training a cross-subject fMRI encoder to decode sentences, with results indicating performance above chance levels and some potential for understanding brain-based semantic representations. However, reviewers identified several significant weaknesses: the method shows limited novelty, closely following approaches like CLIP and BrainCLIP, and lacks rigorous comparisons to state-of-the-art baselines. The study's small dataset (three subjects) raises concerns about generalizability, and there is limited analysis of the learned representations, reducing the empirical contributions. Additionally, reviewers noted that the claims about the paper's contributions were overstated, and recognition of prior work was insufficient.

Despite these issues, acceptance is suggested because the extended abstract track aims to support early-stage research which are fit in the workshop topics, and presenting at the poster session provides an excellent opportunity for the authors to engage in constructive discussions and refine their work. However, it is strongly recommended that the authors address the reviewers' concern in the camera ready version of the paper,, particularly by recognizing prior literature more clearly and incorporating the suggested feedback, to strengthen the paper’s impact and future development.

Reviews: review: title: title: Promising decoding results; can we interpret the representations?
main_review: main_review: The authors train a cross-subject fMRI encoder and show successful sentence identification using a retrieval module. The model significantly outperforms chance levels, suggesting that the learned representations may capture some useful information about how semantic representations are organized in the brain, and how these representations may differ between subjects. 

Despite the promising decoding results, there are some shortcomings of the paper. In general, for a 4 page format, there is too much space allocated to Methods and not enough space for the Results. For example, I feel that Figure 2 does not add much to the paper, and that space could have been used for more interesting analyses that actually seek to interpret the learned representation spaces to understand the features that are supporting good retrieval. There are many analyses (dim reduction, clustering of the cosine similarity matrices, etc) that could shed light on the underlying representational dimensions. Punting more methods detail to the Supplement could have created room for these sorts of analyses. As a field, we must not be satisfied with mere prediction scores - the authors' modeling paradigm seems ideally suited to translate good decoding into some new ideas about how the information is actually organized in the brain.

That said, because the methodology appears sound overall, and the paper is well motivated, I will recommend acceptance.  UniReps seems like a good venue for the authors to receive feedback on this work.
title: title: This paper uses contrastive learning to present a cross-subject model for decoding language from fMRI brain activity. While the approach shows some promise in sentence identification, it primarily builds on existing methods without significant novel contributions.
main_review: main_review: Quality:
- The paper provides a detailed description of the model architecture and training process.
- The evaluation metrics (Top-1, Top-10, and Identification Accuracy) are appropriate for the task.
- However, the paper needs a more rigorous comparison with state-of-the-art methods.
- The small sample size (3 subjects) limits the generalizability of the results.
- The paper needs to address potential overfitting concerns adequately.

Clarity:
- The overall structure of the paper is logical and easy to follow.
- The methodology section is detailed, with helpful diagrams and tables.
- The appendix provides comprehensive information on neural network architectures and training setup.

Originality:
- The core idea of using contrastive learning for aligning fMRI and language data is not novel, as similar approaches have been explored in previous works such as BrainCLIP and MindEye.
- The cross-subject aspect of the model is interesting but needs to be sufficiently developed and validated.

Significance:
- The reported performance metrics show improvement over chance levels, but the practical significance of these improvements needs to be well-established.
- The lack of comparison with a broader range of state-of-the-art methods limits assessing the model's true impact.

Pros:
1. Interesting and relevant problem statement.
2. Detailed methodology and appendix enhancing reproducibility.
3. Consideration of ethical implications and privacy concerns.
4. The model does show performance above chance levels.

Cons:
1. The claim of novelty in using contrastive learning for this task is overstated, as similar approaches of aligning into a common shared space via contrastive learning have been used before in the field.
2. The paper needs to sufficiently explore different language models or fMRI preprocessing techniques, limiting the scope of the work.
3. The qualitative comparative analysis with the Bayesian approach is exciting but must demonstrate the proposed method's superiority conclusively through quantification.
title: title: The authors propose a method to align fMRI samples with corresponding sentences by training them on the CLIP loss. However, at this stage, the manuscript lacks significant novelty and insight, so I recommend rejection.
main_review: main_review: The authors propose a method to align fMRI samples with corresponding sentences by training them with a contrastive loss. However, at this stage, the manuscript lacks significant novelty and insight, so I recommend rejection.

Strengths
- The paper is easy to understand, and the results indicate that some useful learning does take place.
- Pearson correlation maps show a consistently higher correlation among the same areas, which indicates language-related cortical areas.

Weaknesses                 : 
- The novelty of the paper is very limited: it is just one ingredient from the paper BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding (https://arxiv.org/abs/2302.12971), where they align text and image samples with fMRI recordings.
title: title: Novel application; however, the method is not novel and without proper citation; limited data and scope.
main_review: main_review: This paper presents a contrastive learning approach to decode sentences from fMRI brain activity by mapping neural recordings and text embeddings into a shared representational space. While the results appear promising, with reported top-1 accuracy of up to 49.2% and top-10 accuracy of up to 84%, several critical weaknesses must be discussed and addressed.

The primary weakness is that the method itself is not particularly novel. While the authors state that "the core contribution of this work lies in the proposed representation learning pipeline," the approach appears to be a modified version of the contrastive learning framework CLIP introduced by Radford et al. (2021) in their paper "Learning Transferable Visual Models From Natural Language Supervision." The framework presented in Figure 1 reveals that the approach is a relatively straightforward adaptation of existing contrastive learning principles.

Moreover, despite the high similarity between this work and Radford et al.'s framework, the authors neither cite the original work nor adequately address the differences or provide appropriate credit. The authors essentially apply this established technique to brain decoding without significant methodological innovations and without acknowledging the foundational contributions of the earlier work.

Another concern is that the experimental evaluation is notably limited. The results are presented only for three subjects (Table 1), without any comparison to previous work or baseline methods. This lack of comparative analysis makes it impossible to assess the true advancement of their approach over existing techniques in the field. The absence of baselines, ablation studies, or comparative experiments significantly weakens the paper's empirical contribution.

The study relies on data from only three subjects. This limited sample size raises questions about the generalizability of the results and the robustness of the model when applied to a broader population. The authors do not adequately address this limitation or discuss how their findings might scale to larger, more diverse datasets.

In addition, as the authors have pointed out, a key limitation of the proposed approach is that their "decoding is performed through a retrieval module (i.e., sentence identification). This requires access to candidate sentences beforehand, limiting the model’s ability to generalize to brain activity related to sentences that differ significantly from those in the training dataset."

In conclusion, while the paper presents an interesting application of contrastive learning to brain decoding, it represents an incremental advance rather than a significant breakthrough in the field. Future work would benefit from larger sample sizes and more innovative methodological contributions.

Reference:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML 2021).