Title: Brain-Inspired fMRI-to-Text Decoding via Incremental and Wrap-Up Language Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: fMRI-to-text

brain decoding

sequential decoding

text-guided mask
Abstract: Decoding natural language text from non-invasive brain signals, such as functional magnetic resonance imaging (fMRI), remains a central challenge in brain-computer interface research. While recent advances in large language models (LLMs) have enabled open-vocabulary fMRI-to-text decoding, existing frameworks typically process the entire fMRI sequence in a single step, leading to performance degradation when handling long input sequences due to memory overload and semantic drift. To address this limitation, we propose a brain-inspired sequential fMRI-to-text decoding framework that mimics the human cognitive strategy of segmented and inductive language processing. Specifically, we divide long fMRI time series into consecutive segments aligned with optimal language comprehension length. Each segment is decoded incrementally, followed by a wrap-up mechanism that summarizes the semantic content and incorporates it as prior knowledge into subsequent decoding steps. This sequence-wise approach alleviates memory burden and ensures semantic continuity across segments. In addition, we introduce a text-guided masking strategy integrated with a masked autoencoder (MAE) framework for fMRI representation learning. This method leverages attention distributions over key semantic tokens to selectively mask the corresponding fMRI time points, and employs MAE to guide the model toward focusing on neural activity at semantically salient moments, thereby enhancing the capability of fMRI embeddings to represent textual information. Experimental results on the two datasets demonstrate that our method significantly outperforms state-of-the-art approaches, with performance gains increasing as decoding length grows.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method for fMRI-to-text decoding, where they first pretrain a brain encoder on a large-scale fMRI dataset with a self-supervised MAE-style objective, then finetune the model on the target fMRI-to-text paired dataset. In particular, they introduce two mechanisms inspired by human language processing: segmental and incremental processing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Good conceptual contributions of the text-guided masking strategy for MAE fMRI pretraining, and incremental, segmented fMRI-to-text decoding. \n2. Their proposed method outperforms prior methods (UniCoRN, EEG-Text, BP-GPT) on all quantitative metrics evaluated (BLEU, ROUGE, BERTScore).\n\nWeaknesses: \n1. I think the paper would benefit from more qualitative results (see Question 3 below) to understand how the proposed innovations improve fMRI decoding, and how close their method is to \"solving\" the fMRI decoding problem.\n2. I am interested in understanding the impact of their proposed sequential decoding framework (incremental and segmental processing) as that seems to be a key difference between this method and simply using a standard transformer architecture. However, this is challenging to understand empirically without a direct ablation. Referring to Table 2, first row, they ablate it when pretraining and text-guided masking are absent. But it would be most ideal, if possible, if there were an ablation where pretraining and text-guided masking are present, but not the sequential decoding strategy. This might be an important ablation to understand if adding structure (e.g., through their proposed sequential decoding method) is still helpful and important when there is sufficiently large pretraining data.\n\nQUESTIONS:\nQuestions:\n1. fMRI-to-text is like translation in NLP between two languages, where transformers succeed without explicitly adding segmental or incremental processing mechanisms. I am curious to understand why the authors think these will specifically benefit the problem of fMRI-to-text. Perhaps adding some elaboration would be helpful.\n2. For Table 1, it seems that 60 > 40 > 20 TRs. I would be interested to see if increasing the number of TRs beyond 60 would lead to higher BLEU, ROUGE, and BERTScore?\n3. What does the decoded text actually look like? Beyond the BLEU/ROUGH/etc. scores, it would be helpful to show some examples. For example, showing the original text, and different generations of decoded text under various random seeds. Also, perhaps it would be useful to qualitatively contrast this against generations from prior methods, so that it would be easier to understand the ways in which the proposed method makes an advance over prior methods.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CogReader, a brain-inspired framework for fMRI-to-text decoding that mimics human cognitive strategies for language comprehension. The key innovation is a sequential decoding approach that divides long fMRI sequences into segments, performs incremental decoding on each segment, and uses a wrap-up mechanism to maintain semantic continuity. Additionally, the authors introduce a text-guided masking strategy for fMRI representation learning that leverages BERT attention scores to identify semantically important timepoints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework meaningfully incorporates human language processing mechanisms into computational design, which is well-motivated and represents a principled approach.\n2. The experimental design is comprehensive with proper baselines, thorough ablation studies, and validation against noise data.\n\nWeaknesses:\n1. The method is only evaluated on one dataset, which significantly limits generalizability claims. \n2. While cognitively inspired, the paper lacks deeper theoretical analysis of why segmented processing should outperform holistic approaches. \n3. The wrap-up mechanism (BERT + MLP) and attention score aggregation seem overly simplistic given the complexity involved.\n\nQUESTIONS:\n1. Can you provide evaluation on additional datasets to support broader applicability? Even preliminary results would strengthen the paper significantly.\n2. What happens with dynamic segment length based on text complexity rather than fixed length? How do you analyze the trade-off between local and global context?\n3. Have you experimented with more sophisticated semantic integration beyond BERT+MLP? How sensitive is performance to the wrap-up mechanism design?\n4. Can you provide theoretical analysis of why segmented processing should outperform holistic approaches?\n5. How does the method perform when text-guided masking isn't available during training?\n\nLIMITATIONS:\nThe authors adequately address some limitations in the discussion, particularly the static segmentation strategy and single dataset evaluation. However, they could better discuss: (1) potential negative effects of losing global context through segmentation, (2) computational overhead of the sequential approach, and (3) applicability constraints when paired training data isn't available.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for decoding fMRI to text using a segmented approach inspired by how humans process language to mange the load in the working memory. Instead of decoding the full fMRI sequence at once, the input is split into segments that are decoded incrementally. After each segment, there is a “wrap-up” phase that integrates the current semantic understanding and passes it forward to inform the next decoding step, helping maintain coherence across segments. In addition to this sequential decoding setup, the authors introduce a two-stage pretraining scheme for learning better fMRI representations. First, they use a large-scale dataset for general pretraining. Then, they propose a text-guided masked autoencoder (MAE) that leverages attention scores from the associated text to guide which fMRI time points to mask and reconstruct—essentially forcing the model to focus on semantically important parts. Experiments on the Narratives dataset show that their approach outperforms existing methods, especially for longer sequences. In addition, the paper shows ablation studies to support the design decisions including the advantage of using sequential decoding, pretraining + text-guided pretraining, studies on the segment length, and various other experiments like testing the model with random data.\n\nSTRENGTHS AND WEAKNESSES:\n##  Strengths:\n\n1. Text-guided masking pretraining: The paper introduces a novel text-guided masking strategy during pretraining. By using attention scores from the text to guide which fMRI timepoints to reconstruct, the model learns to focus on semantically important tokens. This is an intersting idea that ties textual semantics directly to fMRI representation learning.\n2. Sequential, segment-based decoding: Inspired by human language processing, the model decodes fMRI inputs in segments, with a wrap-up phase that integrates prior context. This helps preserve semantic flow across longer sequences and is a step forward compared to one-shot decoding approaches.\n3. Thorough experimental validation: The authors present extensive ablation studies and comparisons against existing methods. Their approach consistently outperforms baselines, especially for longer fMRI sequences.\n\n## Weaknesses:\n\n1. Limited contextual integration in wrap-up phase: While the paper claims to model human-like wrap-up effects, the implementation appears to simply pass the sentence embedding of the previous segment (through an MLP) into the decoder for the next segment. This means the model only conditions on the immediately preceding segment, not the full history, which weakens the claim of human-like contextual integration. The current explanation of the wrap-up phase is a bit vague and it would be valuable to clarify this mechanism and test a version where more cumulative context is aggregated. For example, it's unclear how the wrap-up embedding is merely summed with the fMRI embedding (the only information available is the symbol ⊕ symbol in the figure, but it is nowhere described whether those embeddings are simply added).\n2. Potentially unfair comparison to prior work: The paper shows that performance of prior models degrades with longer sequences, but this comparison may be unfair. Earlier methods (e.g., UniCoRN) are not designed to decode \\textit{long} sequences at once, and in the original paper it is presented by decoding signals of a few TRs (in fact looking at UniCoRN paper they provide a table showing that the length T=10 works best for their experimental setting). In other words, if UniCoRN works better by first segmenting, and then decoding each segment independently, than that is a better baseline instead of using UniCoRN with a very long sequence which might suffer due to the big context window. Additionally, some strong results from prior work—such as BLEU scores around 34.77% reported in UniCoRN—are not directly reconciled with the numbers presented here. It is important to use the setup for which UniCoRN is designed for and compare to this approach. \n3. Text-guided masking is not self-supervised: Although the pretraining approach is compelling, it depends on access to paired text to compute attention scores. This limits its applicability as a general-purpose pretraining strategy. Moreover, this stage is performed on the same dataset used for evaluation, raising concerns about generalization and potential data leakage. It is not clearly stated whether the same splits were used across pretraining and decoding to avoid this issue. If not carefully separated, this could significantly inflate performance.\n4. Fixed segmentation scheme: Segmenting the fMRI time series into fixed-length chunks may lead to cuts at unnatural points in the narrative, disrupting the underlying semantic coherence. Exploring adaptive or linguistically informed segmentation (e.g., based on sentence or clause boundaries) could improve results.\n5. Unclear role of wrap-up in actual fMRI decoding: It's unclear whether the observed performance gain from the wrap-up phase stems from better modeling of brain activity or simply from smoother language generation. As an ablation, it would be useful to decode segments independently (as in UniCoRN), and then apply a language model post-hoc to enforce consistency, isolating the contribution of the fMRI model from the decoding pipeline.\n6. The model is only evaluated at a single public dataset. Although fMRI-to-Text are difficult datasets to get, there are still other publically available datasets where one can validate the approach.\n\nQUESTIONS:\nSee weakness, but here are the most important questions:\n1. How are the splits done for the text-guided pretraining and decoding. Looking briefly at the supplementary material I can see the splits done at 60, 20, 20, however, can you please confirm whether the exact same splits are used for both stages (to prevent pretraining on data that is used for evaluating the decoding process).\n2. Can you clarify why in UniCoRN they report 34% in BLEU score, whereas in this paper the number is different. \n\nA minor thing:\n3. In line 296, instead of \"time series of 20 TRs\", was it meant to be written \"time series of 60 TRs\"?\n\nLIMITATIONS:\nThe paper mentions already some critical limitations like the fixed segment size, as well as evaluation in a single public dataset. However, in addition, it is important to mention that even the text-guided pretraining is strictly speaking supervised and as such it is limited in applying at other datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes \"CogReader,\" a new framework for decoding natural language text from fMRI signals, specifically designed to handle long sequences. The authors identify that existing methods, which process an entire fMRI sequence at once, suffer from performance degradation on longer inputs. To address this, their method is inspired by human language comprehension and involves two main contributions. First, it employs a sequential decoding strategy where the fMRI time series is divided into fixed-length segments. Each segment is decoded incrementally, and a \"wrap-up\" mechanism summarizes the output and passes it as a semantic prior to the next segment to maintain continuity. Second, for learning fMRI representations, the paper introduces a text-guided masking strategy within a Masked Autoencoder (MAE) framework, which aims to focus the model on semantically important time points. Experiments on the Narratives dataset show that the proposed method outperforms state-of-the-art approaches, with the performance gap widening as the length of the decoded text increases\n\nSTRENGTHS AND WEAKNESSES:\nWeaknesses\n\n1. The paper's claim to novelty is somewhat overstated. The general framework—pre-training an fMRI encoder on a large dataset like HCP , then fine-tuning it to produce embeddings that are fed into a conditional generation model or LLM (like BART )—is a standard paradigm in both visual and language decoding from fMRI. This approach has been widely used in prior work such as [1-3] which are not cited, including the cited UniCoRN paper.  This means the primary novelty rests almost entirely on the incremental decoding and \"wrap-up\" mechanism. While this is a reasonable contribution, it feels more like an incremental improvement on a common backbone rather than a fundamentally new framework.\n\n\n2.  The framing of the work as \"Human-Like Language Comprehension\" is not sufficiently justified. While inspired by cognitive science, the implementation uses fixed-length segmentation (N_s=20 TRs), which is more of an engineering heuristic to manage memory than a true model of human cognition. Human reading involves dynamic, content-aware segmentation, not arbitrary fixed-length chunking. The paper lacks a deeper discussion or evidence to support why this specific mechanism is a faithful model of human comprehension.\n\n\n3. A major weakness in evluation is the complete absence of qualitative examples of the decoded text in the main paper or appendix (the appendix is mentioned but not provided). Metrics like BLEU and ROUGE are known to be limited and can be misleading. Without seeing the samples actual generated sentences, it is diffcult to assess crucial aspects like semantic coherence, factual accuracy, and the specific types of errors the model makes compared to baselines. \n\n\nMissed references include but are not limited to:\n\n[1] Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. CVPR 2023\n\n[2] Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities. NeurIPS 2023\n\n[3] MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities. NAACL 2024\n\n\n\nStrengths\n\n1 The paper addresses a well-defined and significant problem in the field of neural decoding. The difficulty of decoding long, continuous language sequences from fMRI is a major bottleneck, and the authors propose a clear strategy to mitigate this issue.\n\n2 The quantitative results presented are strong. Table 1, in particular, makes a compelling case for the method's effectiveness, showing that while baseline performance degrades with longer sequences (from 20 TRs to 60 TRs), the proposed CogReader model shows marked improvement. The ablation studies also systematically validate the contribution of each component.\n\n\n3  The paper is generally well-written and structured. The core ideas are explained clearly, and the figures, especially Figure 2, provide a helpful illustration of the overall architecture and data flow\n\nQUESTIONS:\n1 The central framing of the paper is its \"human-like\" approach. However, the core mechanism is a fixed-length segmentation of the fMRI signal. Could you provide a stronger justification from cognitive science literature that supports fixed-length (as opposed to content-driven, adaptive) segmentation as a plausible model for human language comprehension? \n\n2 Given that the overall architecture (HCP pre-trained encoder + LLM decoder) is quite standard, could you more precisely delineate the novelty of your \"wrap-up\" mechanism compared to other methods that handle long sequences or context, such as hierarchical models or models with state-passing mechanisms? Clarifying this might help in assessing the originality of the contribution.\n\n3 The absence of qualitative examples makes it difficult to fully evaluate your model's output. Could you please provide a few representative examples of decoded text from your model and the baseline models for a 60 TR sequence? A discussion of the comparative error types (e.g., semantic drift, repetition, hallucination) would be valuable\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}