Brain-Inspired fMRI-to-Text Decoding via Incremental and Wrap-Up Language Modeling
Authors: Wentao Lu
Dong Nie
Pengcheng Xue
Zheng Cui
Piji Li
Daoqiang Zhang
Xuyun Wen
Venue: NeurIPS 2025 spotlight ()
Keywords: fMRI-to-text
brain decoding
sequential decoding
text-guided mask
Primary area: neuroscience_and_cognitive_science

Abstract: Decoding natural language text from non-invasive brain signals, such as functional magnetic resonance imaging (fMRI), remains a central challenge in brain-computer interface research. While recent advances in large language models (LLMs) have enabled open-vocabulary fMRI-to-text decoding, existing frameworks typically process the entire fMRI sequence in a single step, leading to performance degradation when handling long input sequences due to memory overload and semantic drift. To address this limitation, we propose a brain-inspired sequential fMRI-to-text decoding framework that mimics the human cognitive strategy of segmented and inductive language processing. Specifically, we divide long fMRI time series into consecutive segments aligned with optimal language comprehension length. Each segment is decoded incrementally, followed by a wrap-up mechanism that summarizes the semantic content and incorporates it as prior knowledge into subsequent decoding steps. This sequence-wise approach alleviates memory burden and ensures semantic continuity across segments. In addition, we introduce a text-guided masking strategy integrated with a masked autoencoder (MAE) framework for fMRI representation learning. This method leverages attention distributions over key semantic tokens to selectively mask the corresponding fMRI time points, and employs MAE to guide the model toward focusing on neural activity at semantically salient moments, thereby enhancing the capability of fMRI embeddings to represent textual information. Experimental results on the two datasets demonstrate that our method significantly outperforms state-of-the-art approaches, with performance gains increasing as decoding length grows.

Brain-Inspired fMRI-to-Text Decoding via Incremental and Wrap-Up Language Modeling
Wentao Lu', Dong Nie², Pengcheng Xue', Zheng Piji Li', Daoqiang Zhang', Xuyun Wen College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China
*ChatAlpha AI, California, USA
charless cwizheng, pjli, wegzhang wenxuyu)) edu. dongnie@cs edu
Abstract
Decoding natural language text from non-invasive brain signals, such as functional magnetic resonance imaging (fMRI, remains a central challenge brain-computer interface research. While recent advances in large language models (LLMs) have enabled open-vocabulary fMRI-to-text decoding, existing frameworks typically process the entire fMRI sequence in single step, leading performance degrada- tion when handling long input sequences due to memory overload and semantic drift. To address this limitation, we propose brain-inspired sequential IMRI-to- text decoding framework that mimics the human cognitive strategy of segmented and inductive language processing. Specifically, into consecutive segments aligned with optimal language comprehension length. Each segment is decoded incrementally, followed by wrap-up mechanism that summarizes the semantic content and incorporates it as prior knowledge into sub- sequent decoding steps. This sequence-wise wise approach alleviates memory burden and ensures semantic continuity across segments. In addition, we introduce a text- guided masking strategy integrated with masked autoencoder (MAE) framework for IMRI representation learning. This method leverages attention distributions over key semantic tokens selectively mask the corresponding IMRI time points, and employs MAE to guide the model toward focusing on neural activity at seman- tically salient moments, thereby enhancing the capability of fMRI embeddings represent textual information. Experimental results on the two datasets demon- strate that our method significantly outperforms state-of-the-art approaches, with performance gains increasing decoding length grows. The code is available at https:/ggthub..oo///ee.. WWN.YW/ENNNNOOGRGAAaCCC.
divide long fMRI time series
Introduction
Language serves as a window into cognitive processes, conveying vast amounts information through its syntactic and semantic structures [26). Advances in non-invasive neuroimaging, such as functional magnetic resonance imaging (fMRI, have enabled researchers to measure brain activity patterns associated with language processing. Translating cognitive signals into natural language not only deepens our understanding the neural basis of the language system, but also facilitates the development of practical brain-com crainccomputer interfaces (BCIs) by leveraging insights into the decoding
process 36 34
In recent years, advances in deep learning have led to significant progress in short text generation from brain signals, such as mapping fMRI activity to semantic representations of individual words or short
Coorresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(4) Pressous (MRi-ttotest Framework
(c) Language comprecccasion mechanism
IMRI Thme Serdin
DIRE Kmiseddng
Predicted Sentenee
Sqquenttl Praceniig
Dvender
It was a ralay day when Emmm....
Directly decude the cnttre DER tlase setles late teats in single ignoring murrrry load and cegnitive mechanieno when reading test.
(1) incerarrttll pr prasosiing separibed chastic (2) weap chusto underiland scatence
(b) EMRI-to-text Framework
DERT Time Serios
Segucite
DIRE
Prodkidd
Increnentel Prosourig
W
Once upen time, a village..
Braala
D
We chride the ENILL serbee lato groups, and incrementally decade each
Undcrstanding seeed-ly-.oe
group into seatennee fellowed by weep-ap
Figure I: Comparison of fMRI-to-tex decoding frameworks. (a) Existing frameworks directly decode the entire fMRI sequence corresponding to the target text in a single step. (b) Our proposed segment- based sequential decoding framework. (c) Cognitive mechanisms of human language comprehension, where incremental processing and segmental wrap-up operating parallel.
phrases 55] However, these approaches are limited in scope, often restricted to closed vocabularies or single-word word outputs, lacking the capacity to decode full natural language sentences. With the rapid development of large language models (LLM), researchers have begun to explore their application in
various brain decoding tasks [ (0) 22 29 4. Recent studies have shown encouraging progress in
open-vocabulary (MRI-to-text generation by incorporating large language models (LLMs) (37 3. However, these methods still face significant challenges in decoding long sequences. As illustrated in Figure a), most current approaches process the entire fMRI sequence corresponding in single step, overlooking the segmented and inductive processing strategy that the human brain adopts to manage memory load during language comprehension [14] [T]. As the length of the input increases, these approaches lead to excessive memory burden and semantic drift (25), ultimately impairing decoding performance. Unlike the traditional machine translation paradigm employed in existing methods, which enables one-to-one mapping between two independent modalities, fMRI- to-text decoding reconstructs textual content from neural activity patterns elicited during human language comprehension. Consequently, models tailored for cross-modal translation are not directly transferable to this task. Guided by this distinction, we hypothesize that a decoding framework better aligned with human language processing mechanisms would more effectively address this challenge. Therefore, essential to incorporate insights from human language comprehension mechanisms into the design of fMRI-to-text decoding models.
given text
Human language understanding is neither a passive nor a strictly linear process. Instead, it emerges from a complex interplay between incremental processing and segmental integration [30, as shown in Figure cc.. Incremental processing enables the brain to construct semantic representations in real time, interpreting linguistic input on a word-by-wool bwor basis. While this allows for immediate comprehension, it imposes a heavy load on working memory and becomes less effective for complex or long-form text. In contrast, segmental integration provides a complementary mechanism, wherein the brain periodically aggregates information across semantically coherent segments. This wrap-up process facilitates semantic consolidation and disambiguation at key structural boundaries, thereby reducing cognitive load and enhancing comprehension accuracy. Despite growing evidence supporting the importance of these dual mechanisms, they remain largely underexplored in existing IMRI-to-text decoding frameworks.
Inspired by human cognitive mechanisms for language processing, we propose novel IMRI-to-text decoding framework that combines incremental processing with a wrap-up-bast up-desed semantic integration strategy, named as CogReader. As shown in Figure (b), we first divide the continuous fMRI time series into multiple sequential segments. For each segment, the model performs incremental decoding, generating the corresponding text word by word in real time. We also design wrap-up integration module that summarizes the decoding results of the current segment into a semantic representation. This representation is then passed prior knowledge guide the decoding of the next segment,

---

enabling effective cross-segment information flow. Furthermore, to learn fMRI features containing more textual information, we introduce a text-guided masking strategy, integrated into Masked Autoencoder (MAE)-basee framework for fMRI representation learning. Our main contributions are summarized as follows:
1. Motivated by human language comprehension mechanisms, we design a new fMRi-to-text decoding framework that integrates incremental processing and wrap-up semantic integration. Our model enables real-time decoding for each segment and progressively incorporates cross-segment knowledge, offering effective solution for decoding long-form text from neural activity.
2. We propose text-guided masking strategy. By leveraging attention distributions over key semantic tokens, our method selectively masks corresponding fMRI time points and incorporates MAE to encourage the model to focus on neural activity at key time points to learn brain representations with more key textual information.
3. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-are approaches on standard fMRI-to-text decoding benchmarks. Moreover, the performance advantage becomes increasingly pronounced sentence length grows, under- scoring the feasibility and effectiveness of our cognitively inspired decoding framework.
Related Works
2.1 IMRI Representation Learning
Due to the complex spatiotemporal structure of IMRI data and the variability across subjects, learning robust and high-quality IMRI representations remains a significant challenge. In recent years, variety of deep learning paradigms have been proposed to improve IMRI representation quality. For
example, Kim al. [15] utilized a variational autoencoder (VAE) 17 to model the distribution of fMRI signals while disentangling spatial and temporal components. Asadi et al. [2] proposed a hybrid model that combines spatial attention with temporal Transformers, to better model long- rong-range spatiotemporal dependencies. While model architecture innovation has advanced fMRI representation learning, complementary learning paradigm improvements have emerged as another research focus. In fMRl-to-text decoding tasks, fully supervised learning has become standard for enhancing semantic richness of neural representations [3]. Building on this foundation, contrastive learning frame- works 3 further optimize cross- modal alignment by treating paired IMRI-text data as positive samples. However, the reliance on scarce paired datasets limits these supervised approaches. To mitigate this bottleneck, recent work integrates self-supervised pre-training paradigms that leverage abundant unlabeled data. For example, masked autoencoding (MAE) inspiired methods [12] have demonstrated effectiveness in capturing spatiotemporal features from raw fMRI signals through reconstruction- based learning 371, establishing a synergistic pipeline with task-specific supervision.
2.2 IMRI-to-text Decoding
Decoding natural language from non-invasive brain imaging modalities such functional magnetic resonance imaging (fMRI) has long posed core challenge in brain-compute interface (BCI) research. Early efforts predominantly focused on closed-vocabular mapped to a fixed set of candidate words. For instance, 2rain2Word [1 employed a classification- based approach to decode individual words from fMRI activity, while Défossez al. 5 utilized contrastive learning to decode words and short phrases from auditory-evoken brain signals. With the advent of large-scala scale pretrained language models (LLMs), recent studies have pivoted toward open- vocabulary decoding, aiming to reconstruct fluent and unconstrained natural language from brain activity. For example, (ni treated fMRI time series foreign language and leveraged a BART-style translation architecture to generate continuous text. In addition, Tang et al. [31] employed a hybrid model that combines linear regression with a generative pretrained transformer (GPT) to perform similarity-bart based decoding. Most recently. BP-GPT 3 introduced a prompt- based decoding paradigm, where embeddings derived from fMRI sequences serve as prompts to condition large language models (e.g., GPT-2) for coherent text generation. To better align the modalities of brain signals and natural language, BP-GPT further integrates contrastive learning to align fdrI-derived and text-derived prompts, significantly improving decoding accuracy.
decoding, wherein brain signals were
3

---

(4) Stage A: (MRI Representation Learning
Phase 1: RCP Preteaining
S1200 [MRI Time Series
Random Mask Layer
Brain Enceder
Brain Decader
HCPS1220
Narratives Flactuning
Narratives (MRI Time Series
Text-guided Mask Layer
Brain Enceder
Brain Decader
Natratives
(b) Stage IMRII-I- text Decoding
Andio Silenalon
(MARI Now Bacr
(MER Migill
Wisppepp
Wrap-ug.
Blstrton easerr
Incremental Decoding
(ARRI
OIRE Eatalling
Prodicted
BART Decoder
Brain Encoler
BART Decoder
Wrap-up
Poolited Seakire
Eesheriding
Predicial Sentrace
"I began my iluerteu canter jarralion
Figure 2: Framework CogReader, comprising two main components: (A) fMRI representation leamning and (B) fMRI-to-text decoding.
Method
In this section, we introduce the overall structure the proposed iMRI-to-text decoding method,
CogReader consists of two main components: (A) fMRI representation learning and (B) fMRi-to-ter decoding. stage A, we employ a text-guided masking strategy within the MAE framework to train the brain encoder in self-supervised manner. In stage B, brian-inspired decoding framework is applied to generate natural language descriptions from the learned fMRI representations. Specifically, given an fMRI signal X with T time frames (each time frame with TR seconds), denoted as = {I1 T2 TTT). our goal is to decode the corresponding natural language sequence presented during scanning, represented words W [w, W2, W., J. where each token w; E V, and denotes an open vocabulary. The implementation details of each component are described in the following subsections.
iee. CogReader. As illustrated in Figure 2
sequence of
3.1 IMRI Representation Learning
Given the limited size of currently available fMRI-text paired datasets, it is challenging to jointly optimize both the fMRI representation learning module and the decoding module. We thus introduce sulf-supervised MAE-based pretraining task (i.e., fMRI reconstruction task) prior to the decoding stage obtain robust brain encoder for fMRI representation learning. However, existing MAE- based approaches for fMRI representation learning suffer from critical limitation: they fail to account for varying semantic importance of different grouped text corresponding to different time frames. To address the above issues, we adopt two-stage training framework for fMRI representation learning and introduce text-guided masking strategy within the MAE to enhance the stability of the learned fMRI embeddings and improve their ability to capture the semantic information of the corresponding text.
Two-Stage Training Strategy: To address the data scarcity in fMRI-text paired datasets, adopt a two-stage training process consisting pretraining and fine-tuning as illustrated Figure 2(a). In the first phase, we pretrain the model on a large-scale public fMRI dataset from the Human Connectome Project (HCP), using random masking strategy to guide the reconstruction of missing signals (mask ratio 75%). This encourages the model to learn spatial coherence and temporal dynamies across brain regions, allowing the encoder and decoder to capture generalizable neural patterns and providing well-initialized parameters for downstream task. In the second phase,

---

finetuned the HCP pretrained model on the target fMRI-text paired dataset (Narratives). Unlike conventional MAE frameworks, we introduce text-guided masking strategy in this stage to replace the random masking approach. Through the two-phase training paradigm, we incorporate large-scale fMRI data, thereby enhancing the general representation capability of the brain encoder.
Calculate word attention
Group words within TR
TR Atteattionguillel Mask
Target
Werd
Grouped Text
TR
Masked
Scalcoce
atiention map
Aftcation
Thne Scrtes
began
I began
TR:
Righe
Pretrained BERT
Brons where tolled
Taler
TR,
Figure Illustration of text-guided masking strategy.
To enhance the semantic relevance of learned fMRI representations, we propose text-guided masking strategy that directs the model's attention toward fMRI signals associated with highly informative semantic content, thereby improving the semantic quality of the learned representations. Given paired input consisting of an fMRI time series X { ( --.. 22,,-.,,r) its corresponding stimulus text W = {wi, W2,..., ..., Wn we first compute the semantic importance of each word in the text using a pre-trained BERT model. Specifically, for each word wy, we calculate its attention-based relevance score:
Text-Guided Masking:
exp(score(w) w;)) j=1 Ek= Exp(score(w)) (wy, wx))
Aw.
(1)
T1
where score(w;, wy) Qi KJ represents the attention score between word w, and word wj, and Q: and K, are the query and key vectors from the BERT self-attention layer. Since BERT uses multi-head attention, we average the attention scores across all heads and apply normalization to obtain the final importance score A., for each word wy:
1 Σ A
Ag,
13
1 E1=1 H 2 A
where Abe donates the attention score of word from head h, and H is the total number of attention heads. To align word-level attention scores with fMRI frames (TRs), we associate each fMRI time point Te with group of words according to fMRI-text pairs. The attention score for each TR is then computed as the sum of its associated word importance: As, Ww,E,, Aw.. Using the attention vector Ax = {Azz: Arg...... Arr J. we apply a differentiable masking strategy across fMRI frames to encourage the model to focus more on neural activity during key moments. Specifically, for the top 40% of fMRI frames with the highest attention scores, we randomly mask 75% of the vertex signals. For the remaining 60% of fMRI frames with lower importance, we randomly mask only 25% of the vertex signals. This strategy encourages the model to focus more heavily on reconstructing fMRI signals from semantically rich time points, improving the quality of the learned representations and ultimately boosting downstream decoding performance.
Autoencoder: We employ a transformer-based autoencoder to capture the overall IMRI representation [6]. The brain encoder consists of spatial module and a temporal module. The spatial module is designed to capture the spatial structural relationships among cortical vertices, while the temporal module models the dynamic changes across time. Both modules are composed of multiple stacked Transformer blocks, each receiving positional embeddings corresponding to either spatial or temporal locations. Specifically, given the input fMRI time series X = {21. .... Fr }. the spatial module first projects it through linear layer, followed by a stack of 8 Transformer blocks, producing a feature matrix Fe E N&dp, where N is the number of vertices and dxpat is the dimension of the spatial module. global average pooling layer then aggregates these vertex-level features into single vector s; de representing the spatial embedding at time t. The temporal module then takes the sequence of spatial embeddings { and encodes temporal dependencies via Transformer layers, producing latent sequence [h+] h4 E Rd The brain decoder takes (he VT as input and adopts 4 transformer blocks and a linear layer to reconstruct the fMRI time
series X {æ1, 22, 2T.
5

---

Reconstruction Loss: Considering that fMRI signals are time series with strong temporal dependen- cies, our reconstruction loss designed account for both the masked and unmasked portions of the sequence. Specifically, given the original fMRI signal X and the reconstructed fMRI signal X, the reconstruction loss formatted as Liecon MSE(X, X).
3.2 IMRI-to-Text Decoding
Current fMRI-to-text decoding frameworks typically generate complete text segments directly from the entire 1MRI sequence, neglecting the segmented and inductive processing strategy adopted by the human brain to manage memory load during language comprehension. This often results in performance degradation when decoding long text sequences. To address this issue, we propose a brain-inspired sequence-by- seeuuuecbb--eeuencee sequence fMRI-to-text decoding framework that integrates incremental decoding with semantic wrap-up mechanism, as illustrated in Figure 2(b). Specifically, we divide the long fMRI sequence into consecutive segments, each aligned with the optimal length for human language processing (discussed in section 4.4). Within each segment, we perform incremental decoding to generate partial textual outputs. After decoding each segment, a wrap-up mechanism summarizes its semantic content, which is then incorporated as prior knowledge into the subsequent segment's decoding process. This sequence-wiss decoding strategy not only mitigates the memory burden and performance drop associated with long-text decoding but also ensures semantic continuity across successive segments.
Incremental Decoding within fMRI Segments: Given an input fMRI time series X {21, Fr J. we first divide it into consecutive segments of equal length, where each seg- ment contains N, IMRI frames. The segmented sequence is denoted as X = (X?,," where K T a brain encoder to extract its corresponding representation, which is then fed into a well-established brain-to-text decoder to directly generate the associated text. This process is referred to incremental decoding, resembling the human brain's 's real-time comprehension of incoming language input. In this study, we adopt the BART model as the fMRI-to-text decoder. due to its well performance on language understanding tasks and its suitability for sequence- -o-eeuceet reconstruction [33 19]. This aligns well with our task structure, where the IMRI representations are translated into corresponding textual sequences 337. Specifically, given the fMRI segment X we first use brain encoder to extract its fMRI representation F; E R RNNxdyaax linear projection layer is used to map F, into the embedding space of the BART decoder, resulting in FBART E N, T.R.Nxxdaaaa xdaaar, where dBART represents the dimensionality the decoder's embedding space. Finally, the projected embedding F.BART, is fed into the BART model to generate the predicted text W; corresponding to the current fMRI segment.
N. The optimal N, is discussed in Section 4.2. For each IMRI segment, we first employ
where dIMRS is the number of feature dimensions. Next, a
Semantic Wrap-Up across fMRI Segments: To address the potential issue of semantic discontinuity across segments in the final decoded text, we incorporate a wrap-up mechanism into the sequential decoding framework, inspired by the human brain's cognitive strategy for integrating information during language comprehension. Specifically, after obtaining the decoding text W; from the IMRI segment X via incremental decoding, we employ a pretrained BERT 7 model extract its contextualized embedding representation, denoted as F: This embedding is then passed through an MLP layer Pe derive a semantic summary vector:
= =PP(PPeen
This process is designed to simulate the inductive summarization mechanism observed in human reading. During the decoding of the next IMRI segment X we incorporate the summarized embedding & into its corresponding representation vector Fi+1, guiding the decoding process for X By incorporating the summarized semantic knowledge from the previous text segment into the decoding of the subsequent segment, the model enhances semantic continuity across successive segments. The dimensionality of the MLP Po is discussed in detail Section 4.2.
Decoding Loss: Through incremental decoding and semantic wrap-up, we decode the complete text
W = { Wi, W.,.... ... from fMRI segments X {X', X2,... ..,,... X*) } in sequence-by-sequene
manner. The objective function for the decoding stage is defined as the cross-entropy loss between

---

the generated text W and the corresponding ground truth text W, which is formatted as
I Liceoodnng
EE Wi, W:)
(4)
1=1
No' wA log( ú;)) where No the number of words in text squence W;. Here, CE(W;, w;) Em
4 Experiments
4.1 Experimental Setup
Datasets This study employs three neuroimaging datasets: HCP S1200 [3). Narratives [24] and Huth dataset [8]. The Human Connectome Project's HCP S1200 dataset provides extensive fMRI data from 1,206 healthy young adults across seven cognitive domains. We primarily use this dataset to pretrain the brain encoder for fMRI representation learning, addressing the scarcity of fMRI-text paired data and enhancing the encoder's generalization. The Narratives dataset, a paired [MRI-text benchmark, contains fMRI recordings from 345 participants during naturalistic auditory comprehension of 27 real-world narrative stories, totaling approximately 6.4 days of functional imaging data. The Huth dataset comprises fMRI data from subjects recorded while they passively listened naturally spoken English stories, and the stories were sourced from The Month and New York Times Modern Love podcasts. Narratives dataset and Huth Dataset are used for the decoding task. All fMRI data from these datasets were preprocessed [10] and projected onto cortical surface using the standardized preprocessing pipelines provided by each source.
Implementaion Details Our model is built using the PyTorch framework (28 and the Huggingface Transformers package [35]. All models utilize the Adam optimizer [16], with a warmup strategy. All experiments are conducted on CUDA 12.2 and the computer with NVIDIA GeForce RIX 3090 GPU, Additional implementation details can be found in the Appendix.
Evaluation Metrics To comprehensively evaluate decoding performance, adopt three text gen- eration metrics: BLEU-N (7). ROUGE-1 (21), and BERTSSore [39]. Among them, BLEU-N and ROUGE-1 assess word-level overlaps between the generated and reference texts, while BERTScore evaluates semantic similarity based on contextual embeddings. Specifically, we report BLEU-1 to BLEU-4 scores for BLEU-N; ROUGE-F, ROUGE-P, and ROUGE-R scores for ROUGE-1; and BERTScore-FF BERTScore-P. and BERTScore-R for BERTScore.
4.2 Parameter Settings
This section discusses the optimal configuration of two key parameters for the IMRI-to-image decoding task. The evaluation is conducted on the Narrative dataset using BLEU-1, ROUGE-R, and BERTScore-R as performance metrics.
Segment Length Ng: : To determine the optimal segment length Ns, we vary from 10 to in steps of 10. For each value of N.. we train and test the CogReader framework accordingly. As shown in Figure 4 all metries exhibit trend of first increasing and then decreasing with longer segment lengths, reaching peak performance at Ng- 20. Therefore, we set Nş- = 20 for all subsequent experiments.
60
BLEU
64-
Input Langth
Input Length (TT1)
Irpect Length (TR)
Figure 4: Parameter settings for segment length N, in incremental decoding.
Dimensionality of the MIP: For the MLP dimensionality, we evaluate five configurations: 0, 32, 64.
128, and 256. The corresponding performance CegReader under each setting shown Figure [ Taking into account both word-level and semantic-lewe evaluation metrics, we ultimately set the
7

---

MLP dimension to 128, as this configuration demonstrates consistently good and stable performance across all metrics. In comparison, other configurations perform well at most a single metric.
13
WOML
-0-
23-
14P235
Inpul MOR Langth (TR)
Irput 9M Langth (TR)
19871
Figure 5: Parameter settings for dimensionality of the MLP in semantic wrap-up.
4.3 Comparison with State-of-the-art methods
We compared our proposed EMRI-to-text method with four statc-of-the-art (SOTA) approaches: UniCoRN (37], EEG-Text 33 BP-GPT 3 and PREDFT [8]. Similar our method, UniCoRN adopts a two-stage decoding framework consisting of fMRI representation learning followed by fMRI-to-text decoding. EEG-Text was originally designed for EEG signal generation tasks. We retrained its encoder using fMRI data to adapt it for fMRI decoding. BP-GPT is prompt-based decoding method that guides text generation by aligning fMRI representations with text embeddings via contrastive learning. PREDFT is an end-to-end fMRI-to-text decoding model that jointly models neural decoding and brain predictive coding. our experiments, we conducted comparisons under three different text decoding lengths, corresponding to fMRI time series of 20 TRs, 40 TRs, and 60 TRs. The comparative results in Narratives are given in Tables [ and 2 The results in Huth dataset can be found in the Appendix A.3 Further qualitative analyses of decoded text cases included in Appendix A.4
Table 1: Comparison of our method and SOTA methods under different text decoding lengths on the Narrative dataset.
Length
Method
BLFU-N(S)
RODGE-1(())
S
BLFU--1
BLBU--
BLEU-3
BLEU--
ROUGE-F
ROUGE-P
ROUGE-R
BERTScure-F
BERTSSome-
BERTSSore-R
UniCoRN
22.9
0
20.3
19.5
20TR
BeG-
GPT
21.5
24.6
2.5
1.9
1.7
21.5
31.9
20.9
23.4
23.4
44.1
42.1
46.3
PREDIT
24.3
0.7
0.1
20.1
22.3
18.3
45.9
43.5
46.7
CogResderrous))
25.4
10.5
4.7
2.6
23.4
22,6
24.6
46.3
45.7
46,9
UnICWRN
19.1
2.3
0.5
0.1
17.8
18.2
17.6
43.8
44.8
40TR
BEG-TTot
BP-GPT
26.1
19.9
3,6
7.3
2.3
1.3
1.5
24.4
21.1
25.1
19.3
22.9
24.7
42.6
45.4
30.4
46.
45.5
PREDFT
25.9
4.5
1.4
0.4
21.1
24.8
18.6
46.3
46.2
46.8
CogReadertours)
31.2
15.3
10.3
8.2
29.6
28.7
30.4
50
49.3
$1.1
UnICoRN
18
1.7
0.2
0.4
16.5
15.9
17
43.2
43.7
42.7
6CTR
BEG Text
BP-GPT
22.1
19.5
8.2
3.4
3.4
1.3
0.6
1.6
28.1
19.4
29.4
19.6
25.1
19.3
47.7
41.6
38.2
47.8
41.7
45.3
PREDFT
26.4
1.9
0.6
28.1
25.5
20.5
48.1
47.7
40.5
CopReadentwwrs)
36.2
20.4
14.7
12.1
36.2
36.6
37.2
53.5
52.6
54.5
Quantitative Comparison As shown Table our method consistently outperforms all SOTA methods across all decoding lengths and evaluation metrics, demonstrating the overall effectiveness of our proposed brain-inspired framework. From the perspective of word-level metrics (BLEU-N and ROUGE-I). 1), the performance of SOTA methods degrades the length of the decoded text increases, whereas our method exhibits an upward trend. In terms semantic-level evaluation (BERTScore), the SOTA methods show relatively stable performance, while our method continues to improve with longer decoding sequences. Similar results are also observed on the Huth dataset (Appendix (A.3). Under the 60TR time window, our method achieves significantly better text decoding performance than other approaches.
Decoding Text Comparison As shown in Table 2 our method consistently outperforms others in both semantic information extraction and narrative consistency. For instance, our model accurately identifies and generates key phrases such as "Pie Man", "Ego sum non an bestia", "scene", "said", and "fled", while producing more coherent and contextually appropriate sentences. In contrast, other methods exhibit greater deviations from the target content. These results suggest that the advantage our approach becomes increasingly evident as the decoding length grows, highlighting the effectiveness and fcasibility of the brain stain-inspired sequential decoding strategy we adopt.

---

Table 2:
Cases analysis for IMRI-text decoding on Narrative dataset. Exact matches between the target and predicted sentences are indicated in bold, while semantic similarity shown in italic font.
Results
Target: was gave the hgurr name I called him Pie Man capital capital M And I described him as a cape wearing masked avenger Though in fact he'd been capeless And said that as he fied the scene he clicked his heels in rakish glee And I gave him catchpbrase in Latin I said that he cried out Ego sum non an bestia which means am not an animal Which makes no sense
UniCoRN: and mc gray to the little Man and no taller than my knee, with coat too big for his frame lingered by the gate. It's been nigh of years he to the then said a case here. The leather sat weathered on the step, its lock rusted shut, but knew what was inside: the girl of the story, the who scribbled tales of dragons in the margins of her notebooks, had tucked her most precious drawings there before she
EEG-Text: and this is where it gets really interesting I' I'm trying to think of a title for the book and I finally come up with an idea that sounds like something out of one of those old pulp stories by John Wilmot who lived in Waukegan Illinois but he was actually born in Westchester County New York which is about twenty five miles outside Chicago So am going to describe my novel The Man Who Walks into Times Square and stops before he can get too lost
BP-GPT: I 'ma teacher and teach people how to make things better by using my own talents And so one day I was walking down the street and this old lady came up to me and she said You know you sound like an intelligent black woman So I told her that's exactly what I I'm here to do PREDFT: : the a said the girl man said the Man and use of best and she me and man of Litgo be and the It's flirting and she guy of the eyes thelee raised non crate and she littleiving it then to of crate non best owan is best me best to
Ours: think that you realize what happened next Pie Man emerged from the late night library drop made his delivery and fled away crying Ego sum non an bestia Or that's 's what it said in my story in the newspaper next day which with photos of him leaving the scene cape flowing behind him doing this And I'm just like praying my life doesn't 't flash before my eyes and ruins
4.4 Ablation Study
In this subsection, we conduct ablation study evaluate the effectiveness of three key components: the HCP pretraining phase, the text-guided masking strategy, and the sequential fMRI-to-text decoding framework. For testing the masking strategy, replace the proposed text-guided masking with a conventional random masking scheme. The experiments are conducted on decoding tasks with fMRI time series of 60 TRs on the Narrative dataset. The results reported in Table The results show a consistent improvement in performance as each module is incrementally added, validating the effectiveness of each individual component. Notably, the brain-inspired sequential decoding framework yields the most significant performance gain, further demonstrating the feasibility and impact of our proposed decoding approach.
Table 3: Ablation Study of our method
Mensad
Sequertial Decoding
Pestriiing
Tou Fooegrddd Marking
BUBU-1
BLBU-2
BLBUSS
B3BB14
SOUGE-F
ROUGGE-P
ROUGE-R R
BERTS F
BERTScoroP
BERTSooeeR
17.7
6.5
32.9
24.7
46.7
32.5
16.5
11.1
8.9
26.2
15.8
3066
51.1
56.3
$1.8
34.0
18.1
10.2
11.1
15.7
52.3
$1.0
36.2
2144
147
36.2
15.6
35.2
535
4.5 Evaluation on IMRI Representation Learning
In this section, evaluate the effectiveness of fMRI representation learning on the Narrative dataset. To this end, we design two experiments, including a comparison with other representation learning methods and an analysis against noise data.
Comparison with other fMRI Representation Learning Method Since the overall framework of UniCoRN is similar to ours in the current fMRI-to-text decoding paradigm, we take UniCoRN as a baseline and replace its fMRI representation learning module with our proposed method, while keeping the fMRI-to-text decoding strategy unchanged. This design allows us to evaluate the effectiveness of our representation learning approach. We conduct comparison experiments on fMRI time series ranging from 10 TRs to 50 TRs. The results, shown in Table 4 demonstrate that

---

under the same decoding strategy, replacing the representation learning component with our method consistently improves decoding performance across all sequence lengths, validating the effectiveness of the proposed representation learning framework.
Table 4: Comparison results of our fMRI representation learning method with other methods
Longth
BLFI-N(S)
ROUGE-109
RFRTSSore1)
BLEU-1
BLBU-2
BLEU-3
BLEU-4
ROUGE
ROUGE-P
ROUGE-R
BERTSooee
BERTScore-
BERTSoore
10TR
UniCoRN
Ours
20.5
2.8
1.3
17.1
16.2
16.6
18.3
41.1
40.5
40.4
41.8
20TR
UniCoRN
Ours
25.4
22.9
10.5
2,5
0.3
4.7
2.6
D
23.4
20.3
22.6
19.6
24.5
21
46.3
43.9
44.2
45.7
42.8
46.9
3UTR
UniCoRN
20.3
2.8
0.5
0.1
18.3
18.3
18.4
41.4
41.5
41.4
Ours
46.8
4UTR
UniCoRN
Ours
21.6
17.8
24.4
46.1
43.8
45.9
42
UnICoRN
SOTR
Ours
7.7
3.2
1.5
26.1
29.4
24
46.5
47.7
45.8
Comparison with Noise Data Previous work 13 has shown that existing open-vocabulary brain decoding methods often yield similar performance on both real and noise data, suggesting that these approaches fail to effectively capture the semantic information encoded in brain signals and instead rely heavily on the memory capacity of the large language model (LLM) decoder. To address this concern, we evaluate the effectiveness of our proposed CogReader model using both real fMRI inputs and noise data as input. The experiment is conducted on fMRI time series with length of 60 TRs. The experimental results are presented in Table [S] The results show that decoding performance is significantly higher when using real fMRI data compared to noise input, providing strong evidence that our iain-inspired method capable of extracting meaningful semantic information from fMRI time series, rather than depending solely on the memorization ability of the LLM.
Table 5: Comparison results between real fMRI data and noise data
BLEU-N(R)
ROUGE-I(9)
BERTSOoRCS)
Train
Test
BLEU-1
BLEU-2
BLBU-3
BLEU-4
ROUGE-F
ROUGE-P
ROUGE-R
BERTScure-F
BERTScoreP
BERTSeore-R
Noise
Noise
27.5
9.4
4.6
1.8
25.6
26.1
25.5
48.2
48.0
48.4
Noise
[MR]
25.3
7.2
2.4
1.4
23.9
23.5
24.8
47.7
47.2
48,3
IMRI
Noise
26.8
1.2
23.1
47.9
47.4
48.5
36,2
12.1
53.5
$2.6
5 Discussion
Limitations and Future Work In the current method, the segment length determined based on optimal decoding performance in a fixed setting and cannot dynamically adjust to the complexity of different stimulus texts. This static segmentation strategy may constrain further improvements in decoding accuracy. Future work could explore content-adaptive segmentation approaches that dynamically predict segment boundaries based on narrative complexity, enabling more flexible adaptation to diverse textual inputs. In light of the relatively low temporal resolution of IMRI, where each TR corresponds to multiple words, making it harder to generate coherent and complete sentences. Future work could explore the strengths of integrating fMRI with EEG, fMRI offers semantic representation capabilities, while EEG provides high temporal resolution, which may help improve decoding accuracy. Moreover, due the limited availability of paired fMRI-to-text datasets, our model was evaluated on a single public dataset. We plan to validate the robustness and generalizability of our approach on multiple datasets in future studies.
Conclusion This work proposes a brain-inspired sequential IMRI-to-text decoding framework that mimics the human cognitive strategy of segmented and incremental language processing. This method divides long fMRI sequences into optimal-length segments, each of which decoded incrementally. A wrap-up mechanism is employed between segments to integrate and propagate semantic information, thereby alleviating memory burden and preserving semantic coherence across the entire sequence.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (62476129) and the STI 2030-Major Projects (2022/D0200000).
References
[1] Nicolas Affolter, Beni Egressy, Damian Pascual, Roger Wattenhofer. Brain2word: decoding brain activity for language generation. arXiv preprint arXiv:2009.04765 2020.
[2] Nima Asadi, Ingrid R Olson, and Zoran Obradovic. A transformer model for learning spatiotemporal contextual representation in fmri data. Network Neuroscience, 7(1):22 47, 2023.
[3] Xiaoyu Chen, Changde Du, Che Liu, Yizhe Wang. and Huiguang He. Open- vocabulary auditory neural decoding using fmri-prompted Ilm. arXiv preprint arXiv:2405.0784/ 2024.
[4] Zheng Cui, Dong Nie, Pengcheng Xue, Xia Wu, Daoqiang Zhang, and Xuyun Wen. Brainx: A universal brain decoding framework with feature aesentannlement and neuro-geometric representation learning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 487, 2025.
[5] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097-1107, 2023.
[6] Xin Deng, Jiahao Zhang, Rui Liu, and Ke Liu. Classifying asd based on time-series fimr using spatial- temporal transformer. Computers biology and medicine, 151:106320, 2022.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171 4186, 2019.
[8] Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang Hc. Decoding visual neural representations by multimodal learning of brain-visual linguissic features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10760- 1077000777-777 2023.
[9] Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. Dewave: Discrete ceg waves encoding for brain dynamics to text translation. arXiv preprint arXiv:2309. /4030, 2023.
[10] Oscar Esteban, Christopher Markiewicz, Ross W Blair, Craig A Moodie, A Ilkay Isik, Asier Erra- muzpe, James D Kent, Mathias Goncalves, Elizaheth DuPre, Madeleine Snyder, et al. firrprep: a robust preprocessing pipeline for functional mri. Nature methods, 16(1):1 1111:111-116,, 2019.
[11] Edward Gibson. Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1):1-76, 1998.
[12] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[13] Hyejeong Jo. Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Are eeg-to-text models working? arXiv preprint 2056099405.06459, 2024.
[14] Marcel Adam Just and Patricia A Carpenter. A capacity theory of comprehension: Individual differences in working memory. Psychological review, 19)1;;12,, 1992.
[15] Jung-Hoon Kim, Yizhen Zhang, Kuan Han, Zheyu Wen, Minkyu Choi, and Zhongming Liu. Representation learning of resting state fmri with variational autoencoder. Neurolmage, 241:118423. 2021.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14X2:6412.6980 2014.
[17] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiw:1312.6114, 2013.
[18] Amanda LeBel. Lauren Wagner, Shailee Jain, Aneesh Adhikari- Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang. Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.

---

[19] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed. Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeecc pre-training for natural language generation, translation, and comprehension. urXiv preprint wwXiv:I910.Ia... 2019.
[20] Rui Li. Yiting Wang. Wei-Long Zheng, and Bao-Liang Lu. A multi-viww spectral- spataa-----aaaa--pppreae masked autoencoder for decoding emotions with self-supervised learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6-14, 2022.
[21] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarizzaion branches out, pages 74-81, 2004.
[22] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-29636 2022.
[23] Shiv Kumar Mudgal, Suresh K Sharma, Jitender Chaturvedi, and Anil Sharma. Brain computer interface advancement neurosciences: Applications and issues. Interdisciplinary Neurosurgeyy. 20:100694, 2020.
[24] Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asich Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey. Yaara Yeshurun, Mor Regev, et al. The "narratives" fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):250, 2021.
[25] Alexander J Oh, Roger Levy, and Richard Futrell. Modeling memory effects neural language models. In NAACL, 2022.
[26] Mark Pagel. Q&a: What human language, when did it evolve and why should we care? BMC biology, 15:1-6, 2017.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318 2002.
[28] A Paszke. Pytorch: imperative style, high-performance deep learning library. arXiv preprint MMXiv:1912.01100 2019.
[29] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to- image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 24728, 2023.
[30] Elizaheth AL Stine-Morrow and Giavanna S McCall. Reading comprehension is both incremental and segmental and the balance may shift with aging, In Psychology of Learning and Motivation, volume 77, pages
Elsevier, 2022.
[31] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858 866, 2023.
[32] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, al. The wu-minn human connectome project: an overview. Neuroimage, 80: 62-79, 2013.
[33] Zhenhailong Wang and Heng Ji. Open vocabulary clettnencephalggrahby-eeeee decoding and zero-shot sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. pages 5350-5358. 2022.
[34] Xuyun Wen, Qumei Cao, Yunxi Zhao, Xia Wu, and Daoqiang Zhang. D-mhgen: An end-to-end individual behavioral prediction model using dual multi-hop graph convolutional network. IEEE Journal of Biomedical and Health Informatics, 28( 28(10)61330 6140, 2024.
[35] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Statc-of-the-a natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations pages 48, 2020.
[36] Jonathan R Wolpaw. Brain-computer interfaces (bcis) for communication and control. In Proceedings of the 9th international ACM SIGACCESS conference Computers and accessibility, pages 1-2, 2007.
[37] Nuwa Xi. Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, and Ting Liu. Unicorn: Unified cognitive signal reconstruction bridging cognitive signals and human language. arXiv preprint arXiv:2307.05355 2023.
12

---

[38] Congchi Yin. Ziyi Ye, and Piji Li. Language reconstruction with brain predictive coding from fmri data. arXiv preprint arXiv:2405. 1159,,
[39] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint ar00:: 09675, 2019.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
See Method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[Yes]
Justification:
The code is available at http://aaaooonnnn...aapsasceeerr
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pulli///uuii wwwtii/gguie///eeeeeeoecc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: See Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /at::://nuuiispe http:///.ccc..uccc.tttccccstttaa
Answer:
[Yes]
Justification:
We strictly adhered the relevant guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer:
[NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification: NA.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreerp for what should or should not be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Implementation Details
BLEU-N Metric Implementation We use the BLEU-N scores as the main evaluation metrics for word-level similarity. This evaluation method is summarized in the algorithm below:
Algorithm 1 BLEU-N Score calculation process
Require: Predicted sentence W, Reference sentence(s) W, n-gram order n
Ensure: BLEU-N score list
1: Initialize: P <- list of n-gram precisions
2: for to n do
3:
n.UMppredd + Count all i-grams in W
4: nuUmrref < Count all i-grams in W
5: TUUMouerla +
wwwwww min(reumref, pred)
6: Ps mam
wwmasariag
Add p; to
8: end for
ROUGE-1 Metric Implementation We compute the ROUGE-1 precision, recall, and score to evaluate unigram overlap between the predicted and reference sentences. The computation process is as follows:
Algorithm 2 ROUGE-1 Score (Precision, Recall, F1) calculation process
Require: Predicted sentence W, Reference sentence
Ensure: ROUGE-1 Recall Rt Precision P. and F1 Score F
1: Extract all unigrams w from W and W
2: Count overlapping unigrams between W and W
3: nulllooerrap &- Ewewnww min(count countw)
4: numree & total number of unigrams in W
5: nuMprrdd total number of unigrams in W
6: R + ANWmorerrag
numree RMTHaveria
7:
nampprdd
8: if R
+ P= then
9:
F F
10: else
F
2-R-P R+P
end
if
BERTScore Metric Implementation We choose BERTScore to evaluate the similarity between the predicted sentences and the reference sentences by computing the alignment- based similarity of contextualized token embeddings. The computation steps are as follows:
A.2 Experimental Settings
In the IMRI Repersentation Learning Stage, during HCP pretraining phase, we split the HCP dataset into training and testing sets in a 4:1 ratio. While for the Narratives dataset, to avoid text leakage, adopt a stimulus split approach to ensure that the train, validation and test sets use different story content, with ratio of 60%, 20% and 20%, during Narratives pretraining stage and fMRI-to-text decoding stage. We finalized the model parameters as detailed Table
During the phase 2 in fMRI Representation Stage, we adopt the proposed Text-guided masking strategy to finetuned the HCP pretrained model on the target fMRI-text paired dataset. The parameters shown in Table 7
21

---

Algorithm 3 BERT (Precision, Recall, F1) Calculation
Require: Pretrained language model M.), Predicted sentence W, Reference sentence W
Ensure: BERTScore Precision Recall R. F1 Score F
1: Tokenize W and W into subword tokens
2: Compute contextual embeddings: En k- M (W), Ew M(W)
3: Compute cosine similarity matrix S 5( = cos Fw [i], Ew Li))
4: P f W
Σ
| max j
i=
5: R f
IW
1
Σ
=1
max
[
$
6: if P + R
0
then
7:
F 0+0
8: else
9:
F < P+R
end if
Table 6: Parameter setting in HCP Pretraining.
parameter
value
parameter
value
parameter
value
mask ratio
0.75
epochs
20
Temporal encoder embed dim
batch
warm-up epochs
Temporal encoder depth
optimizer
Adam
initial LR
le-4
Temporal encoder heads
LR scheduler
StepLR
Spatial encoder embed dim
128
decoder embed dim
64
step size
Spatial encoder depth
8
decoder depth
gamma
0.5
Spatial encoder heads
decoder heads
Table 7: Parameter setting in Narratives Finetuning.
parameter
value
parameter
value
parameter
value
high attention ratio
0.4
batch size
16
LR scheduler
StepLR
mask ratio for high attention
0.75
epochs
40
step size
5
low attetnion ratio
0.6
optimizer
Adam
gamma
0.5
mask ratio for low atlention
0.25
initial LR
lc-4
warm-up epochs
For the fMRl-to-text decoding, we use the pretrained spatial-temporal encoder from Stage and BART as the decoder for text generation from fMRI embedding, with pretrained BERT as the encoder for summarized embedding. The detailed parameters this stage shown in Table
Table Parameter setting fMRI-to-text decoding.
parameter
value
parameter
value
parameter
value
batch size
16
LR scheduler
StepLR 5
fMRI embed dim
256
epochs
20
step size
BART embed dim
1024
optimizer
Adam le-5
gamma warm-up epochs
0.5
BERT embed dim
768
initial LR
5
The equipment used in the experiment is configured as follows: AMAX Tower Workstation TS40-X3, equipped with dual Intel Xeon 43 16 CPUs (2.3 GHz, cores), 256 GB of DDR4 memory (32 GB modules at 3200 MHz), a 480 GB SSD for the system disk, a 3.84 TB SSD for hot data, and 16 TB 7200 RPM SATA enterprise HDD for data storage. The system powered by dual power supplies rated at 2000W and 1650W.
22

---

A.3 Other Results
We evaluate our method on Huth dataset obtained during apassive natural language listening task 18 The comparative results with four SOTA methods in 60 TR are given in Table below.
Table 9: Comparison between our method and SOTA methods
Length
Method
BLEU-NO%)
ROUGE
BLBU-1
BLEU-2
BLEU-3
BLBU-
ROUGE-F
ROUGE-P
ROUGE-R
BERTScure- P
BERT Score P
BERRTSoore
UniCoRN
21.5
2.3
0.6
17.1
14.1
19.7
40.1
57.7
42.7
60TR
BEG Text
BP
20.7
23.6
7.5
2.2
1.0
1.2
27.5
28.1
30.4
23.5
26.1
47.2
47.3
47.2
PREDFT
1.1
26.6
35.2
30,9
A.4 Text cases
Table 10 and II present the comparision results of representative examples in Narratives, comparing the predicted sentence and the reference sentence in different 1MRI time series lengths, including 20 TRs. 40 TRs, and 60 TRs between our propoesed method CogReader and SOTA methods.
Table 10:
Cases analysis for fMR12text. Exact matches between the target and predicted sentences indicated in bold, while semantic similarity is shown in italic font.
IMRI Length
Results
Target: ... campus People started dressing- ... and quoting him class The Ram And toward and I saw I came and I saw in the corner Angela from my Brit Lit class drinking with some friends And now Angela and I had been...
about Pie Man all
20TR
UniCoRN: and the McG first and wearing best and the Man the story Man drinking to the thing and know
EEG-Text: ... the boy climbs with two long arms and two pounces on him and tries to like looking at me give him ... he sees this girl coming after ... that looks like ... on the ground
BP-GPT: ... and
Ours: ... Pie Man So
... from the Ram And sure ... Sheila Beale student ... And now Sheila was different from and all the other Fordham students ... Sheila was the kind of student
Target I called him Pie Man he fled ... And gave ... said that he cried Ego sum non
bestia and he says Pie Man love
40TR
UniCoRN:... and the scene plattorm and I and the rumor stories Sherila non my place
my story
EEG-Text: I'm looking at her and ... kind of figure as than his and then it got good
educations went to work
BP-GPT: know how you do it end of long day ... Clara is holding onto my hand The gray
haired man
Ours: ... you Pie Man And ... And wasn't really Pie Man made delivery and fled away
crying Ego non an bestia what it said in
Target : ... moved New York And I went and looked ... what he did in Brazil And he suid
be wanted Bob to go hire ... because Alan going to do this stupid thing ... And so I'm J Jhon
60TR
Moscow
UniCoRN: he's the girl You't want Sherlock so Watson two things with not Well Arthur
almost days when before
EEG-Text: with the press and came to Bosion ... And he said ...he asked Ryan to lind the same Ryan contacted four officials and ...
BP-GPT: ... went downtown and started ... knew people in the group ... he helped move items but ... was unclear who was from Clara... ... Ours: I did
I'm going stort ... who And Sheila had collaborated with the Dean to ... radio station in St Louis...
got plenty of But the fact that
and I1, our proposed method, CogReader, outperforms SOTA methods in terms of capturing semantics and syntax in tokens, with more accuracy of individual words, which indicated in bold. Specifically, our decoding results capture more key content words ranging from verbs (such as "fled" and "cried") to nouns ("scene" and "night"), including more accurate named entities such as person and place names ("Pie Man" and "St Louis"). and produce sentences that are semantically more aligned with the intended meaning. Despite other SOTA methods are still able to decode some accurate information such as "Man" and "came", they fail to decode more meaningful words such as "The Ram" and "Ego sum non an bestia", which is paired in our method. Therefore, our method produces decoded outputs that exhibit higher word-level overlap and better semantic alignment with the reference texts, demonstrating superior decoding performance.
As shown in Table an
23

---

Table 11: Cases analysis for fMRI2text. Exact matches between the target and predicted sentences are indicated in bold, while semantic similarity is shown in italic font.
fMRI Length
Results
Target: ...I gave the figure... ... called him Pic Man as he fled And gave said that he cried
Ego sum non bestia
20TR
UniCoRN: and me gray to little the Man and ..oo years to the then said a case girl of the story
EEG- ... I'm trying to ... aille for the book ... stories by Woukegan but he was which is about .... So am going to
teach people how make ... one day walking ... and she said you sound Ours: Ihink happened next Pie Man and fied away crying Ego
BP-GPT:
like
bestia Or
said in my story...
Target she and her friends all the at night... She drives around I ever saw mother My mom and dad was government worker
40TR
UniCoRN: ... she remembered how they used to the evenings sometimes when she drove back
and that was when she felt
EEG-Text: ... walking down the street and highway And then home from work she says to me .. There's something wrong with
BP-GPT: ... about
off ... coming down from the top playing with this guy ...
... for no good reason
Ours: My mom as she cried she stopped and she drove home later that night An organization one of many onto the highway... ...
Target: he uh like moves Um the scene then the lady and she uh she looks come Merlin like ... and leaves Um so
60TR
UniCoRN: I says at takes and then and um says him guy like the scene of she is the man of the friending to...
EEG- ... he slowly walks ... and moves ... under the books she where something powerful had ... tries to walk away...
BP-GPT: ... uh stands rather ... when she's about open ... and then so ... when Arthur yells and he finally responds... ...
Ours: like he hears this uh scene takes Probably she looks like the woman who
So when Mutorelli and I...
24


Reviews:
[{"review_id": "xi03baBAmK", "reviewer": "Reviewer_4jo4", "summary": "This paper proposes \"CogReader,\" a new framework for decoding natural language text from fMRI signals, specifically designed to handle long sequences. The authors identify that existing methods, which process an entire fMRI sequence at once, suffer from performance degradation on longer inputs. To address this, their method is inspired by human language comprehension and involves two main contributions. First, it employs a sequential decoding strategy where the fMRI time series is divided into fixed-length segments. Each segment is decoded incrementally, and a \"wrap-up\" mechanism summarizes the output and passes it as a semantic prior to the next segment to maintain continuity. Second, for learning fMRI representations, the paper introduces a text-guided masking strategy within a Masked Autoencoder (MAE) framework, which aims to focus the model on semantically important time points. Experiments on the Narratives dataset show that the proposed method outperforms state-of-the-art approaches, with the performance gap widening as the length of the decoded text increases", "questions": "1 The central framing of the paper is its \"human-like\" approach. However, the core mechanism is a fixed-length segmentation of the fMRI signal. Could you provide a stronger justification from cognitive science literature that supports fixed-length (as opposed to content-driven, adaptive) segmentation as a plausible model for human language comprehension? \n\n2 Given that the overall architecture (HCP pre-trained encoder + LLM decoder) is quite standard, could you more precisely delineate the novelty of your \"wrap-up\" mechanism compared to other methods that handle long sequences or context, such as hierarchical models or models with state-passing mechanisms? Clarifying this might help in assessing the originality of the contribution.\n\n3 The absence of qualitative examples makes it difficult to fully evaluate your model's output. Could you please provide a few representative examples of decoded text from your model and the baseline models for a 60 TR sequence? A discussion of the comparative error types (e.g., semantic drift, repetition, hallucination) would be valuable", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Weaknesses\n\n1. The paper's claim to novelty is somewhat overstated. The general framework—pre-training an fMRI encoder on a large dataset like HCP , then fine-tuning it to produce embeddings that are fed into a conditional generation model or LLM (like BART )—is a standard paradigm in both visual and language decoding from fMRI. This approach has been widely used in prior work such as [1-3] which are not cited, including the cited UniCoRN paper.  This means the primary novelty rests almost entirely on the incremental decoding and \"wrap-up\" mechanism. While this is a reasonable contribution, it feels more like an incremental improvement on a common backbone rather than a fundamentally new framework.\n\n\n2.  The framing of the work as \"Human-Like Language Comprehension\" is not sufficiently justified. While inspired by cognitive science, the implementation uses fixed-length segmentation (N_s=20 TRs), which is more of an engineering heuristic to manage memory than a true model of human cognition. Human reading involves dynamic, content-aware segmentation, not arbitrary fixed-length chunking. The paper lacks a deeper discussion or evidence to support why this specific mechanism is a faithful model of human comprehension.\n\n\n3. A major weakness in evluation is the complete absence of qualitative examples of the decoded text in the main paper or appendix (the appendix is mentioned but not provided). Metrics like BLEU and ROUGE are known to be limited and can be misleading. Without seeing the samples actual generated sentences, it is diffcult to assess crucial aspects like semantic coherence, factual accuracy, and the specific types of errors the model makes compared to baselines. \n\n\nMissed references include but are not limited to:\n\n[1] Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. CVPR 2023\n\n[2] Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities. NeurIPS 2023\n\n[3] MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities. NAACL 2024\n\n\n\nStrengths\n\n1 The paper addresses a well-defined and significant problem in the field of neural decoding. The difficulty of decoding long, continuous language sequences from fMRI is a major bottleneck, and the authors propose a clear strategy to mitigate this issue.\n\n2 The quantitative results presented are strong. Table 1, in particular, makes a compelling case for the method's effectiveness, showing that while baseline performance degrades with longer sequences (from 20 TRs to 60 TRs), the proposed CogReader model shows marked improvement. The ablation studies also systematically validate the contribution of each component.\n\n\n3  The paper is generally well-written and structured. The core ideas are explained clearly, and the figures, especially Figure 2, provide a helpful illustration of the overall architecture and data flow", "was_revised": true, "final_justification": "Thanks for the rebuttal. If the authors can waive in all their described modifications in their revised manuscript, I can raise my recommendation to boarderline accept.", "rebuttal": "We sincerely appreciate the reviewers’ positive feedback on our research topic, experimental results, and overall presentation of the paper. Below, we provide detailed responses to all comments.\n \n(1) W1 Novelty of Work：\nOur work is very novel on achieving better brain signal decoding, in particular, addressing or mitigating the insufficiency of data and long-context challenges.\nFirst, we propose a brain-inspired sequential decoding paradigm for fMRI-to-text decoding that incorporates both incremental processing and segmental integration. The motivation for this framework arises from the fundamental differences between fMRI-to-text decoding and conventional cross-modal translation tasks. Unlike translation, which performs a one-to-one mapping between two independent modalities, fMRI-to-text decoding reconstructs textual content from neural activity patterns generated during language comprehension. As a result, models designed for cross-modal translation are not directly applicable to this task. Based on this insight, we hypothesize that adopting a decoding framework more aligned with human language processing mechanisms could better address this challenge. Our experimental results (Section 4.3, Table 1) and ablation studies (Section 4.4, Table 2) support this hypothesis, demonstrating the feasibility of brain-inspired fMRI-to-text decoding and providing a new perspective for future model design. In the final version, we will add a clearer explanation of the motivation behind this brain-inspired framework.\nSecond, we design a text-guided masking strategy during fMRI representation learning to guide the model to focus on fMRI time points that correspond to high-value text tokens, thereby enhancing the semantic relevance and expressiveness of the learned fMRI representations. The comparison with other representation learning methods (Section 4.5, Table 3) further demonstrates the effectiveness and advantages of this approach.\n\n(2) W2&Q1 Fixed Segmentation Scheme：\nThe fixed segment length of 20 TRs was determined via systematic validation in our work (Section 4.2, Figure 4). This finding is roughly consistent with evidence from cognitive science, which shows that text passages of approximately 55 words support efficient reading at both normal and fast speeds and lead to the highest comprehension levels (Dyson et al., 2001). Given that each TR (1.5s) corresponds to roughly three words, a 20TR segment covers approximately 60 words, aligning well with the optimal text length for comprehension reported in. This provides theoretical support for selecting 20 TRs as the optimal text segment length in our work.\nSince the main objective of this study was to validate the feasibility of applying brain-inspired mechanism to fMRI-to-text decoding, our focus was on evaluating the overall framework performance rather than extensively exploring text segmentation strategies. As discussed in our paper and highlighted by the reviewers, our future work will further investigate content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity, enabling the model to more flexibly adapt to diverse textual inputs.\n\n[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.\n \n(3) W3&Q3 Examples of Decoded Text：\nDue to page limitations, detailed examples of decoded text are provided in the supplementary materials. Specifically, in Section A.3 (\"Text Cases\") of the Appendix, we provide representative decoded examples for our method and SOTA approaches across different input lengths. Our proposed method outperforms SOTA methods in terms of capturing semantics and syntax in tokens and captures more key content words ranging from verbs to nouns, including more accurate named entities such as person and place names, and produce sentences that are semantically more aligned with the intended meaning. Therefore, our method produces decoded outputs that exhibit higher word-level overlap and better semantic alignment with the reference texts, demonstrating superior decoding performance. In the final version, we will include some of the decoded text in the main body of the paper.\n\n(4) Q2 Novelty of wrap-up mechanism：\nIn our framework, the proposed wrap-up mechanism is superficially similar to existing state-passing mechanisms but differs fundamentally in both the modality and the structural flow of the transferred information. \nTraditional state-passing mechanisms typically operate within a single modality (e.g., text-to-text), propagating hidden states from one segment to the next to maintain temporal continuity, akin to how recurrent neural networks or autoregressive models carry over internal states across time steps. In contrast, our wrap-up mechanism is a cross-modal semantic summarization process: it decodes an fMRI segment into text, encodes the generated text using BERT to extract a rich semantic representation, and then projects this representation back into the fMRI latent space of the subsequent segment via an MLP to guide future decoding. This is not merely state propagation, but a semantic-level integration that involves modality translation and higher-level abstraction.\nWhile our method shares superficial resemblance to state-passing, particularly in its sequential setup, it is structurally distinct. State-passing emphasizes retention and transfer of temporal internal states (e.g., hidden activations) across steps, whereas our wrap-up mechanism summarizes each fMRI window into a semantic embedding that represents integrated content, without explicitly transferring low-level hidden states. In that sense, it is more analogous to a block-wise summarization approach, such as using a [CLS] token or attention-based pooling to encode the essence of an input segment.\nWe believe this difference is not just architectural but also functional: the wrap-up mechanism aims to provide a semantic bridge across fMRI segments by leveraging language-domain priors, rather than preserving low-level continuity."}, {"review_id": "lHyoL9ERt7", "reviewer": "Reviewer_nUQu", "summary": "The paper proposes a method for decoding fMRI to text using a segmented approach inspired by how humans process language to mange the load in the working memory. Instead of decoding the full fMRI sequence at once, the input is split into segments that are decoded incrementally. After each segment, there is a “wrap-up” phase that integrates the current semantic understanding and passes it forward to inform the next decoding step, helping maintain coherence across segments. In addition to this sequential decoding setup, the authors introduce a two-stage pretraining scheme for learning better fMRI representations. First, they use a large-scale dataset for general pretraining. Then, they propose a text-guided masked autoencoder (MAE) that leverages attention scores from the associated text to guide which fMRI time points to mask and reconstruct—essentially forcing the model to focus on semantically important parts. Experiments on the Narratives dataset show that their approach outperforms existing methods, especially for longer sequences. In addition, the paper shows ablation studies to support the design decisions including the advantage of using sequential decoding, pretraining + text-guided pretraining, studies on the segment length, and various other experiments like testing the model with random data.", "questions": "See weakness, but here are the most important questions:\n1. How are the splits done for the text-guided pretraining and decoding. Looking briefly at the supplementary material I can see the splits done at 60, 20, 20, however, can you please confirm whether the exact same splits are used for both stages (to prevent pretraining on data that is used for evaluating the decoding process).\n2. Can you clarify why in UniCoRN they report 34% in BLEU score, whereas in this paper the number is different. \n\nA minor thing:\n3. In line 296, instead of \"time series of 20 TRs\", was it meant to be written \"time series of 60 TRs\"?", "limitations": "The paper mentions already some critical limitations like the fixed segment size, as well as evaluation in a single public dataset. However, in addition, it is important to mention that even the text-guided pretraining is strictly speaking supervised and as such it is limited in applying at other datasets.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "##  Strengths:\n\n1. Text-guided masking pretraining: The paper introduces a novel text-guided masking strategy during pretraining. By using attention scores from the text to guide which fMRI timepoints to reconstruct, the model learns to focus on semantically important tokens. This is an intersting idea that ties textual semantics directly to fMRI representation learning.\n2. Sequential, segment-based decoding: Inspired by human language processing, the model decodes fMRI inputs in segments, with a wrap-up phase that integrates prior context. This helps preserve semantic flow across longer sequences and is a step forward compared to one-shot decoding approaches.\n3. Thorough experimental validation: The authors present extensive ablation studies and comparisons against existing methods. Their approach consistently outperforms baselines, especially for longer fMRI sequences.\n\n## Weaknesses:\n\n1. Limited contextual integration in wrap-up phase: While the paper claims to model human-like wrap-up effects, the implementation appears to simply pass the sentence embedding of the previous segment (through an MLP) into the decoder for the next segment. This means the model only conditions on the immediately preceding segment, not the full history, which weakens the claim of human-like contextual integration. The current explanation of the wrap-up phase is a bit vague and it would be valuable to clarify this mechanism and test a version where more cumulative context is aggregated. For example, it's unclear how the wrap-up embedding is merely summed with the fMRI embedding (the only information available is the symbol ⊕ symbol in the figure, but it is nowhere described whether those embeddings are simply added).\n2. Potentially unfair comparison to prior work: The paper shows that performance of prior models degrades with longer sequences, but this comparison may be unfair. Earlier methods (e.g., UniCoRN) are not designed to decode \\textit{long} sequences at once, and in the original paper it is presented by decoding signals of a few TRs (in fact looking at UniCoRN paper they provide a table showing that the length T=10 works best for their experimental setting). In other words, if UniCoRN works better by first segmenting, and then decoding each segment independently, than that is a better baseline instead of using UniCoRN with a very long sequence which might suffer due to the big context window. Additionally, some strong results from prior work—such as BLEU scores around 34.77% reported in UniCoRN—are not directly reconciled with the numbers presented here. It is important to use the setup for which UniCoRN is designed for and compare to this approach. \n3. Text-guided masking is not self-supervised: Although the pretraining approach is compelling, it depends on access to paired text to compute attention scores. This limits its applicability as a general-purpose pretraining strategy. Moreover, this stage is performed on the same dataset used for evaluation, raising concerns about generalization and potential data leakage. It is not clearly stated whether the same splits were used across pretraining and decoding to avoid this issue. If not carefully separated, this could significantly inflate performance.\n4. Fixed segmentation scheme: Segmenting the fMRI time series into fixed-length chunks may lead to cuts at unnatural points in the narrative, disrupting the underlying semantic coherence. Exploring adaptive or linguistically informed segmentation (e.g., based on sentence or clause boundaries) could improve results.\n5. Unclear role of wrap-up in actual fMRI decoding: It's unclear whether the observed performance gain from the wrap-up phase stems from better modeling of brain activity or simply from smoother language generation. As an ablation, it would be useful to decode segments independently (as in UniCoRN), and then apply a language model post-hoc to enforce consistency, isolating the contribution of the fMRI model from the decoding pipeline.\n6. The model is only evaluated at a single public dataset. Although fMRI-to-Text are difficult datasets to get, there are still other publically available datasets where one can validate the approach.", "was_revised": true, "final_justification": "I thank the authors for their thoughtful rebuttal and additional experiments. While the changes in the rebuttal address all my main concerns, some overall limitations remain in the work. In particular, the reliance on a fixed window size and the assumption in having access to paired fMRI-text data, may not hold in many real-world applications. Overall, their experiments clearly show that their method improves decoding performance (fmri-to-text) in their evaluation datasets. Therefore, I am increasing my score from 3 to 4.", "rebuttal": "We appreciate your positive comments on our work, including “novel strategy” and “interesting idea”. Detailed responses to all comments are provided below.\n \n(1) W1 Limited contextual integration in wrap-up：\nOur wrap-up mechanism performs incremental semantic integration rather than relying solely on the immediately preceding segment. At each step, the text decoded from the current fMRI segment is encoded using BERT to obtain a rich semantic representation, which is then projected into the fMRI feature space of the next segment via an MLP to guide subsequent decoding. Because this process is applied recursively, the semantic embedding passed to the next segment already incorporates information from all previous segments, accumulated through the progressively decoded text. In other words, the wrap-up embedding is not merely the representation of a single sentence but a continuously integrated semantic summary that reflects the entire preceding context. We will provide a more detailed explanation of this process in the final version.\n\n(2) W2&Q2 Unfair comparison to prior work：\nFollowing the reviewer’s suggestion, we modified UniCoRN’s decoding strategy for 60TR sequences by dividing them into 10TR segments, decoding each segment independently, and concatenating the outputs. As shown in the table below, our method still substantially outperforms UniCoRN. We attribute this superiority to two key innovations: 1) the introduction of a text-guided masking strategy during representation learning, which improves the semantic relevance and effectiveness of the learned fMRI representations; and 2) the use of a sequence-wise decoding framework with a wrap-up mechanism that leverages semantic information from preceding segments to guide subsequent decoding, thereby improving accuracy while maintaining semantic coherence in the generated text.\n\n|Method|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R|\n|-|-|-|-|-|-|-|-|-|-|-|\n|UniCoRN|23.2|7.4|3.3|1.6|21.8| 23.9|17.5|46.2| 46.7| 46.7|\n|CogReader(ours)|**38.3**|**23.4**|**18.3**|**15.9**|**36.4**|**37.6**|**36.2**|**54.4**|**53.8**| **55.1**|\n\nAdditionally, the discrepancy between the results reported for UniCoRN and ours stems from a data leakage issue during its evaluation phase, which has also been reported in other paper (Yin et al., 2024). Specifically, according to the code provided by UniCoRN, its decoder model uses the same forward method for both training and inference. During inference, the decoder_input_ids (shifted ground-truth labels) are fed into the model, allowing real text to influence the generation process and thus causing data leakage. In our evaluation, we fixed this issue by removing the ground-truth input during inference. As a result, the BLEU scores decreased compared to those originally reported for UniCoRN.\n \n[1] Yin et al., 2024. Language reconstruction with brain predictive coding from fmri data. arXiv preprint arXiv:2405.11597. 2024.\n \n(3) W3&Q1 Text-guided masking：\nOur fMRI representation learning adopts a “self-supervised pretraining + text-guided fine-tuning” approach. Specifically, we first perform self-supervised pretraining on the HCP dataset (non-fMRI-text paired data) and then apply fine-tuning on fMRI-text paired data using the text masking strategy proposed in this work. The adoption of this strategy has two advantages: 1) it alleviates the limitation of scarce fMRI-text paired data; 2) it effectively guides the model to focus on fMRI time points corresponding to high-semantic-value text tokens, thereby enhancing the semantic relevance and effectiveness of the learned representations. The comparison with other representation learning methods (see Section 4.5, Table 3) validates the effectiveness and advantages of our method.\nRegarding the potential issue of data leakage, we clarify that the first-stage fMRI representation learning and the second-stage decoding adopt an identical data-splitting scheme. Specifically, the data are split based on stimuli into training (60%), validation (20%), and test (20%) sets, which are saved in separate directories. During both the representation learning and decoding stages, the model loads data exclusively from these corresponding directories, ensuring that the data splits used in both stages are completely consistent and that there is no overlap between the data used for pretraining and that used for decoding evaluation. Therefore, the results reported in this paper are free from any risk of data leakage.\n\n(4) W4 Fixed Segmentation Scheme：\nThe primary goal of this study was to validate the feasibility of the proposed brain-inspired fMRI-to-text decoding framework, particularly to examine whether the incremental decoding and wrap-up mechanisms can enhance decoding performance for long text sequences. Therefore, our focus was on evaluating the overall framework performance rather than extensively exploring segmentation strategies. To this end, we adopted a relatively simple yet effective fixed segmentation scheme and experimentally verified that a segment length of 20 TRs (approximately 60 words) achieves the best decoding performance. This finding is consistent with results from cognitive science, which show that humans achieve efficient reading and optimal comprehension with text passages of about 55 words (Dyson et al., 2001). Such consistency provides theoretical support for our choice of 20 TRs as the optimal segment length.\nHowever, as noted in both the discussion section and the reviewers’ comment, a fixed segmentation strategy has inherent limitations because it may split the narrative at unnatural points. In future work, we plan to explore content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity or syntactic structure, allowing the model to more flexibly adapt to diverse textual inputs.\n\n[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.\n  \n(5) W5 Unclear role of wrap-up in actual fMRI decoding：\nWe conducted an additional experiment to further validate the effectiveness of the wrap-up mechanism. Specifically, we divided the 60TR fMRI sequence into three segments, performed independent decoding for each segment, and then applied GPT-2 to assess the semantic coherence between consecutive segments, and generate simple connecting phrases and adjust words in original segments to enhance consistency. As shown in the table below, our proposed wrap-up-based method achieves significantly better decoding performance compared to this post-processing approach. We will include the results of this ablation study in the final version.\n\n|Method|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R|\n|-|-|-|-|-|-|-|-|-|-|-|\n|post_hoc|26.9|9.7|4.5|2.1|28.3|30.6|26.1|49.3|48.4|50.5|\n|Wrap-up|**38.3**|**23.4**|**18.3**|**15.9**|**36.4**|**37.6**|**36.2**|**54.4**|**53.8**|**55.1**|\n\n(6) W6 Evaluation on Other Dataset：\nWe conducted additional comparative experiments on a new public dataset (LeBel et al., 2023). This dataset contains fMRI signals recorded from 8 participants when they listened to 27 complete and natural narrative stories (with a total duration of approximately 6 hours). The experimental setup is consistent with that used for the Narratives dataset in the paper, and the comparison results for 60TR fMRI signals are shown in the table below. As can be seen, our method still significantly outperforms existing SOTA methods. These results will be included in the supplementary materials of our final version.\n\n|Methods|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R| \n|-|-|-|-|-|-|-|-|-|-|-|\n|UniCoRN|21.5|5.4| 2.3|0.6|17.1|14.1|19.7|40.1|37.7|42.7| \n|EEG-Text|23.6|7.5| 2.2|1.0|27.6|28.1|26.1|47.2|47.3|47.2| \n|BP-GPT|20.7|8.1| 2.6|1.2|27.1|30.4|23.5|50.7|51.3|49.2| \n|CogReader(ours)|**35.2**|**18.0**|**10.2**|**7.1**|**32.2**|**34.0**|**30.9**|**53.9**|**54.5**|**53.3**|\n\n[1] LeBel et al., 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1): 555.\n \n(7) Q3 Writing Error：\nThank you for pointing out this writing error. We will correct it in our final vision."}, {"review_id": "D6i6mW6Lvj", "reviewer": "Reviewer_4tPt", "summary": "This paper proposes CogReader, a brain-inspired framework for fMRI-to-text decoding that mimics human cognitive strategies for language comprehension. The key innovation is a sequential decoding approach that divides long fMRI sequences into segments, performs incremental decoding on each segment, and uses a wrap-up mechanism to maintain semantic continuity. Additionally, the authors introduce a text-guided masking strategy for fMRI representation learning that leverages BERT attention scores to identify semantically important timepoints.", "questions": "1. Can you provide evaluation on additional datasets to support broader applicability? Even preliminary results would strengthen the paper significantly.\n2. What happens with dynamic segment length based on text complexity rather than fixed length? How do you analyze the trade-off between local and global context?\n3. Have you experimented with more sophisticated semantic integration beyond BERT+MLP? How sensitive is performance to the wrap-up mechanism design?\n4. Can you provide theoretical analysis of why segmented processing should outperform holistic approaches?\n5. How does the method perform when text-guided masking isn't available during training?", "limitations": "The authors adequately address some limitations in the discussion, particularly the static segmentation strategy and single dataset evaluation. However, they could better discuss: (1) potential negative effects of losing global context through segmentation, (2) computational overhead of the sequential approach, and (3) applicability constraints when paired training data isn't available.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The framework meaningfully incorporates human language processing mechanisms into computational design, which is well-motivated and represents a principled approach.\n2. The experimental design is comprehensive with proper baselines, thorough ablation studies, and validation against noise data.\n\nWeaknesses:\n1. The method is only evaluated on one dataset, which significantly limits generalizability claims. \n2. While cognitively inspired, the paper lacks deeper theoretical analysis of why segmented processing should outperform holistic approaches. \n3. The wrap-up mechanism (BERT + MLP) and attention score aggregation seem overly simplistic given the complexity involved.", "was_revised": true, "final_justification": "My questions have been satisfactorily answered, and I find the overall contribution of the paper to be both valuable and suitable for acceptance at NeurIPS. I am raising my score from 4 to 5.", "rebuttal": "We appreciate your valuable feedback, and our detailed responses are provided below.\n \n(1) W1&Q1 Evaluation on Other Dataset：\nWe further evaluated our method on an additional public dataset (LeBel et al., 2023), which includes fMRI recordings from 8 participants when they listened to 27 full-length natural narrative stories (totaling about 6 hours). The experimental configuration follows the same setup as used for the Narratives dataset in the study. The results for the 60TR decoding task are presented in the table below. As shown, our method also consistently achieves superior performance compared to current SOTA methods. These results will be added to the supplementary materials of our final version.\n\n|Methods|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R| \n|-|-|-|-|-|-|-|-|-|-|-|\n|UniCoRN|21.5|5.4| 2.3|0.6|17.1|14.1|19.7|40.1|37.7|42.7| \n|EEG-Text|23.6|7.5| 2.2|1.0|27.6|28.1|26.1|47.2|47.3|47.2| \n|BP-GPT|20.7|8.1| 2.6|1.2|27.1|30.4|23.5|50.7|51.3|49.2| \n|CogReader(ours)|**35.2**|**18.0**|**10.2**|**7.1**|**32.2**|**34.0**|**30.9**|**53.9**|**54.5**|**53.3**|\n\n[1] LeBel et al., 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1): 555.\n \n(2) W2&Q4 Theoretical analysis of why segmented processing outperform holistic approaches：\nSegmented processing offers clear theoretical advantages over holistic approaches, which can be explained from both cognitive science and modeling perspectives.\nFrom a neuroscience perspective, our segmented and sequential decoding framework that integrates incremental decoding and the wrap-up mechanism is consistent with numerous findings on how the human brain processes language. Prior studies have shown that humans typically divide linguistic input into smaller, manageable units (e.g., clauses or sentences) and integrate information at natural boundaries (Tiffin-Richards et al., 2018). This strategy effectively reduces cognitive load and improves comprehension accuracy. Behavioral evidence further indicates that when text passages are approximately 55 words long, humans achieve optimal reading efficiency and comprehension at both normal and fast reading speeds (Dyson et al., 2001). The optimal segment length identified in our model, 20 TRs (about 60 words), closely aligns with these cognitive findings, providing strong theoretical support for our brain-inspired segmented decoding design.\nFrom a modeling perspective, fMRI signals inherently have a low signal-to-noise ratio, and holistic decoding of long time series significantly increases the difficulty of extracting meaningful information. Inspired by the advantages of state-passing approaches for handling long sequences, we divide the fMRI time series into multiple segments, perform incremental decoding on each segment, and use the wrap-up mechanism to pass a semantic summary of the previous segment to guide the decoding of the next. This design not only mitigates performance degradation caused by excessive sequence length but also improves semantic consistency across segments.\nComparative experiments with existing holistic decoding approaches (e.g., UniCoRN, EEG-Text, BP-GPT) demonstrate that our segmented decoding method achieves significantly better performance, further validating its effectiveness and practical advantages.\n\n[1] Tiffin-Richards et al., 2018. The development of wrap-up processes in text reading: A study of children’s eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(7): 1051.\n[2] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.\n\n(3) W3&Q3 Over-Simplicity and Sensitivity of the Wrap-up Mechanism：\nOur proposed wrap-up mechanism is designed to semantically compress the textual information decoded from the previous fMRI segment and pass it as a summary vector to guide the decoding of the next segment. Thus, it is not a simple state propagation but rather a cross-modal semantic integration process involving modality translation and higher-level abstraction. Specifically, the mechanism first decodes an fMRI segment into text, encodes the generated text using BERT to obtain a rich semantic representation, and then projects this representation into the fMRI latent space of the subsequent segment via an MLP to assist in decoding. This design achieves semantic summarization and cross-modal transfer in a lightweight manner, avoiding the introduction of excessive parameters that could lead to unstable training.\nWe systematically evaluated different wrap-up designs and found that more complex models do not necessarily yield better performance. For example, in experiments with varying MLP hidden dimensions, a dimension of 128 achieved the best performance on both word-level and semantic-level metrics, whereas increasing the dimension to 256 led to a significant drop in performance, even falling below that of 32 or 64 dimensions (Section 4.2, Figure 5). We also explored more sophisticated frameworks, such as integrating BLIP for semantic aggregation (Li et al., 2022). However, experimental results showed that the BLIP-based approach achieved comparable performance to the MLP-based approach but exhibited less stable training, likely due to its higher data requirements. Based on these findings, we ultimately adopted the simpler and more stable BERT+MLP design. In future work, we plan to explore more efficient wrap-up mechanisms to achieve better semantic compression and further improve model performance.\n\n[1] Li et al., 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning 2022 Jun 28 (pp. 12888-12900).\n \n(4) Q2 Fixed Segmentation scheme：\nThe primary goal of this study was to validate the feasibility of the proposed brain-inspired fMRI-to-text decoding framework, particularly to examine whether the incremental decoding and wrap-up mechanisms can improve decoding performance for long text sequences. Therefore, in the current work, we adopted a relatively simple segmentation strategy with a fixed segment length. Through systematic experiments, we verified that using a segment length of 20 TRs (approximately 60 words) yields the best decoding performance. Interestingly, this result is consistent with findings from cognitive science showing that humans achieve optimal comprehension when reading passages of around 55 words (Dyson et al., 2001). This consistency provides theoretical support for the fixed segmentation strategy adopted in our study.\nAs discussed in our paper, we acknowledge that a fixed segmentation strategy may split the narrative at unnatural points, resulting in improper semantic segmentation. Determining how to segment text effectively is a challenging but important topic. In future work, we plan to explore content-adaptive segmentation methods that dynamically predict segment boundaries based on narrative complexity or syntactic structure, enabling the model to flexibly adapt to diverse textual inputs. Furthermore, we aim to investigate approaches that combine dynamic segmentation with more flexible cross-segment information integration to achieve a better balance between local decoding accuracy and global semantic coherence.\n\n[1] Dyson et al., 2001. The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4): 585-612.\n\n \n(5) Q5 Ablation of text-guided masking：\nIn Section 4.4, we conducted an ablation study to verify the effectiveness of text-guided masking. Specifically, we retained the pretraining module (HCP dataset pretraining in the first stage) and the sequential decoding module (i.e., incremental and wrap-up based fMRI-to-Text decoding framework) while removing the text-guided masking strategy, and then evaluated the model’s performance (see the 3th row in the table below). To enhance the semantic relevance of learned fMRI representations, the proposed text-guided masking strategy identifies words with high semantic relevance, then prioritizes the reconstruction of fMRI time points associated with these key words, which directs the model’s attention toward fMRI signals associated with highly informative semantic content, thereby improving the semantic relevance and effectiveness of fMRI representations. The results show that removing text-guided masking led to a decrease in decoding performance across all metrics, further demonstrating the importance of this module in improving the model’s performance. \n\n|Sequential Decoding|Pretraining|Text-guided Masking|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R|\n|-|-|-|-|-|--|-|-|-|-|-|-|-|\n|✗|✗|✗|17.7|6.5|2.4|1.1|29.2|32.9|24.7|46.7|47.6|45|\n|✓|✗|✗|32.5|16.5|11.1|8.9|28.2|25.8|30.6|51.1|50.3|51.8|\n|✓|✓|✗|35.9|20.5 |15.1|12.8|34|34.4|34.4|53.4|53.1|54.1|\n|✓|✓|✓|**38.3**|**23.4**|**18.3**|**15.9**|**36.4**|**37.6**|**36.2**|**54.4**|**53.8**|**55.1**|"}, {"review_id": "ka7SS8wLOY", "reviewer": "Reviewer_aaPP", "summary": "The paper introduces a new method for fMRI-to-text decoding, where they first pretrain a brain encoder on a large-scale fMRI dataset with a self-supervised MAE-style objective, then finetune the model on the target fMRI-to-text paired dataset. In particular, they introduce two mechanisms inspired by human language processing: segmental and incremental processing.", "questions": "Questions:\n1. fMRI-to-text is like translation in NLP between two languages, where transformers succeed without explicitly adding segmental or incremental processing mechanisms. I am curious to understand why the authors think these will specifically benefit the problem of fMRI-to-text. Perhaps adding some elaboration would be helpful.\n2. For Table 1, it seems that 60 > 40 > 20 TRs. I would be interested to see if increasing the number of TRs beyond 60 would lead to higher BLEU, ROUGE, and BERTScore?\n3. What does the decoded text actually look like? Beyond the BLEU/ROUGH/etc. scores, it would be helpful to show some examples. For example, showing the original text, and different generations of decoded text under various random seeds. Also, perhaps it would be useful to qualitatively contrast this against generations from prior methods, so that it would be easier to understand the ways in which the proposed method makes an advance over prior methods.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Good conceptual contributions of the text-guided masking strategy for MAE fMRI pretraining, and incremental, segmented fMRI-to-text decoding. \n2. Their proposed method outperforms prior methods (UniCoRN, EEG-Text, BP-GPT) on all quantitative metrics evaluated (BLEU, ROUGE, BERTScore).\n\nWeaknesses: \n1. I think the paper would benefit from more qualitative results (see Question 3 below) to understand how the proposed innovations improve fMRI decoding, and how close their method is to \"solving\" the fMRI decoding problem.\n2. I am interested in understanding the impact of their proposed sequential decoding framework (incremental and segmental processing) as that seems to be a key difference between this method and simply using a standard transformer architecture. However, this is challenging to understand empirically without a direct ablation. Referring to Table 2, first row, they ablate it when pretraining and text-guided masking are absent. But it would be most ideal, if possible, if there were an ablation where pretraining and text-guided masking are present, but not the sequential decoding strategy. This might be an important ablation to understand if adding structure (e.g., through their proposed sequential decoding method) is still helpful and important when there is sufficiently large pretraining data.", "was_revised": true, "final_justification": "The authors added new experimental results and motivations for their method. One weakness that is outstanding currently is that the qualitative results of the model still seem significantly different from the actual text (see my comment to the authors for more details). This makes it hard to tell if the method has actually made a substantial methodological advance toward solving the problem. Because of this, it is hard to evaluate the results in the context of the problem being tackled.", "rebuttal": "We appreciate your valuable feedback, and our detailed responses are provided below.\n \n(1) W1&Q3 Examples of Qualitative Results：\nDue to page limitations, we provide the qualitative analysis results (i.e., actual text decoding examples) in the supplementary materials, specifically in Section A.3 (“Text Cases”) of the appendix. In this section, we present representative decoding examples of our method and SOTA approaches under different input lengths. The results show that our approach outperforms SOTA methods in capturing both semantic and syntactic information at the token level. It can identify more key content words, such as verbs, nouns, and named entities like person and place names. The generated sentences are also more semantically aligned with the reference texts. In the final version, we will include some of the decoded text in the main body of the paper.\n \n(2) W2 Ablation of Sequential Decoding：\nFollowing your suggestion, we added an ablation study to evaluate the effectiveness of Sequential Decoding. Specifically, we retained the text-guided masking module and the pretraining module, but replaced our proposed Sequential Decoding (i.e., segmented processing combined with incremental decoding and the wrap-up mechanism) with a holistic decoding approach. The new results, shown in the 4th row of the table below, demonstrate a clear performance drop when Sequential Decoding is removed. For example, BLEU-1 decreases from 38.3 to 21.6, ROUGE-R drops from 36.5 to 25.0, and BERTScore-R decreases from 55.1 to 47.2. These findings highlight that Sequential Decoding plays a crucial role in improving the decoding performance, validating its importance in our proposed framework. This ablation result will be added in our final version. \n\n|Sequential Decoding|Pretraining|Text-guided Masking|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R|\n|-|-|-|-|-|--|-|-|-|-|-|-|-|\n|✗|✗|✗|17.7|6.5|2.4|1.1|29.2|32.9|24.7|46.7|47.6|45|\n|✓|✗|✗|32.5|16.5|11.1|8.9|28.2|25.8|30.6|51.1|50.3|51.8|\n|✓|✓|✗|35.9|20.5 |15.1|12.8|34|34.4|34.4|53.4|53.1|54.1|\n|✗|✓|✓|21.6|7.9|3.2|1.5|26.6|29.4|25|47.4|47.7|47.2|\n|✓|✓|✓|**38.3**|**23.4**|**18.3**|**15.9**|**36.4**|**37.6**|**36.2**|**54.4**|**53.8**|**55.1**|\n\n(3) Q1 Motivation of Segmental and Incremental Mechanisms based Method：\nfMRI-to-text decoding is fundamentally different from traditional machine translation tasks. Unlike translation, which performs a one-to-one mapping between two independent modalities, fMRI-to-text decoding reconstructs textual content from neural activity patterns generated during language comprehension of the human brain. As a result, models designed for cross-modal translation are not directly applicable to this task. Based on this insight, we hypothesize that adopting a decoding framework more aligned with human language processing mechanisms could better address this challenge. To this end, we propose a brain-inspired sequential decoding paradigm that incorporates both incremental processing and segmental integration. Both our experimental results (Section 4.3, Table 1) and ablation studies (Section 4.4, Table 2) strongly support this hypothesis. Compared with existing SOTA methods, our approach shows significant overall improvements, with particularly strong performance on long-sequence decoding tasks. The ablation studies further demonstrate that Sequential Decoding (i.e., segmented processing combined with incremental decoding and the wrap-up mechanism) contributes the most to performance gains, highlighting the importance and effectiveness of incorporating cognitively inspired components in fMRI-to-text decoding. In the final version, we will add a clearer explanation of the motivation behind segmental and incremental mechanisms.\n\n(4) Q2 More Results on Longer fMRI：\nDue to the limited availability of fMRI-text paired data, increasing the fMRI sequence length further cannot provide a sufficient number of training samples to meet the requirements for effective model training. For example, when the fMRI length is increased to 80 TRs, the total number of samples is only around 1,600. Therefore, in our main text, we report results only for 20, 40, and 60 TRs. Following the reviewer’s suggestion, we have also included decoding results for fMRI signal with 80TRs. We observed performance degradation across all metrics. However, where the drop is caused by the model’s limitation in handling longer sequences or by the lack of sufficient training data still needs further exploration. This question can be further investigated in future work when more data becomes available.\n\n|Length|BLEU-1|BLEU-2|BLEU-3|BLEU-4|ROUGE-F|ROUGE-P|ROUGE-R|BERTScore-F|BERTScore-P|BERTScore-R|\n|-|-|-|-|-|-|-|-|-|-|-|\n|20|25.4|10.5|4.7|2.6|23.4|22.6|24.6|46.3|45.7|46.9|\n|40|33.6|18.4|13.4|11.2|30.6|30.5|31.5|51|50.4|51.6|\n|60|38.3|23.4|18.3|15.9|36.4|37.6|36.2|54.4|53.8|55.1|\n|80|32.1|13.7|7.4|4.9|26.9|28|26.5|51|49.9|52.2|"}]

Author rebuttal:

Meta-review: This paper introduces a framework for decoding natural language text from fMRI signals that is specifically designed to handle long sequences. Current methods process entire fMRI sequences at once and experience performance degradation with longer inputs. To address this issue, the proposed method involves two main contributions. First, it uses a sequential decoding strategy in which the fMRI time series is divided into fixed-length segments. Each segment is decoded incrementally, and a "wrap-up" mechanism summarizes the output and passes it as a semantic prior to the next segment to maintain continuity. Second, the paper introduces a text-guided masking strategy within a masked autoencoder (MAE) framework for learning fMRI representations. This strategy focuses on semantically important time points. Experiments on the Narratives dataset demonstrate that the proposed method outperforms SOTA approaches, with a wider performance gap as the length of the decoded text increases.
The paper is interesting and convincing but makes slight overclaims. These have been identified and discussed with the reviewers and will be mitigated in the final version. Additionally, experiments will be presented on a second dataset, which is a significant advantage.
Overall, the authors did a good job of presenting and defending their strategy, and the performance gap with respect to SOTA is large enough to warrant publication. While the paper's message is not extremely sophisticated, it is certainly useful.
Decision: Accept (spotlight)