Language Reconstruction with Brain Predictive Coding from fMRI Data

## Abstract

Abstract Many recent studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language.
However, there is a lack of neurological basis for how the semantic information embedded within brain signals can be used more effectively to guide language reconstruction.
The theory of predictive coding suggests that human brain naturally engages in continuously predicting future word representations that span multiple timescales.
This implies that the decoding of brain signals could potentially be associated with a predictable future.
To explore the predictive coding theory within the context of language reconstruction, this paper proposes a novel model PredFT for jointly modeling neural decoding and brain prediction.
It consists of a main decoding network for language reconstruction and a side network for predictive coding.
The side network obtains brain predictive coding representation from related brain regions of interest with a multi-head self-attention module.
This representation is fused into the main decoding network with cross-attention to facilitate the language models’ generation process.
Experiments are conducted on the largest naturalistic language comprehension fMRI dataset Narratives. PredFT achieves current state-of-the-art decoding performance with a maximum BLEU-1 score of 27.8 % percent 27.8 27.8\% 27.8 % .

## 1 Introduction

Reconstructing natural language from functional magnetic resonance imaging (fMRI) signals offers potential insights into understanding the formation of language in the human brain.
Recent studies have attempted to leverage brain signals with computational language models to generate coherent, naturally flowing languages [, , , ].
This advancement is achieved based on the ability of computational language models to craft coherent language, with brain responses serving as guidance in the generation process, thereby aligning the model’s output with human perception.
For example, Tang et al. [] used a GPT [] model to generate semantic candidates, and then brain signals are employed to select the content that is more aligned with the semantic content perceived by humans.
Xi et al. [] proposed to fuse the brain representation as the input for language model and achieves language reconstruction through sequence-to-sequence machine translation [].

Despite relatively good decoding quality, existing works often overlook how the natural language is encoded within human brain as well as its representation within language models, which is significant for understanding the language reconstruction process. Predictive coding [, , ] provides an attractive theory for an unified view of neural encoding and decoding. It suggests that human brain naturally makes predictions of future words over multiple timescales when receiving current phonetic stimuli.
Previous neuroscience studies [, ] have already evidenced such speech prediction in human brain through fMRI. Caucheteux et al. [] further investigated the correlation between language model representation and brain predictive coding. After confirming the activations of modern language models could be linearly mapped to brain responses, they demonstrated that such brain mapping would be enhanced by adding predictive content to language model input. These findings in neural encoding remind us that in the language reconstruction process, treating brain signals as inputs without considering temporal relationships related to the predictive process may not be sufficient. However, whether brain predictive coding could really help facilitate fMRI-to-text decoding and how to exploit the prediction in decoding still remain unexplored.

To investigate predictive coding from the perspective of decoding, we first conduct a preliminary experiment to analyze the capability of brain signals in predicting future content and their associative relationship with the representations of language models.
We verify the predictive coding ability of brain signals and identify six brain regions of interests that are most related to the predictive coding functions.
Based on the observations, we propose PredFT, abbreviation of FMRI-to-Text decoding with Predictive coding, for jointly modeling language reconstruction and brain predictive coding.
PredFT is an end-to-end model with a main decoding network for language reconstruction and a side network for providing brain predictive coding heuristics.
The main decoding network consists of an encoding model for spatial-temporal feature extraction and a Transformer [] variant for generation. The encoding model applies a 3D convolutional neural network [] to extract spatial features of fMRI and a finite impulse response model [] to compensate for the delay of blood-oxygen-level-dependent signal.
At the same time, the side network extracts and fuses six regions of interest related to brain predictive coding, and then builds mappings with the main decoding network through cross-attention mechanism.

Experiments are conducted on a large naturalistic language comprehension fMRI dataset Narratives [].
First, we present the overall decoding performance of PredFT.
We show that its decoding performance outperforms existing proposed methods in terms of a series of language similarity metrics.
Second, we explore whether the selection of regions of interest for the side network will affect the decoding performance of PredFT. This experiment aims to confirm that the brain predictive coding functions only originate in certain spatial regions from decoding perspective.
Last, we analyze the distance in predictive coding to better understand how the human brain makes predictions over multiple timescales and its influence on language reconstruction performance.

The main contributions of this paper can be summarized as follows: (i) To the best of our knowledge, we first investigate the brain predictive coding issue from the perspective of decoding brain signals into natural language.
(ii) We propose an end-to-end model PredFT for fMRI-to-text decoding, which features effectively utilizing brain predictive coding representation to improve decoding performance through a side network.
(iii) Experimental results show that PredFT benefits from the joint modeling of brain predictive coding and achieves state-of-the-art decoding performance.

## 2 Predictive Coding Verification

In this section, we elaborate on predictive coding in human brain by analyzing the correlation between brain responses stimulated by spoken words and activations of language model with the spoken words as input. Following previous study [], the brain score $R(\mathbf{X})=corr(f(\mathbf{X}),\mathbf{Y})$ is first defined, which measures the pearson correlation between language model activations $\mathbf{X}\in\mathbb{R}^{M\times D}$ and brain responses $\mathbf{Y}\in\mathbb{R}^{N\times V}$, where $f$ is a linear ridge regression model with $\ell_{2}$-regularization for linear mapping. $M$ and $N$ stand for the number of words and fMRI frames; $D$ and $V$ stand for the output dimension of language model and number of voxels in brain. Similar to [], prediction score $P_{(d,l)}(\mathbf{X})=R(\mathbf{X}\oplus\mathbf{X}^{d}_{l})-R(\mathbf{X})$ is proposed, which is the difference between brain score of the concatenation of current word with future words and brain score of the current word. $\mathbf{X}^{d}_{l}$ indicates the representation of $l$ continuous future heard words with prediction distance $d$ which measures the distance from current word to the last future heard word. Sub-figure (a) of Figure 1 shows an example of $\mathbf{X}^{5}_{2}$. Prediction score reflects the degree of predictive coding. A positive value suggests long-range prediction helps improve the correlation between language model activations and brain responses. In experiment, the output of eighth layer in GPT-2 [] is applied for $\mathbf{X}$. The brain score is averaged across subjects and computed within one fMRI frame, namely $N$ is set as 1. The prediction score is calculated with fixed window length $l=2$ and distance $d$ ranging from 0 to 11.

Figure: Figure 1: Example and results of the predictive coding verification experiment.
Refer to caption: x1.png

The sub-figure (b) in Figure 1 reports the prediction score across individuals with $95\%$ confidence intervals. Results show prediction score $P_{(d,l)}(\mathbf{X})$ first increases and peaks at $d=4$, then decreases as the prediction distance $d$ increases, and finally goes down below zero when the prediction distance comes to $d=10$.
We also conduct regions of interest (ROI) analysis. Six regions related to brain predictive coding, including superior temporal sulcus, angular gyrus, supramarginal gyrus, and opercular, triangular, and orbital part of the inferior frontal gyrus in the left hemisphere, are selected for experiments. Sub-figure (c) shows the prediction score of six ROIs across individuals with $95\%$ confidence intervals. The prediction distance is set at d = 4 for the best predictive performance, as reflected in sub-figure (b). All the selected ROIs show positive response to prediction words.

The above verification experiment highlights the correlation between language model representation and brain predictive coding. One possible explanation for the phenomenon is that both language models and brain predictive coding are similar in the objective of upcoming words prediction. This inspires us with the following motivation: While predictive coding has been verified from the perspective of encoding, could it help in decoding brain signals into natural language with auto-regressive generation? We propose PredFT for investigating the effectiveness of utilizing brain prediction in fMRI-to-text decoding. Details are introduced in the next section.

## 3 Methodology

We first formalize the fMRI-to-text decoding task.
Given a naturalistic language comprehension fMRI dataset $\mathcal{D}:=\{\left\langle F_{ij},U_{j}\right\rangle\}$, where $U_{j}$ is $j$-th fragment of word stimuli and $F_{ij}$ is the fMRI data collected while the $i$-th subject is hearing the content of $U_{j}$.
The fMRI-to-text decoding task aims to build a model $\mathcal{M}$ that decodes $U_{j}^{\prime}=\mathcal{M}(F_{ij})$ to maximize the similarity between $U_{j}$ and $U_{j}^{\prime}$, which is measured by a text similarity metric $D$.
Specifically, the word stimuli segment $U_{j}=\{u_{j}^{T},u_{j}^{T+1},\ldots,u_{j}^{T+k}\}$ is the auditory content presented to the subject from time step $T$, where each $u_{j}^{T}$ contains a series of corresponding word tokens.
Similarly, $F_{ij}:=\{f_{i,j}^{T},f_{i,j}^{T+1},\ldots,f_{i,j}^{T+k}\}$ consists of $k{+1}$ continuous fMRI images starting from time step $T$. Each $u_{j}$ matches $f_{i,j}$ at corresponding time step $t$.

We propose PredFT, which integrates brain predictive coding function in the language reconstruction process.
PredFT is denoted as $\mathcal{M}_{\theta,\phi}$, containing a main decoding network $\mathcal{M}_{\theta}$ and a side network $\mathcal{M}_{\phi}$ for brain predictive coding.
We first introduce the main decoding network $\mathcal{M}_{\theta}$ which can reconstruct accurate text from fMRI with computational language models.
Then we elaborate on the side network $\mathcal{M}_{\phi}$ which extracts and exploits brain predictive coding.
Finally, the fusion of brain predictive coding representation and the joint training of $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$ are introduced.

### 3.1 Main Decoding Network

As shown in Figure 2, the main decoding network $\mathcal{M}_{\theta}$ consists of an encoder $\mathcal{M}_{\theta_{\text{Enc}}}$ and a decoder $\mathcal{M}_{\theta_{\text{Dec}}}$.
The encoder $\mathcal{M}_{\theta_{\text{Enc}}}$ is stacked with a 3D-CNN module, a finite impulse response (FIR) model [], and a Transformer encoder [].
For each fMRI image $f_{i,j}^{T}\in\mathbb{R}^{w\times h\times d}$, where $w,h,d$ represents the width, height, and depth, voxel-level normalization is first applied. Details are described in Appendix A.
The fMRI image after normalization is denoted as $\hat{f}_{i,j}^{T}$,
which is then fed into the 3D-CNN module.
The 3D-CNN module contains $L$ layers of group normalization [], ReLU activation, and convolution layer [], with residual connection [].
The size of fMRI image $\hat{f}_{i,j}^{T}$ is progressively reduced by convolution layer and finally downsized to $\hat{f}_{i,j}^{T}\in\mathbb{R}^{w^{\prime}\times h^{\prime}\times d^{\prime}
\times c}$ where $c$ is the number of output channels.
A flatten layer and a linear layer are used to obtain a one-dimensional vector $x_{i,j}^{T}\in\mathbb{R}^{d_{m}}$ as the output of the 3D-CNN module.
After that, a FIR model $g_{t}$ is applied to compensate for the latency of blood-oxygen-level-dependent (BOLD) signal.
For $k{+1}$ continuous fMRI images with $t\in\{T,T{+1},\ldots,T{+k}\}$, the temporal transformation $g_{t}$ concatenates $k{-k^{*}}$ future frames to form the representation in time step $t$:

$$ $\displaystyle g_{t}:\mathbb{R}^{k\times d_{m}}$ $\displaystyle\rightarrow\mathbb{R}^{k^{*}\times(d_{m}(k-k^{*}))}$ (1) $\displaystyle x_{i,j}^{t}$ $\displaystyle\mapsto concat(x_{i,j}^{t},x_{i,j}^{t+1},\ldots,x_{i,j}^{t+(k-k^{ *})}),t\in\{T,T{+1},\ldots,T{+k^{*}}\}.$ $$

where $k-k^{*}$ is the number of delays.
Then a linear layer $\mathbf{W}_{ll}\in\mathbb{R}^{(d_{m}\dot{(}k-k^{*}))\times d_{m}}$ is used to fuse delayed brain responses and recover $x_{i,j}^{t}$ into its
original dimension $d_{m}$.
After learning the spatial features of fMRI images, the representations with learnable time positional embeddings, denoted as $\mathbf{H}^{0}_{\theta_{\text{Enc}}}\in\mathbb{R}^{k^{*}\times d_{m}}$, are sent into a Transformer encoder to capture temporal features within given intervals.
The output of the Transformer encoder is $\mathbf{H}^{P}_{\theta_{\text{Enc}}}=\mathcal{M}_{\theta_{\text{Enc}}}(\mathbf
{H}^{0}_{\theta_{\text{Enc}}})$, where $P$ is the number of Transformer encoder layers.

Figure: Figure 2: The general framework of PredFT. The italic type words in the input word sequence stand for the first heard word of each fMRI image while the bold words stand for the prediction words.
Refer to caption: x2.png

Finally, the output from $\mathcal{M}_{\theta_{\text{Enc}}}$, i.e., $\mathbf{H}^{P}_{\theta_{\text{Enc}}}$, is fed into the decoder $\mathcal{M}_{\theta_{\text{Dec}}}$.
$\mathcal{M}_{\theta_{\text{Dec}}}$ contains modules within a standard Transformer decoder consisting of masked multi-head self-attention, multi-head encoder-decoder attention, and feed-forward layer. Besides, additional masked multi-head cross-attention layer and feed-forward layer are designed for $\mathcal{M}_{\theta_{\text{Dec}}}$.
The cross-attention layer integrates brain predictive coding representations inherited from the side network.
More details will be introduced in Section 3.3 and here we just view it as an encoder-decoder attention layer.
The shifted right text $U_{j}^{r}$ is first tokenized and sent into a word embedding layer $\mathbf{W}_{\text{word}}$ to get representations $\mathbf{H}^{0}_{\theta_{\text{Dec}}}$. We denote the input of the $(l{+1})$-th self-attention layer as $\mathbf{H}^{l}_{\theta_{\text{Dec}}}$, so the self-attention is calculated by

$$ $\mathrm{Self\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{ l},\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{K}^{l},\mathbf{H}^{l}_{ \theta_{\text{Dec}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_ {\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{l}_{\theta_{\text{Dec}}} \mathbf{W}_{K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{l}_{\theta_{\text{Dec}}} \mathbf{W}_{V}^{l},$ (2) $$

where $\mathbf{W}_{Q}^{l},\mathbf{W}_{K}^{l},\mathbf{W}_{V}^{l}\in\mathbb{R}^{d_{m}
\times d_{k}}$ are the parameter matrices of projecting query, key, value in the $(l{+1})$-th corresponding layer (i.e. here is the self-attention layer) for simplicity. The encoder-decoder attention aims to integrate fMRI representations. It takes $\mathbf{H}^{l}_{\theta_{\text{Dec}}}$ as query and $\mathbf{H}^{P}_{\theta_{\text{Enc}}}$ for key and value:

$$ $\mathrm{ED\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l} ,\mathbf{H}^{P}_{\theta_{\text{Enc}}}\mathbf{W}_{K}^{l},\mathbf{H}^{P}_{\theta _{\text{Enc}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_{ \theta_{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{P}_{\theta_{\text{Enc}}} \mathbf{W}_{K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{P}_{\theta_{\text{Enc}}} \mathbf{W}_{V}^{l}.$ (3) $$

The attention masks for self-attention and encoder-decoder attention are the same as vanilla Transformer.
The final output of $\mathcal{M}_{\theta_{\text{Dec}}}$ is denoted as $\mathbf{H}^{Q}_{\theta_{\text{Dec}}}$ where $Q$ is the number of decoder layers.

### 3.2 Side Network for Predictive Coding

The idea of designing a side network $\mathcal{M}_{\phi}$ for brain predictive coding is motivated by predictive coding theory [, , ], which indicates the human brain naturally makes predictions about words over multiple timescales.
Since brain predictive coding has been verified from the perspective of encoding (the linear mapping between language models and brain responses), we hope to exploit it by training a neural network to well represent regions involved in prediction and fusing brain predictive coding representations in decoding.
A vanilla Transformer $\mathcal{M}_{\phi}$ with encoder $\mathcal{M}_{\phi_{\text{Enc}}}$ and decoder $\mathcal{M}_{\phi_{\text{Dec}}}$ is applied for brain predictive coding.
Specifically, it takes $R_{ij}:=\{r_{i,j}^{T},r_{i,j}^{T+1},\ldots,r_{i,j}^{T+k}\}$ as encoder input and $V_{j}^{r}:=\{$<$s$>$,v_{j}^{T},v_{j}^{T+1},\ldots,v_{j}^{T+k}\}$ as decoder input.
As shown in Figure 2, each $r_{i,j}^{T}\in\mathbb{R}^{d_{r}}$ in $R_{ij}$ extracted from $f_{i,j}^{T}$ in $F_{ij}$ is one-dimensional cerebral cortex data.
It stands for the concatenation of regions of interest (ROIs) related to brain predictive coding as verified in Section 2. Each $v_{j}^{T}$ in $V_{j}^{r}$ stands for the $l$ future words with prediction distance $d$ (recall the definition in Section 2) of $u_{j}^{T}$ in $U_{j}^{r}$ that human brain predicts.
The side network $\mathcal{M}_{\phi}$ is trained to learn the mapping from brain predictive responses to ground truth future words.

Figure: Figure 3: Illustration of attention masks. Grey color indicates mask. The cross-attention mask is transposed for simplicity of expression.
Refer to caption: x3.png

The ROI fusion layer is a fully connected feed-forward network that outputs the representation $r_{i,j}^{T}\in\mathbb{R}^{d_{m}}$.
The same FIR model in the main decoding network is applied to compensate for the delays of the BOLD signal.
Learnable time positional embedding is added to $r_{i,j}^{T}$ before entering into the Transformer encoder $\mathcal{M}_{\phi_{\text{Enc}}}$ with $M$ layers. The output of $\mathcal{M}_{\phi_{\text{Enc}}}$ is denoted as $\mathbf{H}^{M}_{\phi_{\text{Enc}}}$, which stands for the representation of brain predictive coding. $\mathbf{H}^{M}_{\phi_{\text{Enc}}}$ plays an essential role in PredFT, as it will be fused into the main decoding network to verify the effectiveness of brain predictive coding in the decoding task.
The Transformer decoder $\mathcal{M}_{\phi_{\text{Dec}}}$ follows the conventional practice of masked self-attention and encoder-decoder attention, which has been elaborated in Equation 2 and Equation 3 of Section 3.1, respectively.

### 3.3 Prediction Fusion and Joint Training

The subsection details how the brain predictive coding representation from the side network fuses into the main decoding network.
As shown in Figure 2, the brain predictive coding representation $\mathbf{H}^{M}_{\phi_{\text{Enc}}}$, which is the output of $\mathcal{M}_{\phi_{\text{Enc}}}$, plays as keys and values for the masked cross-attention in the main decoding network.
The queries of cross-attention layer are the output from the previous encoder-decoder attention layer and feed-forward layer.
The cross attention if formularized as:

$$ $\mathrm{Cross\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^ {l},\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{K}^{l},\mathbf{H}^{M}_{\phi_ {\text{Enc}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_{\theta _{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_ {K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{V }^{l}.$ (4) $$

The mask $\mathbf{M}_{\text{ca}}\in\mathbb{R}^{k_{t}\times k^{*}}$ of cross-attention is shown in Figure 3(a).
$k_{t}$ and $k^{*}$ are the numbers of input tokens and fMRI signals, respectively.
The cross-attention mask $\mathbf{M}_{\text{ca}}$ is designed in this way: for each token in the text fragment $u_{j}^{t}$, all the predictive coding representations after time step $t$ are allowed to attend, while previous representations are masked.

PredFT is trained in an end-to-end manner.
The main decoding network $\mathcal{M}_{\theta}$ and the side network $\mathcal{M}_{\phi}$ share the same word embedding layer $\mathbf{W}_{\text{word}}$, whose parameters are only updated with the gradient flow from $\mathcal{M}_{\theta}$ during training.
The training objective follows a left-to-right auto-regressive language modeling manner for both $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$. Specifically, following $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$ are two language model heads $\mathbf{W}_{\theta}$ and $\mathbf{W}_{\phi}$. The training loss for $\mathcal{M}_{\theta}$ is

$$ $\mathcal{L}_{\text{Main}}=-\sum_{t=1}^{n}\log P(y_{t}|y_{<t},U_{j}^{r};\theta),$ (5) $$

where $U_{j}^{r}$ is the input and $y_{t}$ is the generated token. Similarly, the training loss for $\mathcal{M}_{\phi}$ is

$$ $\mathcal{L}_{\text{Side}}=-\sum_{t=1}^{n}\log P(z_{t}|z_{<t},V_{j}^{r};\phi).$ (6) $$

The joint training of $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$ is to optimize
$\mathcal{L}=\mathcal{L}_{\text{Main}}+\lambda\mathcal{L}_{\text{Side}}$, where $\lambda$ is a hyper-parameter.

## 4 Experimental Settings and Results

Several experiments are conducted to (i) evaluate the decoding performance of PredFT (ii) detailedly analyze how brain predictive coding work in PredFT. First we briefly introduce the experimental settings, while more details are shown in Appendix A. Then the results of decoding performance, regions of interest selection analysis, and prediction distance analysis are displayed.
Besides, we investigate the decoding error distribution of different models in Appendix B and conduct ablation study in Appendix C.

**Table 1: Performance of different models in fMRI-to-text decoding. Length denotes to the length of continuous fMRI frames.**
| Length | Model | BLEU-N ($\%$) | ROUGE-1 ($\%$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$ |  |  |
| 10 | UniCoRN | 20.64 | 5.03 | 1.40 | 0.45 | 15.56 | 25.47 | 19.23 |
| PredFT | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53 |  |
| 20 | UniCoRN | 18.02 | 4.71 | 1.32 | 0.4 | 18.01 | 29.46 | 20.82 |
| PredFT | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09 |  |
| 40 | UniCoRN | 21.76 | 5.43 | 1.17 | 0.34 | 19.76 | 35.33 | 25.30 |
| PredFT | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96 |  |

**Table 2: The decoding performance of PredFT when different ROIs are selected for the side network. Random means randomly picking ROIs. Whole stands for using whole human cerebral cortex. BPC denotes to the ROIs related to brain predictive coding.**
| Length | ROIs Selection | BLEU-N ($\%$) | ROUGE-1 ($\%$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$ |  |  |
| 10 | None | 18.08 | 3.98 | 1.05 | 0.28 | 14.96 | 26.21 | 18.96 |
| Random | 16.83 | 3.04 | 0.63 | 0.13 | 16.71 | 19.23 | 17.28 |  |
| Whole | 21.99 | 4.51 | 0.83 | 0.25 | 17.36 | 22.83 | 19.43 |  |
| BPC | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53 |  |
| 20 | None | 20.37 | 3.86 | 1.03 | 0.19 | 17.42 | 22.15 | 19.45 |
| Random | 16.11 | 3.28 | 0.55 | 0.12 | 19.27 | 24.29 | 21.44 |  |
| Whole | 23.55 | 6.39 | 1.33 | 0.39 | 15.66 | 30.98 | 20.72 |  |
| BPC | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09 |  |
| 40 | None | 18.01 | 4.72 | 1.27 | 0.34 | 16.41 | 34.36 | 22.16 |
| Random | 19.71 | 5.01 | 1.22 | 0.39 | 20.02 | 29.61 | 24.55 |  |
| Whole | 24.67 | 5.81 | 1.14 | 0.39 | 20.53 | 29.46 | 24.16 |  |
| BPC | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96 |  |

### 4.1 Baselines and Evaluation Metrics

The decoding performance of PredFT is compared to current state-of-the-art model UniCoRN []. We don’t test [] and [] in experiments because their model settings aren’t compatible with the dataset splitting method or preprocessed version we applied. Details are discussed in Appendix A.

Automatic evaluation metrics including BLEU [] and ROUGE [] are applied to measure decoding performance of different models. BLEU measures the n-gram overlap between decoded content and ground truth. ROUGE-N computes the precision, recall, and F1-score based on the n-gram overlap. Specifically, BLEU-1/2/3/4 and ROUGE1-R/P/F are used for all experiments.

### 4.2 Decoding Performance

We compare the decoding performance of different models on automatic evaluation metrics. To further investigate the effect of fMRI sequence length on decoding model performance, experiments are separately conducted with fMRI sequence length of 10, 20, and 40, which equals to 15s, 30s, and 60s of fMRI with 1.5s TR.
To the best of our knowledge, we are the first to try decoding one-minute of fMRI data.
Results are shown in Table 1. Despite UniCoRN’s good performance in some evaluation index like ROUGE1-R, PredFT achieves the best overall performance on all three experiments with different fMRI sequence lengths. Specifically, it gets the highest BLEU-1 score of $27.8\%$ on decoding 40 continuous fMRI frames. From the relative low results of BLEU-2/3/4, we find all the models struggle to generate long accurate text. It’s still a challenging task to decode fMRI to text.
Furthermore, we don’t find clear evidence indicating that the fMRI sequence length has an impact on the performance of the decoding models from experiments.

### 4.3 Regions of Interest Selection

To better understand whether PredFT benefits from introducing brain predictive coding, we select different regions of interest (ROIs) for the side network to test their influence on decoding performance. Results are listed in Table 2. Specifically, four types of ROIs are selected: (a) “None” indicates no ROIs are chosen. The side network together with the cross-attention layer in the main decoding network is detached from PredFT under this setting. (b) “Random” means we randomly pick six ROIs apart from brain predictive coding regions. Specifically, G_and_S_cingul-Ant, G_and_S_subcentral, G_and_S_transv_frontopol, G_orbital, S_front_middle, S_subparietal are chosen in experiment. (c) “Whole” means the whole human cerebral cortex (74 ROIs in the used atlas, see Appendix A) is applied for brain prediction. (d) “BPC” denotes to the six ROIs related to brain predictive coding as verified in Section 2. BPC leads to the best performance while whole ROIs selection also performs well. However, both random selection and non-selection lead to poor decoding accuracy, and random selection causes worse performance in most cases. Two conclusions could be drawn from the above ROIs analysis. First, the predictive coding information only contains in specific regions of human brain. Second, brain predictive coding can be beneficial to decoding models with proper network architecture design (e.g. PredFT).

### 4.4 Prediction Distance Analysis

In this section, we investigate the influence of prediction distance to the decoding performance of PredFT. Experiments are conducted with a fixed window length $l=2$ and the influence of prediction distance is reflected through BLEU-1 score. Figure 4 shows the results with different fMRI sequence lengths. We notice a similar phenomenon as the predictive coding verification experiment in Section 2. The trend of BLEU-1 score first rises then falls as prediction distance $d$ increases. PredFT achieves the best performance with prediction distance around four. The prediction distance analysis highlights two findings: (i) The design of a side network in PredFT successfully fuses brain prediction for decoding. (ii) The influence of prediction distance in fMRI-to-text decoding is consistent with that in linearly mapping language models to brain response.

Figure: Figure 4: The influence of prediction distance to decoding performance.
Refer to caption: x4.png

**Table 3: Two cases of decoded content. Bold type words indicate key concepts or phrases.**
| Case1 | Truth: It was more real than any dream he had ever had in his life. He could still<br>hear and feel that sharp metal ripping explosion that searing wave of heat. He sat |
| --- | --- |
|  | UniCoRN: It’s than I just a said of a a the hand. You have I the the the you I I I in<br>to at the to to sit. |
|  | PredFT: It’s a more than normal just good dream about to said. He open the<br>eyes I have be her I like she and and and he and of under the next and I look the<br>platform |
| Case2 | Truth: He couldn’t shake the thought out of his mind. It persisted all through the<br>day until dinner. He was still brooding as he and Mary got ready for bed. Guy<br>dear. Hm oh no. Anything wrong |
|  | UniCoRN: And I know a Dean and a eyes it. And of the first and the and a to guy<br>him. I said I to to him out and And to in that in the end. |
|  | PredFT: He don’t know my girl you of the eyes but his girl sleep he and he said<br>and he said and the to the and and which I not wrong. But the Guy |

### 4.5 Case Study

Some fMRI-to-text decoding cases with fMRI sequence length equals to 10 are listed in Table 3. From the cases, we find PredFT successfully decodes some high-level semantic concepts and key words compared to UniCoRN. For example, in case1 PredFT conveys the meaning of “the best dream” while UniCoRN fails to. In case2, PredFT decodes the meaning of “he and Mary prepare to sleep”. Overall, PredFT performs better than UniCoRN.
However, despite the relative good automatic evaluation performance, all models struggle to decode generally accurate content, especially in (i) generating fluent and coherent sentences (ii) capturing fine-grained semantic meanings (iii) decoding terminology (e.g. name “Mary”) or complicated phrases (e.g. “sharp metal ripping explosion”).

## 5 Related Work

#### fMRI-to-Text Decoding.

Previous studies focused on aligning fMRI signal to a limited vocabulary of hundred tokens and performing word-level decoding [, , , ], or incorporating fMRI representation into sentence embedding and applying pairwise classification for sentence-level generation [, ]. Recently, researchers turned to powerful pretrained language models for open-vocabulary fMRI-to-text decoding. Tang et al. [] designed a pipeline model where the encoder is responsible for identifying the most possible word sequence among candidates generated by GPT-2 model with beam search. Xi et al. [] proposed a three-phase training framework UniCoRN which applies BART [] model for generation. Ye et al. [] proposed BrainLLM by concatenating fMRI embedding with word embedding as input prompt to fine-tune a Llama2 [] model.

#### Brain Predictive Coding.

Predictive coding theory [, , ] aims to propose a potential unifying theory for computational and cognitive neuroscience []. It was initially proposed as a neuroscientific theory [] and subsequently developed into its mathematical form of cortical responses []. Although originally formulated to investigate brain visual processing, it was also extended to language processing in human brain in previous work [, ]. Predictive coding suggests that human brain naturally makes predictions about future words and sentences when it perceives natural language stimuli. Such hypothesis has already been evidenced by correlating word or phonetic surprisal with fMRI or EEG [, , , ]. Caucheteux et al. [] further proved a predictive coding hierarchy in the human brain listening to speech by investigating the linear mapping between modern language models and brain responses.

## 6 Discussion

In this presented study, we focus on brain predictive coding theory and neural decoding. Specifically, we step further on the basis of current predictive coding theory by investigating its effectiveness in decoding fMRI signals into natural language. Predictive coding is first verified by linearly mapping the activations of current auto-regressive language model to brain responses. Then fMRI-to-text decoding model PredFT is proposed, which features utilizing a side network to capture and fuse brain prediction. Comprehensive experiments demonstrate the good decoding performance of PredFT benefits from integrating brain predictive coding.

While lots of studies have confirmed the representations drawn from auto-regressive language models are effective at predicting brain responses stimulated by hearing spoken words, the reasons behind such phenomenon are still controversial. A possible convincing explanation is both of the language models and human brain follow a next words prediction pattern. But Antonello and Huth [] questioned this hypothesis and claimed the language models are effective at predicting brain responses because they generally capture a wide variety of linguistic phenomena. In this paper, we don’t seek to find a proper explanation but try to investigate predictive coding from the perspective of decoding. This finding offers a novel view for future research in predictive coding.

Neural decoding is another popular research topic. Thanks to the development of deep learning, decoding non-invasive signals is becoming possible. However, despite the improvement in decoding performance in PredFT, we find it’s still extremely difficult to reconstruct natural language from fMRI signals during experiments. The reasons can be summarized as follows: First, the noise inherited from collecting fMRI data is a natural barrier to decoding. Second, different from fMRI-to-image decoding [, ] whose experimental setting is letting subjects look at pictures one by one with certain intervals, the fast spoken word rate isn’t compatible with the poor temporal resolution of fMRI data in fMRI-to-text decoding. So part of the brain responses are not recorded in fMRI. Such hypothesis has been evidenced through experiments in Appendix B. Building a dataset with high-resolution devices (e.g. 7T MRI scanner) will probably alleviate this problem.

The limitations of this work include: (i) Experiments are only conducted on Narratives. It’s better to try other datasets like [] or []. (ii) We apply cross-subject dataset splitting for all experiments. It’s better to supplement experiments on per-subject splitting to get more comprehensive comparisons. (iii) The window length of future words is set as $l=2$ in experiments due to the limited computational resources. We leave the exploration of future words window length to future work. (iv) Surprising content might become distraction in brain predictive coding. We leave it as future work to analyze.

## Acknowledgements

This research is supported by the National Natural Science Foundation of China (No.62106105), the CCF-Baidu Open Fund (No.CCF-Baidu202307), the CCF-Zhipu AI Large Model Fund (No.CCF-Zhipu202315), the Scientific Research Starting Foundation of Nanjing University of Aeronautics and Astronautics (No.YQR21022), and the High Performance Computing Platform of Nanjing University of Aeronautics and Astronautics.

## Appendix A Experimental Settings

#### Dataset and Splitting Method

All experiments are conducted on the largest naturalistic language comprehension fMRI dataset Narratives [], which contains recordings from 345 subjects listening to 27 diverse stories. Since the data collection process involves different machines, only fMRI data with $64\times 64\times 27$ voxels is considered, which results to 230 subjects. For the dataset preprocess method, we choose the fMRIPrep version. Xi et al. [] also applied fMRIPrep while Ye et al. [] applied the AFNI-smooth version.

How to split datasets for training and evaluation is a matter of debate in fMRI-to-text decoding []. Ye et al. [] and Tang et al. [] trained and evaluated models per subject. Xi et al. [] applied cross-subject splitting but has been identified to have data leakage issue []. To avoid data leakage and test model’s cross-subject generalization ability, we apply the splitting method proposed in [].

#### Implementation Details

The latest version of Destrieux atlas [] is applied for cortical parcellation, which leads to 74 regions per hemisphere. We use six regions of interests that have been proven in [] to contribute to brain prediction, including superior temporal sulcus, angular gyrus, supramarginal gyrus, and opercular, triangular, orbital part of the inferior frontal gyrus.

Voxel-level normalization is performed to raw fMRI data, which separately normalizes the values of each voxel over the time domain.This normalization highlights the relative activation of a specific voxel within given intervals. In the main decoding network, the numbers of Transformer encoders and decoders are set to $P{=4}$ and $Q=12$ respectively. For the side network, both are set as $M{=N{=6}}$. We apply the tokenizer in BART[] for our model. PredFT is trained over 20 epochs with an initial learning rate of 5e-4 which eventually decays to 1e-5. The hyper-parameter $\lambda$ for jointly training the main and side networks is set to 1 for fMRI sequence of length 10, and 0.5 for sequence of length 20 and 40. For baseline methods we strictly follow the settings in the proposed paper []. All experiments are conducted on NVIDIA A100-80G GPUs.

## Appendix B Decoding Error Analysis

In this section, we design an experiment that analyzes the positional distribution of incorrectly decoded words. Figure 5 is an example of two successive fMRI frames containing seven and four spoken words respectively. Three kinds of errors during decoding are defined: (i) The decoding model $\mathcal{M}$ fails to generate the correct word (e.g. “could” is incorrectly decoded as “should”). (ii) The decoding model $\mathcal{M}$ generates redundant words (e.g. repetitive “and”). (iii) Corresponding words are missing during decoding (e.g. “sharp metal ripping” is missing in output). Although some of the generated words are semantically consistent with the ground truth, we apply the strict exact match to facilitate automatic evaluation. Three position counting methods for corresponding errors are proposed: (i) If decoded word is wrong, position index of the corresponding truth word is marked as wrong. (ii) If decoded words are redundant, position index of the last matched truth word is marked as wrong. (iii) If decoded words are missing, position indices of all the missing truth words are marked as wrong. Since different fMRI frames contain different numbers of spoken words, the relative positions of incorrect words within each frame, namely the percentage of index (PosPCT), are considered. The error probability of one specific position is the proportion of errors at this position to the total number of errors.
Table (a) in Figure 5 illustrates the positional distribution of incorrectly decoded words in the example. The distribution is calculated at ten percentiles from $10\%$ to $100\%$. As some positions like $10\%$ or $90\%$ are minority in all statistical positions, we also add the error probabilities of the first and last $50\%$ respectively, as shown in table (b).

Figure: Figure 5: An example of the experiment for decoding error analysis. PosID and PosPCT stand for the word position index of truth and the percentage of index respectively.
Refer to caption: x5.png

Such experiment is conducted on UniCoRN and PredFT. Positional decoding error distribution and sum of error probability are analyzed. Results are shown in Figure 6. We find the error probability of last heard words in TR is significantly higher than words heard at the beginning. However, the error probabilities of decoding the first and last half of text are supposed to be the same in normal cases. This phenomenon leads to the hypothesis that the information of some heard words, especially the last few words in each TR, is lost in fMRI data. It’s caused by the discrete sampling feature of fMRI: Due to the constraints of MRI scanner in strength and speed of switching the magnetic gradients, the fMRI signal is sampled discretely with a fixed time interval called repetition time (TR) in order to achieve the balance between spacial and temporal resolution. The repetition time in fMRI-to-text decoding task is usually around two seconds. However, the average speaking rate of human is about three words per second. If pauses between sentences are excluded, the word rate within one sentence will increase to five per second. This feature of fMRI leads to the information loss problem in fMRI-to-text decoding: While brain responses of the first few heard words are recorded in one fMRI frame, information of the last heard words is lost due to the low temporal resolution, making decoding these words difficult. The latency of BOLD signal complicates the theoretical explanation of this phenomenon. But based on experimental results and previous study [] which indicates the latency of fMRI response is about six seconds, exactly an integer multiple of repetition time (1.5s or 2s), the hypothesis of information loss is reasonable.

From sub-figure (a), (b), (c) in Figure 6, we surprisingly find PredFT successfully reduces the error probability of the last few decoded words compared to UniCoRN. This implies the predictive coding information in brain could be utilized to alleviate the information loss, and such alleviation of information loss is closely related to the decoding accuracy.
To better illustrate the degree of information loss, we propose a novel index information loss slope $\varphi$ measuring the growth rate of error probability from the first half of decoded content to the last half,

$$ $\varphi=\frac{\sum_{i=6}^{10}p_{i}-\sum_{i=1}^{5}p_{i}}{0.5},$ (7) $$

where $p_{i}$ stands for the error probability of $i\%$ position. $\varphi$ is expected to be around zero, as the error probabilities of different positions are supposed to be the same without information loss. However, the $\varphi$ values of all compared models are high, indicating that all models suffer from information loss. PredFT successfully mitigates information loss to some extent. As shown in sub-figure (d), (e), (f) of Figure 6, the $\varphi$ score of PredFT is lower than compared models on all the three experiments with different fMRI sequence lengths.

Figure: Figure 6: Information loss of different models under different fMRI sequence lengths.
Refer to caption: x6.png

## Appendix C Ablation Study

Three aspects of ablation experiments are conducted to analyze PredFT. First we test whether the side network for brain prediction really improves decoding accuracy. The model PredFT without side network (PredFT w/o SideNet) is built with the same settings as PredFT during training except for only keeping the main decoding network (the cross-attention layers in main decoding network are removed). The results of this model’s decoding performance are listed in Table 2, same as not using ROIs in side network (“None” in the table). The performance of PredFT w/o SideNet is significantly worse than PredFT in all the three experiments with different fMRI sequence length. It also performs worse than UniCoRN under most cases, which might be attributed to the pretrained language model used in UniCoRN.

Besides, the decoding error distribution of PredFT w/o SideNet is counted to verify whether the side network helps alleviate information loss. As shown in Figure 6, the error distribution of PredFT w/o SideNet across different positions is similar to that of UniCoRN. The probability of decoding error increases as the word position moves backward within one fMRI frame, peaking at the position of the last word.
PredFT w/o SideNet severely suffers from information loss as shown in sub-figure (d), (e), (f) of Figure 6, with the highest information loss slope. The ablation experiments provide solid evidence on the effectiveness of the side network in PredFT.

We also test the influence of hyper-parameter $\lambda$ to decoding performance of PredFT. As shown in Table 4, four different $\lambda$ values ranging from 0.25 to 1 are tested in experiments with different fMRI sequence length. Empirically, PredFT achieves relatively good decoding accuracy with $\lambda=0.5$ in all experiments. For sequence length of 10, PredFT performs best when $\lambda$ is set as 1.

**Table 4: Decoding performance of PredFT under different hyper-parameter $\lambda$.**
| Length | $\lambda$ | BLEU-N ($\%$) | ROUGE-1 ($\%$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$ |  |  |
| 10 | 1 | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53 |
| 0.75 | 22.32 | 4.44 | 0.87 | 0.12 | 16.57 | 19.77 | 17.96 |  |
| 0.5 | 22.56 | 4.59 | 1.26 | 0.41 | 15.84 | 19.54 | 17.44 |  |
| 0.25 | 21.13 | 5.21 | 1.26 | 0.35 | 14.00 | 26.67 | 18.25 |  |
| 20 | 1 | 18.33 | 5.00 | 1.37 | 0.48 | 15.60 | 31.97 | 20.90 |
| 0.75 | 21.15 | 4.71 | 1.22 | 0.44 | 20.58 | 27.13 | 23.35 |  |
| 0.5 | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09 |  |
| 0.25 | 25.21 | 5.59 | 1.35 | 0.24 | 20.46 | 26.24 | 22.95 |  |
| 40 | 1 | 20.56 | 5.20 | 1.24 | 0.26 | 21.92 | 28.74 | 24.82 |
| 0.75 | 26.73 | 7.13 | 1.55 | 0.49 | 19.21 | 31.17 | 23.72 |  |
| 0.5 | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96 |  |
| 0.25 | 20.28 | 4.73 | 0.84 | 0.21 | 22.12 | 28.40 | 24.82 |  |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weaknesses. I would need to be convinced that majority of the claimed improvements in the model are not merely from a bias towards outputting high-frequency words, and thereby overfitting the chosen test metrics of BLEU and ROGUE, in order to change my score. Right now, I am fairly convinced that this is the case.
soundness: 2
strengths: The attempt to use hypothesized predictive coding representations to enable better text decoding is interesting.
summary: The paper describes a decoding method "PredFT" that uses a main decoding network and a side network to perform decoding from fMRI recordings of subjects listening to stories to text. The side network is responsible for obtaining predictive coding representations from specific brain regions and integrating them into the main network, enhancing language decoding. The authors claim that this integration leverages brain regions known for predictive functions (like the parietal-temporal-occipital areas) to better align brain signal decoding with anticipated semantic content. This is supported by results that have claimed the brain performs predictive coding during language stimulation.
weaknesses: My main concern is that the metric does not seem to produce even locally coherent text, which substantially damages the authors' claims that this method is an advancement over prior work, such as Tang et al., which uses an LM to guarantee local coherence. Consider the following example from the case study: "He don’t know my girl you of the eyes but his girl sleep he and he said and he said and the to the and and which I not wrong. But the Guy". Clearly, this has no meaning, and does not even obey basic local grammatical rules (e.g. "and and"). The problem seems to be that the model has merely learned repeat short, high-frequency words like "the", "he" and "and", which improves BLEU/ROGUE score but does not actually move forward towards the goal of better language decoding. I imagine if you just had the model repeatedly and randomly output words sampled from the top 100 most common English words that it would behave fairly similarly. My expectation is that a small percentage of the improvement in BLEU score is genuinely derived from brain signals, with most of the benefit deriving from this output bias. The unreasonably high 5.62 BLEU-3 score when compared to other methods is more of a red flag, because its pretty clear that the model is simply guessing every high frequency trigram in the English language.

 The paper is also quite difficult to read for no reason and pointlessly notational, for example when the self-attention equation is repeated three separate times in only slightly different ways.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In Section 3.3, the authors state that the decoder in the side network is abandoned during the inference stage. Could the authors provide a detailed explanation of why the decoder is discarded and discuss the potential impact of this decision on the overall performance and functionality of the model?

2. As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R. Could the authors analyze the potential reasons for this and discuss any factors that may have contributed to the lower performance in this specific model? For instance, how might the model's architecture, training process, or characteristics of the ROUGE1-R metric explain this discrepancy? Did the authors observe any patterns in the types of language constructs where PREDFT underperformed on ROUGE1-R?

3. As shown in Table 1, PREDFT without SideNet performs similarly to other methods, while the inclusion of SideNet leads to a significant performance improvement. Could the authors provide a detailed analysis of this phenomenon to explain how SideNet contributes to the model's enhanced performance?

4. Although the authors provide a detailed description of the hyperparameter selection, could they explain the rationale behind these choices? How do these choices relate to the model's performance or the underlying theory of predictive coding?

5. In the regions of interest selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs. Could the authors clarify whether there are other potential ROIs associated with predictive coding? If so, could they provide supporting neuroscience literature for the selection of BPC? Additionally, can the authors explain the process for selecting these particular ROIs and why they believe these are sufficient to demonstrate the effectiveness of their approach? Did the authors consider any other ROIs, and if so, why were those not included in the study?

6. Could the authors provide pseudocode for the method and an analysis of its time complexity to enhance the reproducibility of the article?

7. The results provided by the authors mostly only include the mean value. Could the authors include the variance and statistical test results in the experimental results?

8. In the methods section, some symbols are not defined. Could the authors compile a list of symbols used in the paper in an appendix to help readers understand better?
soundness: 3
strengths: 1. Integrating predictive coding theory into the decoding process offers a fresh perspective on reconstructing language from brain signals.
2. Experimental results demonstrate that PREDFT outperforms other methods across various evaluation metrics, showing significant improvements.
summary: The paper presents PREDFT (FMRI-to-Text Decoding with Predictive Coding), a novel framework that utilizes predictive coding to translate fMRI signals into continuous language. This approach combines a primary decoding network with an auxiliary network focused on capturing brain predictive coding, aiming to improve the accuracy of language reconstruction from brain signals. The authors conduct experiments on two established naturalistic language comprehension fMRI datasets, showing that PREDFT achieves state-of-the-art performance across multiple evaluation metrics.
weaknesses: 1. In Section 3.3, the authors state, 'During the inference stage, as illustrated in Figure 8, the decoder in the side network is abandoned.' However, they do not provide a detailed explanation of why the decoder is discarded or discuss the potential impact of this decision. It is recommended to elaborate on the rationale behind this choice and its implications on the overall performance and functionality of the model.

2. As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R. The authors should analyze the potential reasons for this and discuss any factors that may have contributed to the lower performance in this specific model. For instance, the model's architecture, training process, or characteristics of the ROUGE1-R metric that might explain the discrepancy. If the authors observed any patterns in the types of language constructs where PREDFT underperformed on ROUGE1-R.

3. As shown in Table 1, PREDFT without SideNet performs similarly to other methods. However, the inclusion of SideNet leads to a significant performance improvement. The authors should provide a detailed analysis of this phenomenon to explain how SideNet contributes to the model's enhanced performance.

4. Although the authors provide a detailed description of the hyperparameter selection, they do not explain the rationale behind these choices. How these choices relate to the model's performance or the underlying theory of predictive coding.

5. "In the regions of interests selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs, which appears somewhat limited. The paper does not clarify whether there are other potential ROIs associated with predictive coding, nor does it provide supporting neuroscience literature for the selection of BPC. It is recommended to either justify the choice of BPC with relevant references or explore additional ROIs to strengthen the study's validity. If authors can explain the process for selecting these particular ROIs and why authors believe these are sufficient to demonstrate the effectiveness of their approach. Additionally, if authors considered any other ROIs and why those were not included in the study.

6. It is recommended that the authors provide pseudocode for the method and an analysis of its time complexity to enhance the reproducibility of the article.

7. The results provided by the authors mostly only include the meanvalue. The experimental results should provide the mean, variance, and statistical test results.

8. In the methods section, some symbols are not defined. It is recommended that the authors compile a list of symbols used in the paper in an appendix to help readers understand better.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: (1) Why did the author only use BLEU and ROUGE in the experiment? Why doesn't the author use WER, METEOR, and BERTScore which is used in the Tang and MapGuide? BLEU and ROUGE both evaluate the matching degree of n-grams, which can easily lead to surface matching but semantic mismatch. METEOR and BERTScore can better reflect semantic similarity.
(2) Many of the methods compared by the author incorporate LLM, while the author's model is entirely trained with their own transformer. Does this result in the author's method being inferior to the baseline method in terms of semantic similarity?
(3) The author's method was inspired by predictive coding and validated it on LLM using a prediction score. But can the author's own model still observe the same phenomenon on the prediction score? I haven't seen the same experiment evaluating the author's own model.
(4) In some parts of the paper, fMRI is spell as FMRI.
soundness: 3
strengths: The author provided sufficient experiments to demonstrate the significance of his motivation.
summary: In the submission-6263, the authors  propose PREDFT (FMRI-to-Text decoding with Predictive coding) , which was inspired by predictive coding theory. This theory suggests that when humans listen to a certain speech, their subconscious brain predicts the words they may hear next. Then the author validated this theory through a prediction score. The verification method is to first calculate the correlation coefficient between the features extracted by LLM at the current location and the brain features, and then add the features of an upcoming text segment to the current location features, calculate the correlation coefficient again, and observe the changes in the correlation coefficient. The experimental results show that incorporating upcoming text features can increase the correlation coefficient between LLM features and brain features. Based on the above experimental results, the author designed their own model, which includes the side network to decode upcoming text. In the decoding of current text, the feature from the side network is used to incorporate the predictive coding theory into the method.
weaknesses: Although the author's explanation of motivation is very sufficient, I still have a few major questions about the author's method and list them in the questions part.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What would be the chance-level performance when reconstructing continuous language? Is there a baseline available for comparison? Additionally, what is the percentage of overlap between random ROIs and whole-brain voxels? Did the authors repeat the selection of random ROIs multiple times to ensure robustness, or did they only select a single set of random ROIs?
2. What is the rationale for using 4D volume data from the Narratives dataset while using 2D brain data from the Moth Radio Hour dataset? Since the Narratives dataset includes both smoothed and unsmoothed versions, along with brain masks to select activated voxels from the 4D volume, why did the authors make these choices regarding data representation?
3. There is no interpretation provided for the two encoders used in PREDFT. The authors could project these voxels onto brain maps to verify the quality of their encoders.
4. Figures 3, 4, 6, and 8 appear redundant. The authors could combine these into a single figure with a comprehensive caption, instead of presenting multiple, repetitive figures.
5. What does the y-axis represent in Figure 9?
5. Several major questions are raised in the weaknesses section.

Typos:

1. Line 35: Bhattasali et al. (2019); Wang et al. (2020); Affolter et al. (2020); Zouet al. (2021)  - > (Bhattasali et al. 2019; Wang et al. 2020; Affolter et al. 2020; Zouet al. 2021)
soundness: 3
strengths: 1. The motivation for using predictive coding in continuous language reconstruction is clear and well-explained.
2. The proposed approach aims to improve the reconstruction of narrative stories from fMRI brain data. This is a very interesting research area because reconstructing language is challenging due to the slowness of the hemodynamic response.
3. The authors compared the reconstruction performance using evaluation metrics against recent studies. Additionally, ablation studies were conducted on the proposed approach, with and without the predictive coding component.
summary: Recent brain decoding studies have demonstrated that speech perception can be decoded from fMRI recordings and subsequently reconstructed as continuous language. These studies reconstruct continuous language either from specific regions of interest (ROIs) or from the whole brain, using decoder-based language models like GPT-2. Additionally, recent predictive coding studies reveal that the human brain naturally engages in continuously predicting future words across multiple timescales. Building on recent linguistic brain decoding research and the predictive coding approach, this paper explores predictive coding theory in the context of continuous language reconstruction. To this end, the authors propose PREDFT (fMRI-to-Text decoding with Predictive Coding), which consists of a main decoding network and a side network (the predictive coding component). Experimental results on two naturalistic brain datasets (Moth Radio Hour and Narratives) indicate that PREDFT achieves superior decoding performance when comparing the actual story with the reconstructed story.
weaknesses: 1. There are several major weaknesses in this work, particularly concerning the evaluation of reconstruction results:
	- A major concern is that the current study (PREDFT) does not provide a clear evaluation of reconstruction results compared to the baseline paper by Tang et al. (2023).
	- For example, the authors did not evaluate the word rate in the generated narrative story. Since the fMRI data was captured while participants were listening to stories, each word has an onset and offset. Similarly, during decoding, what is the word rate predicted by the proposed model, and does this word rate match the actual word rate of the original stimuli? 
	- Therefore, comparing the reconstructed stimulus to the ground truth (i.e., the actual transcripts of the stimuli) would provide a good sense of whether the outputs are meaningful, as the dataset includes the ground truth of what words participants heard and when they heard them.

2. Furthermore, the authors performed decoding using either random selections of ROIs, the whole brain, or BPC, which includes language-related ROIs. However, prior studies have focused on specific ROIs, such as the language, prefrontal, and auditory association cortices. Therefore, it is unclear how the proposed method compares with prior methods. Since the authors' main research question revolves around how semantic information is embedded in brain signals to improve decoding, they should consider these ROIs, as they maintain a hierarchy of language processing.
	- The random selection of ROIs generally leads to low decoding performance. What are these random ROIs? Do they have any overlap with BPC ROIs?
	- Previous studies have conducted both quantitative and qualitative analyses, reporting what the stimulus decoded at each ROI, including language-related regions in both the left and right hemispheres, as well as using four evaluation metrics. However, this paper does not report any reconstructed stimulus in the main content, nor does it include analysis at the ROI level. Additionally, the authors only used two metrics, and throughout the paper, the focus is more on the scores rather than on the main reconstructed language results.

3. Although the authors report some results on predictive length and distance from the current word in Figure 1, there are no qualitative reconstruction results for these different predictive lengths and distances. What type of information is the model forecasting based on brain data? Is it syntactic information, such as nouns and verbs, or semantic content? This analysis is clearly missing from the paper.

4. All the figures lack detailed captions. The results presented in the figures are difficult to understand. For instance, what is the prediction score in each subplot of Figure 1? What does each line in the top plots represent? What does prediction distance "d" refer to? Without providing clear details in the figure captions or placing the figures appropriately in the text, it becomes challenging for readers to understand the content and what is being conveyed.

5. Since the authors use two encoders and two decoders in the proposed PREDFT, it is unclear which component is primarily responsible for reconstructing the language and which component provides the theme and narrative structure. It would be interesting if the authors reported the generated stimulus from individual components and from PREDFT as a whole, along with the performance metrics. This would help identify the shared and individual contributions of each component during language reconstruction.

Meta-review: This works aims to improve the SoTA of text decoding (language reconstruction) from fMRI with semantics.

Reviewers have raised a number of concerns concerning mainly the evaluation of the performance of the method arguing that, as is, the evidence of improved text decoding with semantic information is too limited.

Given this clear agreement between reviewers (3 out of 4 with strong domain expertise), the paper cannot be endorsed for publication at ICLR.

Decision: Reject