ICML 2025

MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding
Weikang Qiu 1 Zheng Huang * 2 Haoyu Hu * 3 Aosong Feng 1 Yujun Yan 2 Rex Ying 1
Abstract
Decoding functional magnetic resonance imaging
(fMRI) signals into text has been a key challenge
in the neuroscience community, with the potential
to
advance
brain-computer
interfaces
and
uncover deeper insights into brain mechanisms.
However, existing approaches often struggle with
suboptimal predictive performance, limited task
variety, and poor generalization across subjects.
In response to this, we propose MindLLM, a
model designed for subject-agnostic and versatile
fMRI-to-text decoding. MindLLM consists of
an fMRI encoder and an off-the-shelf LLM. The
fMRI encoder employs a neuroscience-informed
attention mechanism, which is capable of accom-
modating subjects with varying input shapes and
thus achieves high-performance subject-agnostic
decoding. Moreover, we introduce Brain Instruc-
tion Tuning (BIT), a novel approach that enhances
the model’s ability to capture diverse semantic
representations from fMRI signals, facilitating
more versatile decoding. We evaluate MindLLM
on comprehensive fMRI-to-text benchmarks.
Results demonstrate that our model outperforms
the baselines, improving downstream tasks by
12.0%, unseen subject generalization by 24.5%,
and novel task adaptation by 25.0%. Furthermore,
the attention patterns in MindLLM provide inter-
pretable insights into its decision-making process.
Code is available at https://github.com/
Graph-and-Geometric-Learning/
MindLLM.
1. Introduction
Decoding human brain activity (fMRI) to text has sparked
significant interest within the neuroscience community
(Xia et al., 2024; Chen et al., 2023a; Luo et al., 2023;
*Equal contribution
1Yale University 2Dartmouth College
3University of Cambridge. Correspondence to: Weikang Qiu
<weikang.qiu@yale.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyri...