dataset-review/dataset-review huggingface dataset:
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding
ICML 2025

MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding
Weikang Qiu 1 Zheng Huang * 2 Haoyu Hu * 3 Aosong Feng 1 Yujun Yan 2 Rex Ying 1
Abstract
Decoding functional magnetic resonance imaging
(fMRI) signals into text has been a key challenge
in the neuroscience community, with the potential
to
advance
brain-computer
interfaces
and
uncover deeper insights into brain mechanisms.
However, existing approaches often struggle with
suboptimal predictive performance, limited task
variety, and poor generalization across subjects.
In response to this, we propose MindLLM, a
model designed for subject-agnostic and versatile
fMRI-to-text decoding. MindLLM consists of
an fMRI encoder and an off-the-shelf LLM. The
fMRI encoder employs a neuroscience-informed
attention mechanism, which is capable of accom-
modating subjects with varying input shapes and
thus achieves high-performance subject-agnostic
decoding. Moreover, we introduce Brain Instruc-
tion Tuning (BIT), a novel approach that enhances
the model’s ability to capture diverse semantic
representations from fMRI signals, facilitating
more versatile decoding. We evaluate MindLLM
on comprehensive fMRI-to-text benchmarks.
Results demonstrate that our model outperforms
the baselines, improving downstream tasks by
12.0%, unseen subject generalization by 24.5%,
and novel task adaptation by 25.0%. Furthermore,
the attention patterns in MindLLM provide inter-
pretable insights into its decision-making process.
Code is available at https://github.com/
Graph-and-Geometric-Learning/
MindLLM.
1. Introduction
Decoding human brain activity (fMRI) to text has sparked
significant interest within the neuroscience community
(Xia et al., 2024; Chen et al., 2023a; Luo et al., 2023;
*Equal contribution
1Yale University 2Dartmouth College
3University of Cambridge. Correspondence to: Weikang Qiu
<weikang.qiu@yale.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyri...
Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning
ICML 2025

Underestimated Privacy Risks for Minority Populations in Large Language
Model Unlearning
Rongzhe Wei 1 Mufei Li 1 Mohsen Ghassemi 2 Eleonora Kreaˇci´c 2 Yifan Li 3 Xiang Yue 4 Bo Li 5 6
Vamsi K. Potluru 2 Pan Li 1 Eli Chien 1
Abstract
Large Language Models (LLMs) embed sensitive,
human-generated data, prompting the need for un-
learning methods. Although certified unlearning
offers strong privacy guarantees, its restrictive as-
sumptions make it unsuitable for LLMs, giving
rise to various heuristic approaches typically as-
sessed through empirical evaluations. These stan-
dard evaluations randomly select data for removal,
apply unlearning techniques, and use membership
inference attacks (MIAs) to compare unlearned
models against models retrained without the re-
moved data. However, to ensure robust privacy
protections for every data point, it is essential to
account for scenarios in which certain data sub-
sets face elevated risks. Prior research suggests
that outliers, particularly including data tied to mi-
nority groups, often exhibit higher memorization
propensity which indicates they may be more dif-
ficult to unlearn. Building on these insights, we in-
troduce a complementary, minority-aware evalua-
tion framework to highlight blind spots in existing
frameworks. We substantiate our findings with
carefully designed experiments, using canaries
with personally identifiable information (PII) to
represent these minority subsets and demonstrate
that they suffer at least 20% higher privacy leak-
age across various unlearning methods, MIAs,
datasets, and LLM scales. Our proposed minority-
aware evaluation framework1 marks an essential
step toward more equitable and comprehensive
assessments of LLM unlearning efficacy.
1Georgia Institute of Technology 2J.P. Morgan AI Research
3Tsinghua University 4Carnegie Mellon University 5The Univer-
sity of Chicago 6University of Illinois Urbana-Champaign. Cor-
respondence to: Rongzhe Wei <rwei42@gatech.edu>, Pan Li
<panli@gatech.e...
Revisiting Non-Acyclic GFlowNets in Discrete Environments
ICML 2025

Revisiting Non-Acyclic GFlowNets in Discrete Environments
Nikita Morozov * 1 Ian Maksimov * 1 Daniil Tiapkin 2 3 Sergey Samsonov 1
Abstract
Generative Flow Networks (GFlowNets) are a
family of generative models that learn to sample
objects from a given probability distribution, po-
tentially known up to a normalizing constant. In-
stead of working in the object space, GFlowNets
proceed by sampling trajectories in an appro-
priately constructed directed acyclic graph en-
vironment, greatly relying on the acyclicity of
the graph. In our paper, we revisit the theory
that relaxes the acyclicity assumption and present
a simpler theoretical framework for non-acyclic
GFlowNets in discrete environments. Moreover,
we provide various novel theoretical insights re-
lated to training with fixed backward policies,
the nature of flow functions, and connections be-
tween entropy-regularized RL and non-acyclic
GFlowNets, which naturally generalize the re-
spective concepts and theoretical results from the
acyclic setting. In addition, we experimentally
re-examine the concept of loss stability in non-
acyclic GFlowNet training, as well as validate our
own theoretical findings.
1. Introduction
Generative Flow Networks (GFlowNets, Bengio et al., 2021)
are models that aim to sample discrete objects from distribu-
tions known proportionally up to a constant. They operate
by constructing an object through a sequence of stochastic
transitions defined by a forward policy. GFlowNets have
been successfully applied in various areas, starting from
molecule generation (Bengio et al., 2021; Shen et al., 2024;
Koziarski et al., 2024; Cretu et al., 2025) and biological
sequence design (Jain et al., 2022; Kim et al., 2024) to com-
binatorial optimization (Zhang et al., 2023a;b; Kim et al.,
*Equal contribution 1HSE University, Moscow, Russia 2CMAP
– CNRS – ´Ecole polytechnique – Institut Polytechnique de Paris,
91128, Palaiseau, France 3Universit´e Paris-Saclay, CNRS, LMO,
91405, Orsay, France.
Corr...
Categorical Schrödinger Bridge Matching
ICML 2025

Categorical Schr¨odinger Bridge Matching
Grigoriy Ksenofontov 1 2 Alexander Korotin 1 3
Abstract
The Schr¨odinger Bridge (SB) is a powerful frame-
work for solving generative modeling tasks such
as unpaired domain translation. Most SB-related
research focuses on continuous data space RD and
leaves open theoretical and algorithmic questions
about applying SB methods to discrete data, e.g,
on finite spaces SD. Notable examples of such
sets S are codebooks of vector-quantized (VQ)
representations of modern autoencoders, tokens
in texts, categories of atoms in molecules, etc. In
this paper, we provide a theoretical and algorith-
mic foundation for solving SB in discrete spaces
using the recently introduced Iterative Markovian
Fitting (IMF) procedure. Specifically, we theoreti-
cally justify the convergence of discrete-time IMF
(D-IMF) to SB in discrete spaces. This enables
us to develop a practical computational algorithm
for SB, which we call Categorical Schr¨odinger
Bridge Matching (CSBM). We show the perfor-
mance of CSBM via a series of experiments with
synthetic data and VQ representations of images.
The code of CSBM is available at this repository.
1
Introduction
The Schr¨odinger bridge (Schr¨odinger, 1931, SB) problem
has recently attracted the attention of the machine learning
community due to its relevance to modern challenges in gen-
erative modeling and unpaired learning. Recently, a variety
of methods have been proposed to solve SB in continuous
spaces; see (Gushchin et al., 2023b) for a recent survey.
One modern approach to solving SB is the Iterative Marko-
vian Fitting (IMF) framework (Peluchetti, 2023; Shi et al.,
2023; Gushchin et al., 2024b). Specifically, within this
framework, the discrete-time IMF procedure (Gushchin
1Skoltech,
Moscow,
Russia
2MIPT, Moscow,
Russia
3AIRI, Moscow,
Russia.
Correspondence to:
Grigoriy
Ksenofontov <g.ksenofontov@skoltech.ru>, Alexander Korotin
<a.korotin@skoltech.ru>.
Proceedings of the 42 nd International Conference on ...
Universal Neural Optimal Transport
ICML 2025

Universal Neural Optimal Transport
Jonathan Geuter 1 2 * Gregor Kornhardt 3 * Ingimar Tomasson 3 * Vaios Laschos 4
Abstract
Optimal Transport (OT) problems are a corner-
stone of many applications, but solving them
is computationally expensive. To address this
problem, we propose UNOT (Universal Neural
Optimal Transport), a novel framework capable
of accurately predicting (entropic) OT distances
and plans between discrete measures for a given
cost function. UNOT builds on Fourier Neural
Operators, a universal class of neural networks
that map between function spaces and that are
discretization-invariant, which enables our net-
work to process measures of variable resolutions.
The network is trained adversarially using a sec-
ond, generating network and a self-supervised
bootstrapping loss. We ground UNOT in an ex-
tensive theoretical framework. Through experi-
ments on Euclidean and non-Euclidean domains,
we show that our network not only accurately
predicts OT distances and plans across a wide
range of datasets, but also captures the geometry
of the Wasserstein space correctly. Furthermore,
we show that our network can be used as a state-
of-the-art initialization for the Sinkhorn algorithm
with speedups of up to 7.4×, significantly outper-
forming existing approaches.
1. Introduction
Optimal Transport (Villani, 2009; Peyr´e & Cuturi, 2019)
plays an increasing role in various areas in machine learn-
ing, such as domain adaptation (Courty et al., 2017), single-
cell genomics (Schiebinger et al., 2019), imitation learn-
ing (Dadashi et al., 2020), imaging (Schmitz et al., 2018),
dataset adaptation (Alvarez-Melis & Fusi, 2021), and signal
processing (Kolouri et al., 2017). Oftentimes, an entropic
*Equal contribution
1Harvard John A. Paulson School of En-
gineering and Applied Sciences 2Kempner Institute at Harvard
University 3Department of Mathematics, Technische Universit¨at
Berlin, Germany 4Weierstrass Institute, Berlin, Germany. Corre-
spondence to: Jonathan Geuter...
Benign Overfitting in Single-Head Attention
NeurIPS 2025

Benign Overfitting in Single-Head Attention
Roey Magen∗
Weizmann Institute of Science
roey.magen@weizmann.ac.il
Shuning Shang∗†
Princeton University
shuning@princeton.edu
Zhiwei Xu
University of Michigan
zhiweixu@umich.edu
Spencer Frei‡
UC Davis
sfrei@ucdavis.edu
Wei Hu§
University of Michigan
vvh@umich.edu
Gal Vardi§
Weizmann Institute of Science
gal.vardi@weizmann.ac.il
Abstract
The phenomenon of benign overfitting, where a trained neural network perfectly fits
noisy training data but still achieves near-optimal test performance, has been exten-
sively studied in recent years for linear models and fully-connected/convolutional
networks. In this work, we study benign overfitting in a single-head softmax
attention model, which is the fundamental building block of Transformers. We
prove that under appropriate conditions, the model exhibits benign overfitting in
a classification setting already after two steps of gradient descent. Moreover, we
show conditions where a minimum-norm/maximum-margin interpolator exhibits
benign overfitting. We study how the overfitting behavior depends on the signal-
to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of
signal and noise tokens, and prove that a sufficiently large SNR is both necessary
and sufficient for benign overfitting.
1
Introduction
Neural networks often exhibit a remarkable phenomenon, known as benign overfitting, where they
achieve a perfect fit to noisy training examples and still generalize well to unseen data [1, 2]. This
phenomenon contradicts classical wisdom in machine learning, and has become a central research
question in the theory of deep learning. Existing works on benign overfitting study under what
conditions the phenomenon occurs in different architectures. These works focus on linear models,
and on shallow fully-connected and convolutional neural networks.
In recent years, Transformers [3] have emerged as a leading neural network architecture, with
impactful applications ac...
Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
ICML 2025

Adjusting Model Size in Continual Gaussian Processes:
How Big is Big Enough?
Guiomar Pescador-Barrios 1 Sarah Filippi 1 Mark van der Wilk 2
Abstract
Many machine learning models require setting
a parameter that controls their size before train-
ing, e.g. number of neurons in DNNs, or inducing
points in GPs. Increasing capacity typically im-
proves performance until all the information from
the dataset is captured. After this point, compu-
tational cost keeps increasing, without improved
performance. This leads to the question “How big
is big enough?” We investigate this problem for
Gaussian processes (single-layer neural networks)
in continual learning. Here, data becomes avail-
able incrementally, and the final dataset size will
therefore not be known before training, prevent-
ing the use of heuristics for setting a fixed model
size. We develop a method to automatically ad-
just model size while maintaining near-optimal
performance. Our experimental procedure fol-
lows the constraint that any hyperparameters must
be set without seeing dataset properties, and we
show that our method performs well across di-
verse datasets without the need to adjust its hy-
perparameter, showing it requires less tuning than
others.
1. Introduction
Continual learning aims to train models when the data ar-
rives in a stream of batches, without storing data after it has
been processed, and while obtaining predictive performance
that is as high as possible at each point in time (Ring, 1997).
Selecting the size of the model is challenging in this set-
ting, since typical non-continual training procedures do this
by trial-and-error (cross-validation) using repeated training
runs, which is not possible under our requirement of not
storing any data. This is a crucial parameter to set well,
1Department of Mathematics, Imperial College London, UK
2Department of Computer Science, University of Oxford, UK.
Correspondence to: Guiomar Pescador-Barrios <glp22@ic.ac.uk>.
Proceedings of the 42 nd Int...
Large Language Models are Demonstration Pre-Selectors for Themselves
ICML 2025

Large Language Models are Demonstration Pre-Selectors for Themselves
Jiarui Jin 1 2 * † Yuwei Wu 3 * Haoxuan Li 4 Xiaoting He 5 Weinan Zhang 1 Yiming Yang 3 Yong Yu 1
Jun Wang 6 Mengyue YangB 7
Abstract
In-context learning (ICL) with large language
models (LLMs) delivers strong few-shot per-
formance by choosing few-shot demonstrations
from the entire training data. However, exist-
ing ICL methods, which rely on similarity or di-
versity scores to choose demonstrations, incur
high computational costs due to repeatedly re-
trieval from large-scale datasets for each query.
To this end, we propose FEEDER (FEw yet Es-
sential Demonstration prE-selectoR), a novel pre-
selection framework that identifies a core sub-
set of demonstrations containing the most repre-
sentative examples in the training data, tailored
to specific LLMs. To construct this subset, we
introduce the “sufficiency” and “necessity” met-
rics in the pre-selection stage and design a tree-
based algorithm to identify representative exam-
ples efficiently. Once pre-selected, this represen-
tative subset can effectively replace the full train-
ing data, improving efficiency while maintain-
ing comparable performance in ICL. Additionally,
our pre-selected subset also benefits fine-tuning
LLMs, where we introduce a bi-level optimization
method that enhances training efficiency without
sacrificing performance. Experiments with LLMs
ranging from 300M to 8B parameters show that
FEEDER can reduce training data size by over 20%
while maintaining performance and seamlessly in-
tegrating with various downstream demonstration
selection strategies in ICL.
* Equal contributions: Jiarui Jin and Yuwei Wu. † Work done
during Jiarui Jin’s visit at University College London. 1Shanghai
Jiao Tong University 2Xiaohongshu Inc. 3Carnegie Mellon Uni-
versity 4Peking University 5No Affiliation 6University College
London 7University of Bristol. Correspondence to: Mengyue Yang
<mengyue.yang@bristol.ac.uk>.
Proceedings of the 42 nd ...
Protein Structure Tokenization: Benchmarking and New Recipe
ICML 2025

Protein Structure Tokenization: Benchmarking and New Recipe
Xinyu Yuan * 1 2 Zichen Wang † 3 Marcus Collins 3 Huzefa Rangwala † ‡ 3
Abstract
Recent years have witnessed a surge in the devel-
opment of protein structural tokenization meth-
ods, which chunk protein 3D structures into dis-
crete or continuous representations. Structure to-
kenization enables the direct application of pow-
erful techniques like language modeling for pro-
tein structures, and large multimodal models to
integrate structures with protein sequences and
functional texts. Despite the progress, the capa-
bilities and limitations of these methods remain
poorly understood due to the lack of a unified
evaluation framework. We first introduce Struct-
TokenBench, a framework that comprehensively
evaluates the quality and efficiency of structure
tokenizers, focusing on fine-grained local sub-
structures rather than global structures, as typical
in existing benchmarks. Our evaluations reveal
that no single model dominates all benchmarking
perspectives. Observations of codebook under-
utilization led us to develop AminoAseed, a sim-
ple yet effective strategy that enhances codebook
gradient updates and optimally balances codebook
size and dimension for improved tokenizer utiliza-
tion and quality. Compared to the leading VQ-
VAE model ESM3, our method achieves an aver-
age of 6.31% performance improvement across
24 supervised tasks, with sensitivity and utiliza-
tion rates increased by 12.83% and 124.03%,
respectively. Source code and model weights
are
available
at
https://github.com/
KatarinaYuan/StructTokenBench.
*This work was completed while the author was an intern
at Amazon.
†Corresponding author.
‡Huzefa Rangwala is on
LOA as a Professor of Computer Science at George Mason
University.
This paper describes work performed at Ama-
zon.
1Mila - Quebec AI Institute
2University of Montreal
3Amazon.
Emails: Xinyu Yuan <xinyu.yuan@mila.quebec>,
Zichen
Wang
<wangzc921@gmail.com>,
Marcus
Collins
<collmr@...
AutoAL: Automated Active Learning with Differentiable Query Strategy Search
ICML 2025

AutoAL: Automated Active Learning with Differentiable Query Strategy Search
Yifeng Wang 1 Xueying Zhan 2 Siyu Huang 3
Abstract
As deep learning continues to evolve, the need
for data efficiency becomes increasingly impor-
tant. Considering labeling large datasets is both
time-consuming and expensive, active learning
(AL) provides a promising solution to this chal-
lenge by iteratively selecting the most informa-
tive subsets of examples to train deep neural net-
works, thereby reducing the labeling cost. How-
ever, the effectiveness of different AL algorithms
can vary significantly across data scenarios, and
determining which AL algorithm best fits a given
task remains a challenging problem. This work
presents the first differentiable AL strategy search
method, named AutoAL, which is designed on
top of existing AL sampling strategies. AutoAL
consists of two neural nets, named SearchNet
and FitNet, which are optimized concurrently un-
der a differentiable bi-level optimization frame-
work. For any given task, SearchNet and Fit-
Net are iteratively co-optimized using the la-
beled data, learning how well a set of candi-
date AL algorithms perform on that task. With
the optimal AL strategies identified, SearchNet
selects a small subset from the unlabeled pool
for querying their annotations, enabling efficient
training of the task model. Experimental results
demonstrate that AutoAL consistently achieves
superior accuracy compared to all candidate AL
algorithms and other selective AL approaches,
showcasing its potential for adapting and inte-
grating multiple existing AL methods across di-
verse tasks and domains. Code is available at:
https://github.com/haizailache999/AutoAL.
1Department of Electrical and Computer Engineering, Carnegie
Mellon University, Pittsburgh, USA 2Ray and Stephanie Lane
Computational Biology Department, Carnegie Mellon University,
Pittsburgh, USA 3Visual Computing Division, School of Com-
puting, Clemson University, Clemson, USA. Correspondence t...
QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
ICML 2025

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial
Runtime Reconfiguration
HamidReza Imani 1 Jiaxin Peng 1 Peiman Mohseni 2 Abdolah Amirany 1 Tarek El-Ghazawi 1
https://github.com/hamid-74/Multi-MoE
Abstract
The deployment of mixture-of-experts (MoE)
large language models (LLMs) presents signif-
icant challenges due to their high memory de-
mands.
These challenges become even more
pronounced in multi-tenant environments, where
shared resources must accommodate multiple
models, limiting the effectiveness of conventional
virtualization techniques. This paper addresses
the problem of efficiently serving multiple fine-
tuned MoE-LLMs on a single GPU. We propose a
serving system that employs similarity-based ex-
pert consolidation to reduce the overall memory
footprint by sharing similar experts across mod-
els. To ensure output quality, we introduce run-
time partial reconfiguration, dynamically replac-
ing non-expert layers when processing requests
from different models. As a result, our approach
achieves competitive output quality while main-
taining throughput comparable to serving a sin-
gle model, and incurs only a negligible increase
in time-to-first-token (TTFT). Experiments on a
server with a single NVIDIA A100 GPU (80GB)
using Mixtral-8x7B models demonstrate an 85%
average reduction in turnaround time compared to
NVIDIA’s multi-instance GPU (MIG). Further-
more, experiments on Google’s Switch Trans-
former Base-8 model with up to four variants
demonstrate the scalability and resilience of our
approach in maintaining output quality compared
to other model merging baselines, highlighting its
effectiveness.
1Department of Electrical and Computer Engineering, The
George Washington University 2Computer Science and Engineer-
ing Department, Texas A&M University. Correspondence to:
HamidReza Imani <hamidreza@gwu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the auth...