AIS-Bench/AIS-Bench huggingface dataset:
MONA: Myopic Optimization with Non-myopic Approval Can Mitigate
Multi-step Reward Hacking
Sebastian Farquhar * 1 Vikrant Varma * 1 David Lindner * 1 David Elson * 1
Caleb Biddulph 1 Ian Goodfellow 1 Rohin Shah a 1
Abstract
Future advanced AI systems may learn sophisti-
cated strategies through reinforcement learning
(RL) that humans cannot understand well enough
to safely evaluate. We propose a training method
which avoids agents learning undesired multi-step
plans that receive high reward (multi-step “reward
hacks”) even if humans are not able to detect that
the behaviour is undesired. The method, My-
opic Optimization with Non-myopic Approval
(MONA), works by combining short-sighted op-
timization with far-sighted reward. We demon-
strate that MONA can prevent multi-step reward
hacking that ordinary RL causes, even without
being able to detect the reward hacking and with-
out any extra information that ordinary RL does
not get access to. We study MONA empirically
in three settings which model different misalign-
ment failure modes including 2-step environments
with LLMs representing delegated oversight and
encoded reasoning and longer-horizon gridworld
environments representing sensor tampering.
1. Introduction
When training an agent with reinforcement learning (RL)
and imperfectly-specified rewards, the agent may engage
in “reward hacking”, where its behaviour is undesired but
achieves a high reward (Clark & Amodei, 2016; Amodei
et al., 2016). For example, large language models (LLMs)
trained with RL from human feedback (Christiano et al.,
2017) can become sycophantic, where an agent says what
users likely want to hear (Sharma et al., 2023).
Currently, most reward hacking is addressed via “patching”:
noticing bad behavior and changing the reward to stop in-
1Google DeepMind ∗Core contributor aSenior author.
Correspondence to: dlindner@google.com, sebfar@google.com.
See arxiv.org/abs/2501.13011 for an extended version.
Proceedings of the 42 st International Confer...
Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Meera Hahn * 1 Wenjun Zeng 1 Nithish Kannen 1 Rich Galt 1 Kartikeya Badola 1 Been Kim 1 Zi Wang * 1
Abstract
User prompts for generative AI models are often
underspecified, leading to a misalignment be-
tween the user intent and models’ understanding.
As a result, users commonly have to painstakingly
refine their prompts. We study this alignment
problem in text-to-image (T2I) generation and
propose a prototype for proactive T2I agents
equipped with an interface to (1) actively ask
clarification questions when uncertain, and (2)
present their uncertainty about user intent as an
understandable and editable belief graph. We
build simple prototypes for such agents and
propose a new scalable and automated evaluation
approach using two agents, one with a ground
truth intent (an image) while the other tries to
ask as few questions as possible to align with
the ground truth.
We experiment over three
image-text datasets: ImageInWords (Garg et al.,
2024), COCO (Lin et al., 2014) and DesignBench,
a benchmark we curated with strong artistic
and design elements.
Experiments over the
three datasets demonstrate the proposed T2I
agents’ ability to ask informative questions and
elicit crucial information to achieve successful
alignment with at least 2 times higher VQAS-
core (Lin et al., 2024) than the standard T2I
generation.
Moreover, we conducted human
studies and observed that at least 90% of
human subjects found these agents and their
belief graphs helpful for their T2I workflow,
highlighting the effectiveness of our approach.
Code and DesignBench can be found at https:
//github.com/google-deepmind/
proactive_t2i_agents.
*Equal contribution
1Google DeepMind. Correspondence to:
Zi Wang <wangzi@google.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
Figure 1. Our proactive T2I agent clarifies a user prompt with
...
Published as a conference paper at ICLR 2025
LEARNING CHAOS IN A LINEAR WAY
Xiaoyuan Cheng1,†
Yi He1,†
Yiming Yang1,2
Xiao Xue1
Sibo Cheng3
Daniel Giles4
Xiaohang Tang2
Yukun Hu1,*
1Dynamic Systems, University College London, United Kingdom
2Statistical Science, University College London, United Kingdom
3CEREA, ENPC and EDF R&D, Institut Polytechnique de Paris, France
4Computer Science, University College London, United Kingdom
† Equal contribution
* Corresponding author (yukun.hu@ucl.ac.uk)
ABSTRACT
Learning long-term behaviors in chaotic dynamical systems, such as turbulent
flows and climate modelling, is challenging due to their inherent instability and
unpredictability. These systems exhibit positive Lyapunov exponents, which signif-
icantly hinder accurate long-term forecasting. As a result, understanding long-term
statistical behavior is far more valuable than focusing on short-term accuracy.
While autoregressive deep sequence models have been applied to capture long-term
behavior, they often lead to exponentially increasing errors in learned dynamics.
To address this, we shift the focus from simple prediction errors to preserving an
invariant measure in dissipative chaotic systems. These systems have attractors,
where trajectories settle, and the invariant measure is the probability distribution on
attractors that remains unchanged under dynamics. Existing methods generate long
trajectories of dissipative chaotic systems by aligning invariant measures, but it is
not always possible to obtain invariant measures for arbitrary datasets. We propose
the Poincar´e Flow Neural Network (PFNN), a novel operator learning framework
designed to capture behaviors of chaotic systems without any explicit knowledge of
the invariant measure. PFNN employs an auto-encoder to map the chaotic system
to a finite-dimensional feature space, effectively linearizing the chaotic evolution.
It then learns the linear evolution operators to match the physical dynamics by ad-
dressing two ...
Towards ﬂexible perception with visual memory
Robert Geirhos * 1 Priyank Jaini * 1 Austin Stone * 1 Sourabh Medapati * 1
Xi Yi 1 George Toderici 1 Abhijit Ogale 1 Jonathon Shlens 1
Abstract
Training a neural network is a monolithic en-
deavor, akin to carving knowledge into stone:
once the process is completed, editing the knowl-
edge in a network is hard, since all information is
distributed across the network’s weights. We here
explore a simple, compelling alternative by mar-
rying the representational power of deep neural
networks with the ﬂexibility of a database. De-
composing the task of image classiﬁcation into
image similarity (from a pre-trained embedding)
and search (via fast nearest neighbor retrieval
from a knowledge database), we build on well-
established components to construct a simple and
ﬂexible visual memory that has the following key
capabilities: (1.) The ability to ﬂexibly add data
across scales: from individual samples all the way
to entire classes and billion-scale data; (2.) The
ability to remove data through unlearning and
memory pruning; (3.) An interpretable decision-
mechanism on which we can intervene to control
its behavior. Taken together, these capabilities
comprehensively demonstrate the beneﬁts of an
explicit visual memory. We hope that it might
contribute to a conversation on how knowledge
should be represented in deep vision models—
beyond carving it in “stone” weights.
1. Introduction
In the pretty diagrams on ”Intro to Machine Learning” slides,
an ideal ML workﬂow looks like this: Data collection, pre-
processing, choosing a model, training, evaluation, deploy-
ment. Happy ending—the model is deployed, the users love
it, and one can ﬁnally go on that well-deserved vacation and
catch up on the latest AGI memes.
Until, of course, the enemy of any ideal world sets in: reality.
1Google DeepMind. Correspondence to: Robert Geirhos <last-
name@google.com>, Priyank Jaini <pjaini@google.com>.
Proceedings of the 42 nd International Confe...
Beyond Induction Heads:
In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Gouki Minegishi 1 Hiroki Furuta 1 Shohei Taniguchi 1 Yusuke Iwasawa 1 Yutaka Matsuo 1
Abstract
Transformer-based language models exhibit In-
Context Learning (ICL), where predictions are
made adaptively based on context. While prior
work links induction heads to ICL through a sud-
den jump in accuracy, this can only account for
ICL when the answer is included within the con-
text. However, an important property of practical
ICL in large language models is the ability to
meta-learn how to solve tasks from context, rather
than just copying answers from context; how such
an ability is obtained during training is largely un-
explored. In this paper, we experimentally clarify
how such meta-learning ability is acquired by ana-
lyzing the dynamics of the model’s circuit during
training. Specifically, we extend the copy task
from previous research into an In-Context Meta
Learning setting, where models must infer a task
from examples to answer queries. Interestingly, in
this setting, we find that there are multiple phases
in the process of acquiring such abilities, and that
a unique circuit emerges in each phase, contrast-
ing with the single-phases change in induction
heads. The emergence of such circuits can be
related to several phenomena known in large lan-
guage models, and our analysis lead to a deeper
understanding of the source of the transformer’s
ICL ability.
1. Introduction
Transformer-based language models (Vaswani et al., 2017)
show an intriguing ability to perform In-Context Learning
(ICL) (Brown et al., 2020; Xie et al., 2021; Garg et al., 2022;
Dong et al., 2024). ICL is the ability to predict the response
to a query based on context without any additional weight
updates. A widely adopted application of ICL is few-shot
*Equal contribution 1The University of Tokyo. Correspondence
to: Gouki Minegishi <minegishi@weblab.t.u-tokyo.ac.jp>.
Proceedings of the 42 nd International C...
Learning conformational ensembles of proteins based
on backbone geometry
Nicolas Wolf ∗1,2,3
Leif Seute ∗2,1,3
Vsevolod Viliuga 4,1
Simon Wagner 3
Jan Stühmer 2,5
Frauke Gräter 1,2,3
1Max Planck Institute for Polymer Research, Mainz, Germany
2Heidelberg Institute for Theoretical Studies, Heidelberg, Germany
3IWR, Heidelberg University, Heidelberg, Germany
4SciLifeLab and DBB at Stockholm University, Stockholm, Sweden
5IAR, Karlsruhe Institute of Technology, Karlsruhe, Germany
Abstract
Deep generative models have recently been proposed for sampling protein confor-
mations from the Boltzmann distribution, as an alternative to often prohibitively
expensive Molecular Dynamics simulations. However, current state-of-the-art ap-
proaches rely on fine-tuning pre-trained folding models and evolutionary sequence
information, limiting their applicability and efficiency, and introducing potential
biases. In this work, we propose a flow matching model for sampling protein
conformations based solely on backbone geometry – BBFlow. We introduce a
geometric encoding of the backbone equilibrium structure as input and propose to
condition not only the flow but also the prior distribution on the respective equilib-
rium structure, eliminating the need for evolutionary information. The resulting
model is orders of magnitudes faster than current state-of-the-art approaches at
comparable accuracy, is transferable to multi-chain proteins, and can be trained
from scratch in a few GPU days. In our experiments, we demonstrate that the
proposed model achieves competitive performance with reduced inference time,
across not only an established benchmark of naturally occurring proteins but also
de novo proteins, for which evolutionary information is scarce or absent. BBFlow
is available at https://github.com/graeter-group/bbflow.
1
Introduction
In recent years, the field of protein structure prediction has been revolutionized by geometric deep
learning [19, 4, 27]. Jumper et al. [19] introduced A...
Flexibility-conditioned protein structure design with flow matching
Vsevolod Viliuga * 1 2 3 Leif Seute * 1 3 4 Nicolas Wolf 1 3 4 Simon Wagner 4
Arne Elofsson 2 Jan Stühmer 1 5 Frauke Gräter 3 1 4
Abstract
Recent advances in geometric deep learning and
generative modeling have enabled the design of
novel proteins with a wide range of desired proper-
ties. However, current state-of-the-art approaches
are typically restricted to generating proteins with
only static target properties, such as motifs and
symmetries. In this work, we take a step towards
overcoming this limitation by proposing a frame-
work to condition structure generation on flexibil-
ity, which is crucial for key functionalities such
as catalysis or molecular recognition. We first in-
troduce BackFlip, an equivariant neural network
for predicting per-residue flexibility from an in-
put backbone structure. Relying on BackFlip, we
propose FliPS, an SE(3)-equivariant conditional
flow matching model that solves the inverse prob-
lem, that is, generating backbones that display a
target flexibility profile. In our experiments, we
show that FliPS is able to generate novel and di-
verse protein backbones with the desired flexibil-
ity, verified by Molecular Dynamics (MD) simula-
tions. FliPS and BackFlip are available at https:
//github.com/graeter-group/flips.
1. Introduction
The past few years have arguably marked the golden age
of de novo protein design. Rapid advancement of deep
learning-based approaches and their application to the pro-
tein structure prediction problem have enabled an unprece-
dented level of modeling accuracy (Jumper et al., 2021;
Dauparas et al., 2021; Lin et al., 2022). This, in turn, paved
*Equal contribution 1Heidelberg Institute for Theoretical Stud-
ies, Heidelberg, Germany 2Dept. of Biochemistry and Biophysics
at Stockholm University and Science for Life Laboratory, Stock-
holm, Sweden 3Max Planck Institute for Polymer Research, Mainz,
Germany 4IWR, Heidelberg University, Heidel...
Attention-based clustering
Rodrigo Maulen-Soto∗
Sorbonne Université, LPSM
Paris, France
Pierre Marion
EPFL, Institute of Mathematics
Lausanne, Switzerland
Claire Boyer∗
Laboratoire de Mathématiques d’Orsay
Université Paris Saclay
Institut Universitaire de France
Abstract
Transformers have emerged as a powerful neural network architecture capable of
tackling a wide range of learning tasks. In this work, we provide a theoretical anal-
ysis of their ability to automatically extract structure from data in an unsupervised
setting. In particular, we demonstrate their suitability for clustering when the input
data is generated from a Gaussian mixture model. To this end, we study a simplified
two-head attention layer and define a population risk whose minimization with
unlabeled data drives the head parameters to align with the true mixture centroids.
This phenomenon highlights the ability of attention-based layers to capture underly-
ing distributional structure. We further examine an attention layer with key, query,
and value matrices fixed to the identity, and show that, even without any trainable
parameters, it can perform in-context quantization, revealing the surprising capacity
of transformer-based methods to adapt dynamically to input-specific distributions.
1
Introduction
Attention-based models (Bahdanau et al., 2015), in particular Transformers (Vaswani et al., 2017),
have achieved state-of-the-art performance across a wide range of learning tasks. These include
applications in natural language processing (Devlin et al., 2018; Bubeck et al., 2023; Luong et al.,
2015; Bahdanau et al., 2016) and computer vision (Dosovitskiy et al., 2020; Liu et al., 2021;
Ramachandran et al., 2019). The success of the attention mechanism has been linked to its ability to
capture long-range relationships in input sequences (Bahdanau et al., 2015; Vaswani et al., 2017).
They do this by computing pairwise dependencies between tokens based on learned projections,
without regard to the t...
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding
Weikang Qiu 1 Zheng Huang * 2 Haoyu Hu * 3 Aosong Feng 1 Yujun Yan 2 Rex Ying 1
Abstract
Decoding functional magnetic resonance imaging
(fMRI) signals into text has been a key challenge
in the neuroscience community, with the potential
to
advance
brain-computer
interfaces
and
uncover deeper insights into brain mechanisms.
However, existing approaches often struggle with
suboptimal predictive performance, limited task
variety, and poor generalization across subjects.
In response to this, we propose MindLLM, a
model designed for subject-agnostic and versatile
fMRI-to-text decoding. MindLLM consists of
an fMRI encoder and an off-the-shelf LLM. The
fMRI encoder employs a neuroscience-informed
attention mechanism, which is capable of accom-
modating subjects with varying input shapes and
thus achieves high-performance subject-agnostic
decoding. Moreover, we introduce Brain Instruc-
tion Tuning (BIT), a novel approach that enhances
the model’s ability to capture diverse semantic
representations from fMRI signals, facilitating
more versatile decoding. We evaluate MindLLM
on comprehensive fMRI-to-text benchmarks.
Results demonstrate that our model outperforms
the baselines, improving downstream tasks by
12.0%, unseen subject generalization by 24.5%,
and novel task adaptation by 25.0%. Furthermore,
the attention patterns in MindLLM provide inter-
pretable insights into its decision-making process.
Code is available at https://github.com/
Graph-and-Geometric-Learning/
MindLLM.
1. Introduction
Decoding human brain activity (fMRI) to text has sparked
significant interest within the neuroscience community
(Xia et al., 2024; Chen et al., 2023a; Luo et al., 2023;
*Equal contribution
1Yale University 2Dartmouth College
3University of Cambridge. Correspondence to: Weikang Qiu
<weikang.qiu@yale.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyri...
Underestimated Privacy Risks for Minority Populations in Large Language
Model Unlearning
Rongzhe Wei 1 Mufei Li 1 Mohsen Ghassemi 2 Eleonora Kreaˇci´c 2 Yifan Li 3 Xiang Yue 4 Bo Li 5 6
Vamsi K. Potluru 2 Pan Li 1 Eli Chien 1
Abstract
Large Language Models (LLMs) embed sensitive,
human-generated data, prompting the need for un-
learning methods. Although certified unlearning
offers strong privacy guarantees, its restrictive as-
sumptions make it unsuitable for LLMs, giving
rise to various heuristic approaches typically as-
sessed through empirical evaluations. These stan-
dard evaluations randomly select data for removal,
apply unlearning techniques, and use membership
inference attacks (MIAs) to compare unlearned
models against models retrained without the re-
moved data. However, to ensure robust privacy
protections for every data point, it is essential to
account for scenarios in which certain data sub-
sets face elevated risks. Prior research suggests
that outliers, particularly including data tied to mi-
nority groups, often exhibit higher memorization
propensity which indicates they may be more dif-
ficult to unlearn. Building on these insights, we in-
troduce a complementary, minority-aware evalua-
tion framework to highlight blind spots in existing
frameworks. We substantiate our findings with
carefully designed experiments, using canaries
with personally identifiable information (PII) to
represent these minority subsets and demonstrate
that they suffer at least 20% higher privacy leak-
age across various unlearning methods, MIAs,
datasets, and LLM scales. Our proposed minority-
aware evaluation framework1 marks an essential
step toward more equitable and comprehensive
assessments of LLM unlearning efficacy.
1Georgia Institute of Technology 2J.P. Morgan AI Research
3Tsinghua University 4Carnegie Mellon University 5The Univer-
sity of Chicago 6University of Illinois Urbana-Champaign. Cor-
respondence to: Rongzhe Wei <rwei42@gatech.edu>, Pan Li
<panli@gatech.e...
Revisiting Non-Acyclic GFlowNets in Discrete Environments
Nikita Morozov * 1 Ian Maksimov * 1 Daniil Tiapkin 2 3 Sergey Samsonov 1
Abstract
Generative Flow Networks (GFlowNets) are a
family of generative models that learn to sample
objects from a given probability distribution, po-
tentially known up to a normalizing constant. In-
stead of working in the object space, GFlowNets
proceed by sampling trajectories in an appro-
priately constructed directed acyclic graph en-
vironment, greatly relying on the acyclicity of
the graph. In our paper, we revisit the theory
that relaxes the acyclicity assumption and present
a simpler theoretical framework for non-acyclic
GFlowNets in discrete environments. Moreover,
we provide various novel theoretical insights re-
lated to training with fixed backward policies,
the nature of flow functions, and connections be-
tween entropy-regularized RL and non-acyclic
GFlowNets, which naturally generalize the re-
spective concepts and theoretical results from the
acyclic setting. In addition, we experimentally
re-examine the concept of loss stability in non-
acyclic GFlowNet training, as well as validate our
own theoretical findings.
1. Introduction
Generative Flow Networks (GFlowNets, Bengio et al., 2021)
are models that aim to sample discrete objects from distribu-
tions known proportionally up to a constant. They operate
by constructing an object through a sequence of stochastic
transitions defined by a forward policy. GFlowNets have
been successfully applied in various areas, starting from
molecule generation (Bengio et al., 2021; Shen et al., 2024;
Koziarski et al., 2024; Cretu et al., 2025) and biological
sequence design (Jain et al., 2022; Kim et al., 2024) to com-
binatorial optimization (Zhang et al., 2023a;b; Kim et al.,
*Equal contribution 1HSE University, Moscow, Russia 2CMAP
– CNRS – ´Ecole polytechnique – Institut Polytechnique de Paris,
91128, Palaiseau, France 3Universit´e Paris-Saclay, CNRS, LMO,
91405, Orsay, France.
Corr...
Categorical Schr¨odinger Bridge Matching
Grigoriy Ksenofontov 1 2 Alexander Korotin 1 3
Abstract
The Schr¨odinger Bridge (SB) is a powerful frame-
work for solving generative modeling tasks such
as unpaired domain translation. Most SB-related
research focuses on continuous data space RD and
leaves open theoretical and algorithmic questions
about applying SB methods to discrete data, e.g,
on finite spaces SD. Notable examples of such
sets S are codebooks of vector-quantized (VQ)
representations of modern autoencoders, tokens
in texts, categories of atoms in molecules, etc. In
this paper, we provide a theoretical and algorith-
mic foundation for solving SB in discrete spaces
using the recently introduced Iterative Markovian
Fitting (IMF) procedure. Specifically, we theoreti-
cally justify the convergence of discrete-time IMF
(D-IMF) to SB in discrete spaces. This enables
us to develop a practical computational algorithm
for SB, which we call Categorical Schr¨odinger
Bridge Matching (CSBM). We show the perfor-
mance of CSBM via a series of experiments with
synthetic data and VQ representations of images.
The code of CSBM is available at this repository.
1
Introduction
The Schr¨odinger bridge (Schr¨odinger, 1931, SB) problem
has recently attracted the attention of the machine learning
community due to its relevance to modern challenges in gen-
erative modeling and unpaired learning. Recently, a variety
of methods have been proposed to solve SB in continuous
spaces; see (Gushchin et al., 2023b) for a recent survey.
One modern approach to solving SB is the Iterative Marko-
vian Fitting (IMF) framework (Peluchetti, 2023; Shi et al.,
2023; Gushchin et al., 2024b). Specifically, within this
framework, the discrete-time IMF procedure (Gushchin
1Skoltech,
Moscow,
Russia
2MIPT, Moscow,
Russia
3AIRI, Moscow,
Russia.
Correspondence to:
Grigoriy
Ksenofontov <g.ksenofontov@skoltech.ru>, Alexander Korotin
<a.korotin@skoltech.ru>.
Proceedings of the 42 nd International Conference on ...