AIS-Bench/AIS-Bench huggingface dataset:
Published as a conference paper at ICLR 2025
PARFAM—(NEURAL GUIDED) SYMBOLIC REGRES-
SION VIA CONTINUOUS GLOBAL OPTIMIZATION
Philipp Scholl, Katharina Bieker, Hillary Hauger & Gitta Kutyniok∗
Mathematical Institute
LMU Munich
Munich, Germany
scholl@math.lmu.de
ABSTRACT
The problem of symbolic regression (SR) arises in many different applications,
such as identifying physical laws or deriving mathematical equations describ-
ing the behavior of financial markets from given data. In this paper, we present
our new approach ParFam that utilizes parametric families of suitable symbolic
functions to translate the discrete symbolic regression problem into a continuous
one, resulting in a more straightforward setup compared to current state-of-the-
art methods. In combination with a global optimizer, this approach results in a
highly effective method to tackle the problem of SR. We theoretically analyze the
expressivity of ParFam and demonstrate its performance with extensive numerical
experiments based on the common SR benchmark suit SRBench, showing that we
achieve state-of-the-art results. Moreover, we present an extension incorporating
a pre-trained transformer network (DL-ParFam) to guide ParFam, accelerating the
optimization process by up to two magnitudes. Our code and results can be found
at https://github.com/Philipp238/parfam.
1
INTRODUCTION
Symbolic regression (SR) aims to discover concise and interpretable mathematical functions that
accurately model input-output relationships. This focus on simplicity is crucial for applications
requiring model analysis and trustworthiness, such as in physical or chemical sciences (Quade et al.,
2016; Angelis et al., 2023; Wang et al., 2019). SR finds broad application in diverse fields, including
ecosystem dynamics (Chen et al., 2019), solar power forecasting (Quade et al., 2016), financial
market analysis (Liu & Guo, 2023), materials science (He & Zhang, 2021), and robotics (Oplatkova
& Zelinka, 2007). The growing body of SR r...
Equivariant Neural Tangent Kernels
Philipp Misof 1 Pan Kessel 2 Jan E. Gerken 1
Abstract
Little is known about the training dynamics of
equivariant neural networks, in particular how
it compares to data augmented training of their
non-equivariant counterparts. Recently, neural
tangent kernels (NTKs) have emerged as a pow-
erful tool to analytically study the training dy-
namics of wide neural networks. In this work,
we take an important step towards a theoretical
understanding of training dynamics of equivari-
ant models by deriving neural tangent kernels for
a broad class of equivariant architectures based
on group convolutions. As a demonstration of
the capabilities of our framework, we show an in-
teresting relationship between data augmentation
and group convolutional networks. Specifically,
we prove that they share the same expected pre-
diction over initializations at all training times
and even off the data manifold. In this sense, they
have the same training dynamics. We demonstrate
in numerical experiments that this still holds ap-
proximately for finite-width ensembles. By imple-
menting equivariant NTKs for roto-translations
in the plane (G = Cn ⋉R2) and 3d rotations
(G = SO(3)), we show that equivariant NTKs
outperform their non-equivariant counterparts as
kernel predictors for histological image classifica-
tion and quantum mechanical property prediction.
1. Introduction
Equivariant neural networks (Weiler et al., 2023; Gerken
et al., 2023) are widely used in many applications of great
practical importance, for example in medical image analysis
in two and three dimensions (Bekkers et al., 2018; Winkels
& Cohen, 2019; M¨uller et al., 2021; Pang et al., 2023) and in
quantum chemistry (Duval et al., 2023; Batzner et al., 2022;
1Department of Mathematical Sciences, Chalmers University of
Technology and the University of Gothenburg, SE-412 96 Gothen-
burg, Sweden 2Prescient Design, Genentech Roche, Basel, Switzer-
land. Correspondence to: Jan Gerken <gerken@...
Published as a conference paper at ICLR 2025
UNCERTAINTY HERDING:
ONE ACTIVE LEARNING
METHOD FOR ALL LABEL BUDGETS
Wonho Bae
University of British Columbia & Borealis AI
whbae@cs.ubc.ca
Danica J. Sutherland∗
University of British Columbia & Amii
dsuth@cs.ubc.ca
Gabriel L. Oliveira∗
Borealis AI
gabriel.oliveira@borealisai.com
ABSTRACT
Most active learning research has focused on methods which perform well when
many labels are available, but can be dramatically worse than random selection
when label budgets are small. Other methods have focused on the low-budget
regime, but do poorly as label budgets increase. As the line between “low” and
“high” budgets varies by problem, this is a serious issue in practice. We propose
uncertainty coverage, an objective which generalizes a variety of low- and high-
budget objectives, as well as natural, hyperparameter-light methods to smoothly
interpolate between low- and high-budget regimes. We call greedy optimization of
the estimate Uncertainty Herding; this simple method is computationally fast, and
we prove that it nearly optimizes the distribution-level coverage. In experimental
validation across a variety of active learning tasks, our proposal matches or beats
state-of-the-art performance in essentially all cases; it is the only method of which
we are aware that reliably works well in both low- and high-budget settings.
1
INTRODUCTION
In active learning, rather than being provided a dataset of input-output pairs as in passive learning,
a model strategically requests annotations for specific unlabeled inputs. The aim is to learn a good
model while minimizing the number of required output annotations. This procedure is generally
iterative: a model is initially trained on a small, labeled dataset, then selects the most “informative”
data points from an unlabeled pool to annotate. This is particularly useful when labeling is expensive
or time-consuming. For example, manual annotations of medical imaging by radiologists or pathol-
...
Ranked Entropy Minimization for Continual Test-Time Adaptation
Jisu Han 1 Jaemin Na 2 Wonjun Hwang 3
Abstract
Test-time adaptation aims to adapt to realistic en-
vironments in an online manner by learning dur-
ing test time. Entropy minimization has emerged
as a principal strategy for test-time adaptation
due to its efficiency and adaptability. Neverthe-
less, it remains underexplored in continual test-
time adaptation, where stability is more impor-
tant. We observe that the entropy minimization
method often suffers from model collapse, where
the model converges to predicting a single class
for all images due to a trivial solution. We pro-
pose ranked entropy minimization to mitigate
the stability problem of the entropy minimiza-
tion method and extend its applicability to con-
tinuous scenarios. Our approach explicitly struc-
tures the prediction difficulty through a progres-
sive masking strategy. Specifically, it gradually
aligns the model’s probability distributions across
different levels of prediction difficulty while pre-
serving the rank order of entropy.
The pro-
posed method is extensively evaluated across var-
ious benchmarks, demonstrating its effectiveness
through empirical results. Our code is available
at https://github.com/pilsHan/rem
1. Introduction
The real world is non-i.i.d., which demands real-time adap-
tation of AI applications.
Deep learning models have
achieved remarkable progress in recent years; however,
performance degradation caused by distribution shifts be-
tween different domains limits the generalization capabili-
ties (Shimodaira, 2000). Test-time adaptation (TTA) (Wang
et al., 2021) has emerged as a practical approach to address
non-stationary environmental changes by enabling models
trained on source domain to adapt in an online manner to
unlabeled target data during the test-time.
1Ajou University 2Korea Telecom 3Korea University. Corre-
spondence to: Wonjun Hwang <wjhwang@korea.ac.kr>.
Proceedings of the 42 nd International Con...
Boosting Protein Graph Representations through Static-Dynamic Fusion
Pengkang Guo 1 Bruno Correia 1 Pierre Vandergheynst 1 Daniel Probst 1 2
Abstract
Machine learning for protein modeling faces sig-
nificant challenges due to proteins’ inherently
dynamic nature, yet most graph-based machine
learning methods rely solely on static structural
information. Recently, the growing availability of
molecular dynamics trajectories provides new op-
portunities for understanding the dynamic behav-
ior of proteins; however, computational methods
for utilizing this dynamic information remain lim-
ited. We propose a novel graph representation that
integrates both static structural information and
dynamic correlations from molecular dynamics
trajectories, enabling more comprehensive model-
ing of proteins. By applying relational graph neu-
ral networks (RGNNs) to process this heteroge-
neous representation, we demonstrate significant
improvements over structure-based approaches
across three distinct tasks: atomic adaptability
prediction, binding site detection, and binding
affinity prediction. Our results validate that com-
bining static and dynamic information provides
complementary signals for understanding protein-
ligand interactions, offering new possibilities for
drug design and structural biology applications.
1. Introduction
With the recent surge and successes of deep learning meth-
ods in protein structure prediction, attention is rapidly turn-
ing towards the prediction of the temporal behavior of these
highly dynamic macromolecules. Combined with quantita-
tive and qualitative advances in molecular dynamics simu-
lations (Joshi & Deshmukh, 2021; Zeng et al., 2021; Ma-
jewski et al., 2023; Nam & Wolf-Watz, 2023), this attention
is resulting in the increased availability and accessibility of
simulated molecular dynamics trajectories (Vander Meer-
1 ´Ecole Polytechnique F´ed´erale de Lausanne, Lausanne,
Switzerland
2Wageningen University & research, Wagenin-
gen, The Nether...
SAE-V: Interpreting Multimodal Models for Enhanced Alignment
Hantao Lou * 1 2 Changye Li * 1 2 Jiaming Ji 1 2 Yaodong Yang 1 2
Abstract
With the integration of image modality, the seman-
tic space of multimodal large language models
(MLLMs) is more complex than text-only mod-
els, making their interpretability more challeng-
ing and their alignment less stable, particularly
susceptible to low-quality data, which can lead
to inconsistencies between modalities, hallucina-
tions, and biased outputs. As a result, developing
interpretability methods for MLLMs is crucial
for improving alignment quality and efficiency.
In text-only LLMs, Sparse Autoencoders (SAEs)
have gained attention for their ability to interpret
latent representations. However, extending SAEs
to multimodal settings presents new challenges
due to modality fusion and the difficulty of isolat-
ing cross-modal representations. To address these
challenges, we introduce SAE-V, a mechanistic
interpretability framework that extends the SAE
paradigm to MLLMs. By identifying and analyz-
ing interpretable features along with their corre-
sponding data, SAE-V enables fine-grained inter-
pretation of both model behavior and data quality,
facilitating a deeper understanding of cross-modal
interactions and alignment dynamics. Moreover,
by utilizing cross-modal feature weighting, SAE-
V provides an intrinsic data filtering mechanism
to enhance model alignment without requiring ad-
ditional models. Specifically, when applied to the
alignment process of MLLMs, SAE-V-based data
filtering methods could achieve more than 110%
performance with less than 50% data. Our results
highlight SAE-V’s ability to enhance interpretabil-
ity and alignment in MLLMs, providing insights
into their internal mechanisms. *
*Equal contribution 1Institute for AI, Peking University, Beijing,
China 2State Key Laboratory of General Artificial Intelligence, In-
stitute for AI, Peking University, Beijing, China. Correspondence
to: Hantao Lou <hanta...
When Every Millisecond Counts: Real-Time Anomaly Detection via the
Multimodal Asynchronous Hybrid Network
Dong Xiao * 1 2 Guangyao Chen * 1 Peixi Peng 3 Yangru Huang 1 Yifan Zhao 4 Yongxing Dai 5
Yonghong Tian 1 2 3 6
Abstract
Anomaly detection is essential for the safety and
reliability of autonomous driving systems. Cur-
rent methods often focus on detection accuracy
but neglect response time, which is critical in
time-sensitive driving scenarios. In this paper,
we introduce real-time anomaly detection for au-
tonomous driving, prioritizing both minimal re-
sponse time and high accuracy. We propose a
novel multimodal asynchronous hybrid network
that combines event streams from event cameras
with image data from RGB cameras. Our net-
work utilizes the high temporal resolution of event
cameras through an asynchronous Graph Neural
Network and integrates it with spatial features
extracted by a CNN from RGB images. This com-
bination effectively captures both the temporal
dynamics and spatial details of the driving en-
vironment, enabling swift and precise anomaly
detection. Extensive experiments on benchmark
datasets show that our approach outperforms ex-
isting methods in both accuracy and response
time, achieving millisecond-level real-time per-
formance.
The code is available at https:
//github.com/PKU-XD/EventAD.
1. Introduction
Autonomous driving technology has been at the forefront
of research and development in recent years, promising to
*Equal contribution 1National Key Laboratory for Multimedia
Information Processing, School of Computer Science, Peking Uni-
versity 2Department of Software and Microelectronics, Peking
University 3School of Electronic and Computer Engineering,
Peking University 4State Key Laboratory of Virtual Reality Tech-
nology and Systems, SCSE, Beihang University 5Baidu Inc
6Peng Cheng Laboratory. Correspondence to: Guangyao Chen
<gy.chen@pku.edu.cn>, Peixi Peng <pxpeng@pku.edu.cn>,
Yonghong Tian <yhtian@pku.edu.cn>.
Proceedings of the 42 ...
Forecasting in Ofﬂine Reinforcement Learning for
Non-stationary Environments
Suzan Ece Ada1,2
Georg Martius2
Emre Ugur1
Erhan Oztop3,4
1Bogazici University, Türkiye
2University of Tübingen, Germany
3Ozyegin University, Türkiye
4Osaka University, Japan
ece.ada@bogazici.edu.tr
Abstract
Ofﬂine Reinforcement Learning (RL) provides a promising avenue for training poli-
cies from pre-collected datasets when gathering additional interaction data is infea-
sible. However, existing ofﬂine RL methods often assume stationarity or only con-
sider synthetic perturbations at test time, assumptions that often fail in real-world
scenarios characterized by abrupt, time-varying offsets. These offsets can lead to
partial observability, causing agents to misperceive their true state and degrade per-
formance. To overcome this challenge, we introduce Forecasting in Non-stationary
Ofﬂine RL (FORL), a framework that uniﬁes (i) conditional diffusion-based candi-
date state generation, trained without presupposing any speciﬁc pattern of future
non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets
environments prone to unexpected, potentially non-Markovian offsets, requiring
robust agent performance from the onset of each episode. Empirical evaluations on
ofﬂine RL benchmarks, augmented with real-world time-series data to simulate
realistic non-stationarity, demonstrate that FORL consistently improves perfor-
mance compared to competitive baselines. By integrating zero-shot forecasting
with the agent’s experience, we aim to bridge the gap between ofﬂine RL and the
complexities of real-world, non-stationary environments.
1
Introduction
Episodes
Offset time series
Figure 1: Setting. The agent does not know
its location in the environment because its
perception is offset every episode j by an un-
known offset bj (only vertical offsets are illus-
trated). FORL leverages historical offset data
and ofﬂine RL data (from a stationary phase)
to forecast and correct for new ...
JAMUN: Bridging Smoothed Molecular Dynamics and
Score-Based Learning for Conformational Ensemble
Generation
Ameya Daigavane∗
MIT
ameyad@mit.edu
Bodhi P. Vani∗
Prescient Design, Genentech
vanib@gene.com
Darcy Davidson
Prescient Design, Genentech
davidsd5@gene.com
Saeed Saremi
Prescient Design, Genentech
saremis@gene.com
Joshua A. Rackers†
Prescient Design, Genentech
Joseph Kleinhenz†
Prescient Design, Genentech
kleinhej@gene.com
Abstract
Conformational ensembles of protein structures are immensely important both for
understanding protein function and drug discovery in novel modalities such as
cryptic pockets. Current techniques for sampling ensembles such as molecular dy-
namics (MD) are computationally inefficient, while many recent machine learning
methods do not transfer to systems outside their training data. We propose JAMUN
which performs MD in a smoothed, noised space of all-atom 3D conformations of
molecules by utilizing the framework of walk-jump sampling. JAMUN enables
ensemble generation for small peptides at rates of an order of magnitude faster
than traditional molecular dynamics. The physical priors in JAMUN enables trans-
ferability to systems outside of its training data, even to peptides that are longer
than those originally trained on. Our model, code and weights are available at
https://github.com/prescient-design/jamun.
1
Introduction
Proteins are inherently dynamic entities constantly in motion, and these movements can be vitally
important. They are not well characterized as single structures as has traditionally been the case,
but rather as ensembles of structures drawn from the Boltzmann distribution [30]. Protein dynamics
is required for the function of most proteins, for instance the global tertiary structure motions for
myglobin to bind oxygen and move it around the body [57], or the beta-sheet transition to a disordered
strand for insulin to dissociate and find and bind to its receptor [5]. Similarly, drug discovery on
protein kinases depen...
TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Jack Min Ong 1 Matthew Di Ferrante 1 Aaron Pazdera 1 Ryan Garner 1 Sami Jaghouar 1 Manveer Basra 1
Max Ryabinin 2 Johannes Hagemann 1
Abstract
Large language models (LLMs) have proven to
be very capable, but access to frontier models
currently relies on inference providers. This intro-
duces trust challenges: how can we be sure that
the provider is using the model configuration they
claim? We propose TOPLOC, a novel method for
verifiable inference that addresses this problem.
TOPLOC leverages a compact locality-sensitive
hashing mechanism for intermediate activations,
which can detect unauthorized modifications to
models, prompts, or precision with 100% accu-
racy, achieving no false positives or negatives in
our empirical evaluations. Our approach is ro-
bust across diverse hardware configurations, GPU
types, and algebraic reorderings, which allows
for validation speeds significantly faster than the
original inference. By introducing a polynomial
encoding scheme, TOPLOC minimizes the mem-
ory overhead of the generated proofs by 1000×,
requiring only 258 bytes of storage per 32 new
tokens, compared to the 262 KB requirement of
storing the token embeddings directly for Llama
3.1-8B-Instruct. Our method empowers users to
verify LLM inference computations efficiently,
fostering greater trust and transparency in open
ecosystems and laying a foundation for decentral-
ized, verifiable and trustless AI services.
1. Introduction
In recent years, large language models (LLMs) have trans-
formed natural language processing, enabling capabilities
such as high-quality text generation, advanced dialogue
systems, and improved reasoning (Grattafiori et al., 2024;
Gemma Team et al., 2024). Inference providers, entities that
run open-weights LLMs on their own hardware and expose
1Prime Intellect 2Together AI. Correspondence to: Johannes
Hagemann <johannes@primeintellect.ai>.
Proceedings of the 42 nd Intern...
Which Attention Heads Matter for In-Context Learning?
Kayo Yin Jacob Steinhardt
UC Berkeley
Abstract
Large language models (LLMs) exhibit impres-
sive in-context learning (ICL) capability, enabling
them to perform new tasks using only a few
demonstrations in the prompt.
Two different
mechanisms have been proposed to explain ICL:
induction heads that find and copy relevant to-
kens, and function vector (FV) heads whose acti-
vations compute a latent encoding of the ICL task.
To better understand which of the two distinct
mechanisms drives ICL, we study and compare
induction heads and FV heads in 12 language
models. Through detailed ablations, we discover
that few-shot ICL performance depends primarily
on FV heads, especially in larger models. In ad-
dition, we uncover that FV and induction heads
are connected: many FV heads start as induction
heads during training before transitioning to the
FV mechanism. This leads us to speculate that in-
duction facilitates learning the more complex FV
mechanism that ultimately drives few-shot ICL1.
1. Introduction
One of the most remarkable features of large language mod-
els (LLM) is their ability to perform in-context learning
(ICL), where they can adapt to various new tasks using
only context given in the prompt at inference time. This
capability has become crucial to adapt pre-trained LLMs to
specific tasks, sparking significant research into its underly-
ing mechanisms (Olsson et al., 2022; Aky¨urek et al., 2023;
von Oswald et al., 2023).
To date, two key mechanisms have been associated with
ICL, each supported by different lines of evidence. First,
induction circuits (Elhage et al., 2021) were hypothesized
. Correspondence to: Kayo Yin <kayoyin@berkeley.edu>,
Jacob Steinhardt <jsteinhardt@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Code and data:
https://github.com/kayoyin/
icl-heads.
2.5
5.0
7.5
10.0
12.5
15.0
17....
Metadata Conditioning Accelerates Language Model Pre-training
Tianyu Gao 1 Alexander Wettig 1 Luxi He 1 Yihe Dong 1 Sadhika Malladi 1 Danqi Chen 1
Abstract
The vast diversity of styles, domains, and qual-
ity levels present in language model pre-training
corpora is essential in developing general model
capabilities, but efficiently learning and deploy-
ing the correct behaviors exemplified in each
of these heterogeneous data sources is challeng-
ing. To address this, we propose a new method,
termed Metadata Conditioning then Cooldown
(MeCo), to incorporate additional learning cues
during pre-training. MeCo first provides meta-
data (e.g., URLs like en.wikipedia.org)
alongside the text during training, then transi-
tions to a cooldown phase using only standard
text—enabling the model to perform well even
without metadata. MeCo significantly acceler-
ates pre-training across different model scales
(600M to 8B parameters) and training corpora
(C4, RefinedWeb, and DCLM). Notably, a 1.6B
language model trained with MeCo matches the
downstream task performance of standard pre-
training while using 33% less data. Addition-
ally, MeCo allows us to steer language models
by conditioning the inference prompt on either
real or fabricated metadata that encodes the de-
sired output properties—for example, prepend-
ing wikipedia.org to reduce harmful gener-
ations or factquizmaster.com (fabricated)
to improve performance on common knowledge
tasks. We further demonstrate that MeCo is com-
patible with various types of metadata, such as
model-generated topics. MeCo is remarkably sim-
ple, adds no computational overhead, and shows
promise for producing more capable and steer-
able language models. Our models, data, and
code are available at https://github.com/
princeton-pli/MeCo.
1Princeton Language and Intelligence, Princeton University.
Correspondence to: Tianyu Gao <tianyug@princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PM...