dataset-review/dataset-review huggingface dataset:
Ranked Entropy Minimization for Continual Test-Time Adaptation
ICML 2025

Ranked Entropy Minimization for Continual Test-Time Adaptation
Jisu Han 1 Jaemin Na 2 Wonjun Hwang 3
Abstract
Test-time adaptation aims to adapt to realistic en-
vironments in an online manner by learning dur-
ing test time. Entropy minimization has emerged
as a principal strategy for test-time adaptation
due to its efficiency and adaptability. Neverthe-
less, it remains underexplored in continual test-
time adaptation, where stability is more impor-
tant. We observe that the entropy minimization
method often suffers from model collapse, where
the model converges to predicting a single class
for all images due to a trivial solution. We pro-
pose ranked entropy minimization to mitigate
the stability problem of the entropy minimiza-
tion method and extend its applicability to con-
tinuous scenarios. Our approach explicitly struc-
tures the prediction difficulty through a progres-
sive masking strategy. Specifically, it gradually
aligns the model’s probability distributions across
different levels of prediction difficulty while pre-
serving the rank order of entropy.
The pro-
posed method is extensively evaluated across var-
ious benchmarks, demonstrating its effectiveness
through empirical results. Our code is available
at https://github.com/pilsHan/rem
1. Introduction
The real world is non-i.i.d., which demands real-time adap-
tation of AI applications.
Deep learning models have
achieved remarkable progress in recent years; however,
performance degradation caused by distribution shifts be-
tween different domains limits the generalization capabili-
ties (Shimodaira, 2000). Test-time adaptation (TTA) (Wang
et al., 2021) has emerged as a practical approach to address
non-stationary environmental changes by enabling models
trained on source domain to adapt in an online manner to
unlabeled target data during the test-time.
1Ajou University 2Korea Telecom 3Korea University. Corre-
spondence to: Wonjun Hwang <wjhwang@korea.ac.kr>.
Proceedings of the 42 nd International Con...
Boosting Protein Graph Representations through Static-Dynamic Fusion
ICML 2025

Boosting Protein Graph Representations through Static-Dynamic Fusion
Pengkang Guo 1 Bruno Correia 1 Pierre Vandergheynst 1 Daniel Probst 1 2
Abstract
Machine learning for protein modeling faces sig-
nificant challenges due to proteins’ inherently
dynamic nature, yet most graph-based machine
learning methods rely solely on static structural
information. Recently, the growing availability of
molecular dynamics trajectories provides new op-
portunities for understanding the dynamic behav-
ior of proteins; however, computational methods
for utilizing this dynamic information remain lim-
ited. We propose a novel graph representation that
integrates both static structural information and
dynamic correlations from molecular dynamics
trajectories, enabling more comprehensive model-
ing of proteins. By applying relational graph neu-
ral networks (RGNNs) to process this heteroge-
neous representation, we demonstrate significant
improvements over structure-based approaches
across three distinct tasks: atomic adaptability
prediction, binding site detection, and binding
affinity prediction. Our results validate that com-
bining static and dynamic information provides
complementary signals for understanding protein-
ligand interactions, offering new possibilities for
drug design and structural biology applications.
1. Introduction
With the recent surge and successes of deep learning meth-
ods in protein structure prediction, attention is rapidly turn-
ing towards the prediction of the temporal behavior of these
highly dynamic macromolecules. Combined with quantita-
tive and qualitative advances in molecular dynamics simu-
lations (Joshi & Deshmukh, 2021; Zeng et al., 2021; Ma-
jewski et al., 2023; Nam & Wolf-Watz, 2023), this attention
is resulting in the increased availability and accessibility of
simulated molecular dynamics trajectories (Vander Meer-
1 ´Ecole Polytechnique F´ed´erale de Lausanne, Lausanne,
Switzerland
2Wageningen University & research, Wagenin-
gen, The Nether...
SAE-V: Interpreting Multimodal Models for Enhanced Alignment
ICML 2025

SAE-V: Interpreting Multimodal Models for Enhanced Alignment
Hantao Lou * 1 2 Changye Li * 1 2 Jiaming Ji 1 2 Yaodong Yang 1 2
Abstract
With the integration of image modality, the seman-
tic space of multimodal large language models
(MLLMs) is more complex than text-only mod-
els, making their interpretability more challeng-
ing and their alignment less stable, particularly
susceptible to low-quality data, which can lead
to inconsistencies between modalities, hallucina-
tions, and biased outputs. As a result, developing
interpretability methods for MLLMs is crucial
for improving alignment quality and efficiency.
In text-only LLMs, Sparse Autoencoders (SAEs)
have gained attention for their ability to interpret
latent representations. However, extending SAEs
to multimodal settings presents new challenges
due to modality fusion and the difficulty of isolat-
ing cross-modal representations. To address these
challenges, we introduce SAE-V, a mechanistic
interpretability framework that extends the SAE
paradigm to MLLMs. By identifying and analyz-
ing interpretable features along with their corre-
sponding data, SAE-V enables fine-grained inter-
pretation of both model behavior and data quality,
facilitating a deeper understanding of cross-modal
interactions and alignment dynamics. Moreover,
by utilizing cross-modal feature weighting, SAE-
V provides an intrinsic data filtering mechanism
to enhance model alignment without requiring ad-
ditional models. Specifically, when applied to the
alignment process of MLLMs, SAE-V-based data
filtering methods could achieve more than 110%
performance with less than 50% data. Our results
highlight SAE-V’s ability to enhance interpretabil-
ity and alignment in MLLMs, providing insights
into their internal mechanisms. *
*Equal contribution 1Institute for AI, Peking University, Beijing,
China 2State Key Laboratory of General Artificial Intelligence, In-
stitute for AI, Peking University, Beijing, China. Correspondence
to: Hantao Lou <hanta...
When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
ICML 2025

When Every Millisecond Counts: Real-Time Anomaly Detection via the
Multimodal Asynchronous Hybrid Network
Dong Xiao * 1 2 Guangyao Chen * 1 Peixi Peng 3 Yangru Huang 1 Yifan Zhao 4 Yongxing Dai 5
Yonghong Tian 1 2 3 6
Abstract
Anomaly detection is essential for the safety and
reliability of autonomous driving systems. Cur-
rent methods often focus on detection accuracy
but neglect response time, which is critical in
time-sensitive driving scenarios. In this paper,
we introduce real-time anomaly detection for au-
tonomous driving, prioritizing both minimal re-
sponse time and high accuracy. We propose a
novel multimodal asynchronous hybrid network
that combines event streams from event cameras
with image data from RGB cameras. Our net-
work utilizes the high temporal resolution of event
cameras through an asynchronous Graph Neural
Network and integrates it with spatial features
extracted by a CNN from RGB images. This com-
bination effectively captures both the temporal
dynamics and spatial details of the driving en-
vironment, enabling swift and precise anomaly
detection. Extensive experiments on benchmark
datasets show that our approach outperforms ex-
isting methods in both accuracy and response
time, achieving millisecond-level real-time per-
formance.
The code is available at https:
//github.com/PKU-XD/EventAD.
1. Introduction
Autonomous driving technology has been at the forefront
of research and development in recent years, promising to
*Equal contribution 1National Key Laboratory for Multimedia
Information Processing, School of Computer Science, Peking Uni-
versity 2Department of Software and Microelectronics, Peking
University 3School of Electronic and Computer Engineering,
Peking University 4State Key Laboratory of Virtual Reality Tech-
nology and Systems, SCSE, Beihang University 5Baidu Inc
6Peng Cheng Laboratory. Correspondence to: Guangyao Chen
<gy.chen@pku.edu.cn>, Peixi Peng <pxpeng@pku.edu.cn>,
Yonghong Tian <yhtian@pku.edu.cn>.
Proceedings of the 42 ...
Forecasting in Offline Reinforcement Learning for Non-stationary Environments
NeurIPS 2025

Forecasting in Ofﬂine Reinforcement Learning for
Non-stationary Environments
Suzan Ece Ada1,2
Georg Martius2
Emre Ugur1
Erhan Oztop3,4
1Bogazici University, Türkiye
2University of Tübingen, Germany
3Ozyegin University, Türkiye
4Osaka University, Japan
ece.ada@bogazici.edu.tr
Abstract
Ofﬂine Reinforcement Learning (RL) provides a promising avenue for training poli-
cies from pre-collected datasets when gathering additional interaction data is infea-
sible. However, existing ofﬂine RL methods often assume stationarity or only con-
sider synthetic perturbations at test time, assumptions that often fail in real-world
scenarios characterized by abrupt, time-varying offsets. These offsets can lead to
partial observability, causing agents to misperceive their true state and degrade per-
formance. To overcome this challenge, we introduce Forecasting in Non-stationary
Ofﬂine RL (FORL), a framework that uniﬁes (i) conditional diffusion-based candi-
date state generation, trained without presupposing any speciﬁc pattern of future
non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets
environments prone to unexpected, potentially non-Markovian offsets, requiring
robust agent performance from the onset of each episode. Empirical evaluations on
ofﬂine RL benchmarks, augmented with real-world time-series data to simulate
realistic non-stationarity, demonstrate that FORL consistently improves perfor-
mance compared to competitive baselines. By integrating zero-shot forecasting
with the agent’s experience, we aim to bridge the gap between ofﬂine RL and the
complexities of real-world, non-stationary environments.
1
Introduction
Episodes
Offset time series
Figure 1: Setting. The agent does not know
its location in the environment because its
perception is offset every episode j by an un-
known offset bj (only vertical offsets are illus-
trated). FORL leverages historical offset data
and ofﬂine RL data (from a stationary phase)
to forecast and correct for new ...
JAMUN: Bridging Smoothed Molecular Dynamics and Score-Based Learning for Conformational Ensemble Generation
NeurIPS 2025

JAMUN: Bridging Smoothed Molecular Dynamics and
Score-Based Learning for Conformational Ensemble
Generation
Ameya Daigavane∗
MIT
ameyad@mit.edu
Bodhi P. Vani∗
Prescient Design, Genentech
vanib@gene.com
Darcy Davidson
Prescient Design, Genentech
davidsd5@gene.com
Saeed Saremi
Prescient Design, Genentech
saremis@gene.com
Joshua A. Rackers†
Prescient Design, Genentech
Joseph Kleinhenz†
Prescient Design, Genentech
kleinhej@gene.com
Abstract
Conformational ensembles of protein structures are immensely important both for
understanding protein function and drug discovery in novel modalities such as
cryptic pockets. Current techniques for sampling ensembles such as molecular dy-
namics (MD) are computationally inefficient, while many recent machine learning
methods do not transfer to systems outside their training data. We propose JAMUN
which performs MD in a smoothed, noised space of all-atom 3D conformations of
molecules by utilizing the framework of walk-jump sampling. JAMUN enables
ensemble generation for small peptides at rates of an order of magnitude faster
than traditional molecular dynamics. The physical priors in JAMUN enables trans-
ferability to systems outside of its training data, even to peptides that are longer
than those originally trained on. Our model, code and weights are available at
https://github.com/prescient-design/jamun.
1
Introduction
Proteins are inherently dynamic entities constantly in motion, and these movements can be vitally
important. They are not well characterized as single structures as has traditionally been the case,
but rather as ensembles of structures drawn from the Boltzmann distribution [30]. Protein dynamics
is required for the function of most proteins, for instance the global tertiary structure motions for
myglobin to bind oxygen and move it around the body [57], or the beta-sheet transition to a disordered
strand for insulin to dissociate and find and bind to its receptor [5]. Similarly, drug discovery on
protein kinases depen...
TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
ICML 2025

TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Jack Min Ong 1 Matthew Di Ferrante 1 Aaron Pazdera 1 Ryan Garner 1 Sami Jaghouar 1 Manveer Basra 1
Max Ryabinin 2 Johannes Hagemann 1
Abstract
Large language models (LLMs) have proven to
be very capable, but access to frontier models
currently relies on inference providers. This intro-
duces trust challenges: how can we be sure that
the provider is using the model configuration they
claim? We propose TOPLOC, a novel method for
verifiable inference that addresses this problem.
TOPLOC leverages a compact locality-sensitive
hashing mechanism for intermediate activations,
which can detect unauthorized modifications to
models, prompts, or precision with 100% accu-
racy, achieving no false positives or negatives in
our empirical evaluations. Our approach is ro-
bust across diverse hardware configurations, GPU
types, and algebraic reorderings, which allows
for validation speeds significantly faster than the
original inference. By introducing a polynomial
encoding scheme, TOPLOC minimizes the mem-
ory overhead of the generated proofs by 1000×,
requiring only 258 bytes of storage per 32 new
tokens, compared to the 262 KB requirement of
storing the token embeddings directly for Llama
3.1-8B-Instruct. Our method empowers users to
verify LLM inference computations efficiently,
fostering greater trust and transparency in open
ecosystems and laying a foundation for decentral-
ized, verifiable and trustless AI services.
1. Introduction
In recent years, large language models (LLMs) have trans-
formed natural language processing, enabling capabilities
such as high-quality text generation, advanced dialogue
systems, and improved reasoning (Grattafiori et al., 2024;
Gemma Team et al., 2024). Inference providers, entities that
run open-weights LLMs on their own hardware and expose
1Prime Intellect 2Together AI. Correspondence to: Johannes
Hagemann <johannes@primeintellect.ai>.
Proceedings of the 42 nd Intern...
Which Attention Heads Matter for In-Context Learning?
ICML 2025

Which Attention Heads Matter for In-Context Learning?
Kayo Yin Jacob Steinhardt
UC Berkeley
Abstract
Large language models (LLMs) exhibit impres-
sive in-context learning (ICL) capability, enabling
them to perform new tasks using only a few
demonstrations in the prompt.
Two different
mechanisms have been proposed to explain ICL:
induction heads that find and copy relevant to-
kens, and function vector (FV) heads whose acti-
vations compute a latent encoding of the ICL task.
To better understand which of the two distinct
mechanisms drives ICL, we study and compare
induction heads and FV heads in 12 language
models. Through detailed ablations, we discover
that few-shot ICL performance depends primarily
on FV heads, especially in larger models. In ad-
dition, we uncover that FV and induction heads
are connected: many FV heads start as induction
heads during training before transitioning to the
FV mechanism. This leads us to speculate that in-
duction facilitates learning the more complex FV
mechanism that ultimately drives few-shot ICL1.
1. Introduction
One of the most remarkable features of large language mod-
els (LLM) is their ability to perform in-context learning
(ICL), where they can adapt to various new tasks using
only context given in the prompt at inference time. This
capability has become crucial to adapt pre-trained LLMs to
specific tasks, sparking significant research into its underly-
ing mechanisms (Olsson et al., 2022; Aky¨urek et al., 2023;
von Oswald et al., 2023).
To date, two key mechanisms have been associated with
ICL, each supported by different lines of evidence. First,
induction circuits (Elhage et al., 2021) were hypothesized
. Correspondence to: Kayo Yin <kayoyin@berkeley.edu>,
Jacob Steinhardt <jsteinhardt@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Code and data:
https://github.com/kayoyin/
icl-heads.
2.5
5.0
7.5
10.0
12.5
15.0
17....
Metadata Conditioning Accelerates Language Model Pre-training
ICML 2025

Metadata Conditioning Accelerates Language Model Pre-training
Tianyu Gao 1 Alexander Wettig 1 Luxi He 1 Yihe Dong 1 Sadhika Malladi 1 Danqi Chen 1
Abstract
The vast diversity of styles, domains, and qual-
ity levels present in language model pre-training
corpora is essential in developing general model
capabilities, but efficiently learning and deploy-
ing the correct behaviors exemplified in each
of these heterogeneous data sources is challeng-
ing. To address this, we propose a new method,
termed Metadata Conditioning then Cooldown
(MeCo), to incorporate additional learning cues
during pre-training. MeCo first provides meta-
data (e.g., URLs like en.wikipedia.org)
alongside the text during training, then transi-
tions to a cooldown phase using only standard
text—enabling the model to perform well even
without metadata. MeCo significantly acceler-
ates pre-training across different model scales
(600M to 8B parameters) and training corpora
(C4, RefinedWeb, and DCLM). Notably, a 1.6B
language model trained with MeCo matches the
downstream task performance of standard pre-
training while using 33% less data. Addition-
ally, MeCo allows us to steer language models
by conditioning the inference prompt on either
real or fabricated metadata that encodes the de-
sired output properties—for example, prepend-
ing wikipedia.org to reduce harmful gener-
ations or factquizmaster.com (fabricated)
to improve performance on common knowledge
tasks. We further demonstrate that MeCo is com-
patible with various types of metadata, such as
model-generated topics. MeCo is remarkably sim-
ple, adds no computational overhead, and shows
promise for producing more capable and steer-
able language models. Our models, data, and
code are available at https://github.com/
princeton-pli/MeCo.
1Princeton Language and Intelligence, Princeton University.
Correspondence to: Tianyu Gao <tianyug@princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PM...
GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers
ICLR 2025

Published as a conference paper at ICLR 2025
GREATER:
GRADIENTS OVER REASONING MAKES
SMALLER LANGUAGE MODELS STRONG PROMPT OP-
TIMIZERS
Sarkar Snigdha Sarathi Das†
Ryo Kamoi†
Bo Pang⋄
Yusen Zhang†
Caiming Xiong⋄
Rui Zhang†
†The Pennsylvania State University
⋄Salesforce Research
{sfd5525, rmz5227}@psu.edu
ABSTRACT
The effectiveness of large language models (LLMs) is closely tied to the design of
prompts, making prompt optimization essential for enhancing their performance
across a wide range of tasks. Many existing approaches to automating prompt
engineering rely exclusively on textual feedback, refining prompts based solely
on inference errors identified by large, computationally expensive LLMs. Unfor-
tunately, smaller models struggle to generate high-quality feedback, resulting in
complete dependence on large LLM judgment. Moreover, these methods fail to
leverage more direct and finer-grained information, such as gradients, due to op-
erating purely in text space. To this end, we introduce GREATER, a novel prompt
optimization technique that directly incorporates gradient information over task-
specific reasoning.
By utilizing task loss gradients, GREATER enables self-
optimization of prompts for open-source, lightweight language models without the
need for costly closed-source LLMs. This allows high-performance prompt opti-
mization without dependence on massive LLMs, closing the gap between smaller
models and the sophisticated reasoning often needed for prompt refinement. Ex-
tensive evaluations across diverse reasoning tasks including BBH, GSM8k, and
FOLIO demonstrate that GREATER consistently outperforms previous state-of-
the-art prompt optimization methods, even those reliant on powerful LLMs. Ad-
ditionally, GREATER-optimized prompts frequently exhibit better transferability
and, in some cases, boost task performance to levels comparable to or surpass-
ing those achieved by larger language models, highlighting the effectiveness of
prompt optimization guided ...
Compositional Generalization via Forced Rendering of Disentangled Latents
ICML 2025

Compositional Generalization via Forced Rendering of Disentangled Latents
Qiyao Liang* 1 Daoyuan Qian* 2 Liu Ziyin 1 3 Ila Fiete 1
Abstract
Composition—the ability to generate myriad vari-
ations from finite means—is believed to underlie
powerful generalization. However, compositional
generalization remains a key challenge for deep
learning. A widely held assumption is that learn-
ing disentangled (factorized) representations nat-
urally supports this kind of extrapolation. Yet,
empirical results are mixed, with many genera-
tive models failing to recognize and compose fac-
tors to generate out-of-distribution (OOD) sam-
ples. In this work, we investigate a controlled 2D
Gaussian “bump” generation task with fully dis-
entangled (x, y) inputs, demonstrating that stan-
dard generative architectures still fail in OOD
regions when training with partial data, by re-
entangling latent representations in subsequent
layers. By examining the model’s learned kernels
and manifold geometry, we show that this failure
reflects a “memorization” strategy for generation
via data superposition rather than via composition
of the true factorized features. We show that when
models are forced—through architectural modi-
fications with regularization or curated training
data—to render the disentangled latents into the
full-dimensional representational (pixel) space,
they can be highly data-efficient and effective at
composing in OOD regions. These findings un-
derscore that disentangled latents in an abstract
representation are insufficient and show that if
models can represent disentangled factors directly
in the output representational space, it can achieve
robust compositional generalization.
*Equal contribution
1Massachusetts Institute of Technology,
Cambridge MA, USA 02139 2University of Cambridge, Cam-
bridge CB2 1EW, U.K 3NTT Research. Correspondence to: Qiyao
Liang <qiyao@mit.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, ...