AIS-Bench/AIS-Bench huggingface dataset:
Information-Theoretic Discrete Diffusion
Moongyu Jeon1
Sangwoo Shin1
Dongjae Jeon2
Albert No1∗
1Department of Artificial Intelligence, Yonsei University
2Department of Computer Science, Yonsei University
Abstract
We present an information-theoretic framework for discrete diffusion models that
yields principled estimators of log-likelihood using score-matching losses. Inspired
by the I-MMSE identity for the Gaussian setup, we derive analogous results for the
discrete setting. Specifically, we introduce the Information–Minimum Denoising
Score Entropy (I-MDSE) relation, which links mutual information between data
and its diffused version to the minimum denoising score entropy (DSE) loss. We
extend this theory to masked diffusion and establish the Information–Minimum
Denoising Cross-Entropy (I-MDCE) relation, connecting cross-entropy losses to
mutual information in discrete masked processes. These results provide a time-
integral decomposition of the log-likelihood of the data in terms of optimal score-
based losses, showing that commonly used losses such as DSE and DCE are not
merely variational bounds but tight and principled estimators of log-likelihood. The
I-MDCE decomposition further enables practical extensions, including time-free
formula, conditional likelihood estimation in prompt–response tasks, and coupled
Monte Carlo estimation of likelihood ratios. Experiments on synthetic and real-
world data confirm the accuracy, variance stability, and utility of our estimators. The
code is publicly available at https://github.com/Dongjae0324/infodis.
1
Introduction
Diffusion models have emerged as a powerful framework for generative modeling, enabling state-
of-the-art performance in continuous domains such as image and audio generation (Sohl-Dickstein
et al., 2015; Ho et al., 2020; Chen et al., 2021; Kong et al., 2021; Saharia et al., 2022). Central to
these models is the idea of gradually corrupting data through a forward noising process, and learning
to reverse this...
Learning the Electronic Hamiltonian of Large Atomic Structures
Chen Hao Xia * 1 Manasa Kaniselvan * 1 Alexandros Nikolaos Ziogas 1 Marko Mladenovi´c 1 Rayen Mahjoub 1
Alexander Maeder 1 Mathieu Luisier 1
Abstract
Graph neural networks (GNNs) have shown
promise in learning the ground-state electronic
properties of materials, subverting ab initio den-
sity functional theory (DFT) calculations when
the underlying lattices can be represented as small
and/or repeatable unit cells (i.e., molecules and
periodic crystals). Realistic systems are, how-
ever, non-ideal and generally characterized by
higher structural complexity. As such, they re-
quire large (10+ ˚A) unit cells and thousands of
atoms to be accurately described. At these scales,
DFT becomes computationally prohibitive, mak-
ing GNNs especially attractive. In this work, we
present a strictly local equivariant GNN capable
of learning the electronic Hamiltonian (H) of
realistically extended materials. It incorporates
an augmented partitioning approach that enables
training on arbitrarily large structures while pre-
serving local atomic environments beyond bound-
aries. We demonstrate its capabilities by predict-
ing the electronic Hamiltonian of various systems
with up to 3,000 nodes (atoms), 500,000+ edges,
∼28 million orbital interactions (nonzero entries
of H), and ≤0.53% error in the eigenvalue spec-
tra. Our work expands the applicability of current
electronic property prediction methods to some of
the most challenging cases encountered in compu-
tational materials science, namely systems with
disorder, interfaces, and defects.
1. Introduction
Graph neural networks (GNNs) have proven capable of
learning properties that are functions of molecular and
*Equal contribution
1Integrated Systems Laboratory, Depart-
ment of Information Technology and Electrical Engineering, ETH
Zurich, Zurich, Switzerland.
Correspondence to: Chen Hao
Xia <chexia@iis.ee.ethz.ch>, Manasa Kaniselvan <mkanisel-
van@iis.ee.ethz.ch>.
Proce...
PENCIL: Long Thoughts with Short Memory
Chenxiao Yang 1 Nathan Srebro 1 David McAllester 1 Zhiyuan Li 1
Abstract
While state-of-the-art LLMs have demonstrated
great promise of using long Chains-of-Thought
(CoT) to boost reasoning, scaling it up to more
challenging problems is fundamentally limited by
suboptimal memory usage — intermediate com-
putations accumulate indefinitely in context even
no longer needed for future thoughts. We intro-
duce PENCIL, which incorporates a novel reduc-
tion mechanism into the autoregressive genera-
tion process that recursively clean up intermedi-
ate thoughts based on patterns learned from train-
ing. By alternately generating and erasing, PEN-
CIL can think deeper to solve harder problems
using shorter context and less computes. Empiri-
cally, for example, we demonstrate PENCIL with
a small 25M-parameter transformer and 2048 con-
text length solves Einstein’s puzzle — a task that
challenges much larger models like GPT-4. Theo-
retically, we prove PENCIL can perform universal
efficient computation by simulating any Turing
machines with optimal time and space complexity,
and thus can solve arbitrary computable tasks that
are otherwise intractable for vanilla CoT.
1. Introduction
Recently, there has been a surge of interest in reasoning
with Chain-of-Thought (CoT) (Wei et al., 2022) and gen-
erating longer thoughts at test-time to tackle larger-scale
and more complicated problems (OpenAI, 2024; Guo et al.,
2025; Snell et al., 2024; Muennighoff et al., 2025). CoT is
an iterative generation process: each intermediate reason-
ing step is appended to the current context and treated as
the input in subsequent reasoning. The context grows until
reaching a final answer. While such an iterative model is
theoretically powerful – capable, in principle, of tackling
many intricate problems given unlimited length (Merrill &
1Toyota Technological Institute at Chicago. Correspondence
to: Chenxiao Yang <chenxiao@ttic.edu>.
Proceedings of the 42 nd I...
Feedforward Few-shot Species Range Estimation
Christian Lange 1 Max Hamilton 2 Elijah Cole 3 Alexander Shepard 4 Samuel Heinrich 5 Angela Zhu 2
Subhransu Maji 2 Grant Van Horn 2 Oisin Mac Aodha 1
Abstract
Knowing where a particular species can or can-
not be found on Earth is crucial for ecological
research and conservation efforts. By mapping
the spatial ranges of all species, we would ob-
tain deeper insights into how global biodiversity
is affected by climate change and habitat loss.
However, accurate range estimates are only avail-
able for a relatively small proportion of all known
species. For the majority of the remaining species,
we typically only have a small number of records
denoting the spatial locations where they have
previously been observed. We outline a new ap-
proach for few-shot species range estimation to
address the challenge of accurately estimating the
range of a species from limited data. During infer-
ence, our model takes a set of spatial locations as
input, along with optional metadata such as text or
an image, and outputs a species encoding that can
be used to predict the range of a previously un-
seen species in a feedforward manner. We evalu-
ate our approach on two challenging benchmarks,
where we obtain state-of-the-art range estimation
performance, in a fraction of the compute time,
compared to recent alternative approaches.
1. Introduction
Understanding the spatial distribution of plant and ani-
mal species is essential to mitigate the ongoing decline
in global biodiversity (Jetz et al., 2019). Monitoring these
distributions over time allows us to quantify the impacts
of climate change, habitat loss, and conservation interven-
tions (Mantyka-pringle et al., 2012). Estimating a species’
spatial distribution typically starts with collecting a set of
observations that denote the locations where the species has
been confirmed to be present or absent. Traditionally, this
1University of Edinburgh
2UMass Amherst
3GenBio AI
4iNaturalist
5Co...
Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos
Christian Pedersen 1 2 Laure Zanna 1 2 Joan Bruna 1 2
Abstract
Autoregressive surrogate models (or emulators)
of spatiotemporal systems provide an avenue for
fast, approximate predictions, with broad applica-
tions across science and engineering. At inference
time, however, these models are generally unable
to provide predictions over long time rollouts due
to accumulation of errors leading to diverging tra-
jectories. In essence, emulators operate out of
distribution, and controlling the online distribu-
tion quickly becomes intractable in large-scale
settings. To address this fundamental issue, and
focusing on time-stationary systems admitting an
invariant measure, we leverage diffusion models
to obtain an implicit estimator of the score of this
invariant measure. We show that this model of
the score function can be used to stabilize autore-
gressive emulator rollouts by applying on-the-fly
denoising during inference, a process we call ther-
malization. Thermalizing an emulator rollout is
shown to extend the time horizon of stable pre-
dictions by two orders of magnitude in complex
systems exhibiting turbulent and chaotic behav-
ior, opening up a novel application of diffusion
models in the context of neural emulation.
1. Introduction
The modeling of dynamical systems is a cornerstone task
in the physical sciences and engineering, with applications
across weather (Pathak et al., 2022; Watt-Meyer et al., 2023;
Lam et al., 2023; Lang et al., 2024) and climate modeling
(Kochkov et al., 2024; Subel & Zanna, 2024; Lupin-Jimenez
et al., 2025). The standard approach to modeling these sys-
tems is to solve the underlying partial differential equation
(PDE) describing the system using numerical methods. In
the case of high-dimensional systems, the computational
1Courant Institute of Mathematical Sciences, New York Univer-
sity, USA 2Center for Data Science, New York University, USA.
Correspondence ...
Spatial Reasoning with Denoising Models
Christopher Wewer 1 Bart Pogodzinski 1 Bernt Schiele 1 Jan Eric Lenssen 1
SRMs (Ours)
Standard DDPM
Playing MNIST Sudoku with Image Generators….
Figure 1: We introduce Spatial Reasoning Models (SRMs), a framework to systematically investigate diffusion/flow-based
generative models with respect to their reasoning capabilities over multiple variables. We introduce benchmarks that allow
quantification of higher-level reasoning capabilities and show that manual and automatic schemes for sequentialization can
heavily reduce hallucination. Here, we show the solving process of one of the benchmarks, a Sudoku game consisting of
MNIST images, which is solved correctly by our SRMs, while standard diffusion models fail.
Abstract
We introduce Spatial Reasoning Models (SRMs),
a framework to perform reasoning over sets of
continuous variables via denoising generative
models. SRMs infer continuous representations
on a set of unobserved variables, given observa-
tions on observed variables. Current generative
models on spatial domains, such as diffusion and
flow matching models, often collapse to hallucina-
tion in case of complex distributions. To measure
this, we introduce a set of benchmark tasks that
test the quality of complex reasoning in generative
models and can quantify hallucination. The SRM
framework allows to report key findings about
importance of sequentialization in generation, the
associated order, as well as the sampling strate-
gies during training. It demonstrates, for the first
time, that order of generation can successfully be
predicted by the denoising network itself. Using
these findings, we can increase the accuracy of
specific reasoning tasks from < 1% to > 50%.
Our project website provides additional videos,
code, and the benchmark datasets.
1Max Planck Institute for Informatics, Saarland Informat-
ics Campus, Germany. Correspondence to: Christopher Wewer
<cwewer@mpi-inf.mpg.de>, Bart Pogodzinski <bpogodzi@mpi-
inf.m...
Published as a conference paper at ICLR 2025
IN-CONTEXT TIME SERIES PREDICTOR
Jiecheng Lu, Yan Sun, Shihao Yang
Georgia Institute of Technology
{jlu414,yansun}@gatech.edu, shihao.yang@isye.gatech.edu
ABSTRACT
Recent Transformer-based large language models (LLMs) demonstrate in-context
learning ability to perform various functions based solely on the provided context,
without updating model parameters. To fully utilize the in-context capabilities
in time series forecasting (TSF) problems, unlike previous Transformer-based or
LLM-based time series forecasting methods, we reformulate "time series fore-
casting tasks" as input tokens by constructing a series of (lookback, future) pairs
within the tokens. This method aligns more closely with the inherent in-context
mechanisms, and is more parameter-efficient without the need of using pre-trained
LLM parameters. Furthermore, it addresses issues such as overfitting in existing
Transformer-based TSF models, consistently achieving better performance across
full-data, few-shot, and zero-shot settings compared to previous architectures 1.
1
INTRODUCTION
Transformer-based large language models (LLMs) have significantly impacted various research
and application areas (Brown et al., 2020). Their inherent in-context learning (ICL) capabilities
highlighted by previous studies (Müller et al., 2022; Min et al., 2022; Wei et al., 2023; Xie et al.,
2022) allow them to adapt and generalize from context examples provided in input prompts without
any parameter updates. This enables LLMs to effectively handle few-shot and zero-shot tasks. Recent
research on ICL (Zhang et al., 2023; Garg et al., 2022; Akyürek et al., 2023; Li et al., 2023; Dai et al.,
2023; Bai et al., 2023) has shown that Transformers can adaptively learn to perform various functions
including linear predictors, shallow MLPs, gradient descent, algorithm selection, etc., based on a
series of (input, label) pairs as input tokens, as shown in Figure 1 (a).
Timestep
Input
Labe...
KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache
Quantization for Efficient and Nearly Lossless LLM Inference
Xing Li * 1 Zeyu Xing * 2 Yiming Li 1 Linping Qu 1 Hui-Ling Zhen 1 Yiwu Yao 3 Wulong Liu 1
Sinno Jialin Pan 2 Mingxuan Yuan 1
Abstract
KV cache quantization can improve Large Lan-
guage Models (LLMs) inference throughput and
latency in long contexts and large batch-size
scenarios while preserving LLMs effectiveness.
However, current methods have three unsolved
issues: overlooking layer-wise sensitivity to KV
cache quantization, high overhead of online fine-
grained decision-making, and low flexibility to
different LLMs and constraints. Therefore, we
theoretically analyze the inherent correlation of
layer-wise transformer attention patterns to KV
cache quantization errors and study why key
cache is generally more important than value
cache for quantization error reduction. We fur-
ther propose a simple yet effective framework
KVTuner to adaptively search for the optimal
hardware-friendly layer-wise KV quantization
precision pairs for coarse-grained KV cache with
multi-objective optimization and directly utilize
the offline searched configurations during on-
line inference. To reduce the computational cost
of offline calibration, we utilize the intra-layer
KV precision pair pruning and inter-layer clus-
tering to reduce the search space. Experimen-
tal results show that we can achieve nearly loss-
less 3.25-bit mixed precision KV cache quantiza-
tion for LLMs like Llama-3.1-8B-Instruct and
4.0-bit for sensitive models like Qwen2.5-7B-
Instruct on mathematical reasoning tasks. The
maximum inference throughput can be improved
by 21.25% compared with KIVI-KV8 quantiza-
tion over various context lengths. Our code and
searched configurations are available at https:
//github.com/cmd2001/KVTuner.
*Equal contribution
1Huawei Noah’s Ark Lab 2The Chi-
nese University of Hong Kong 3Huawei Computing Product
Line.
Correspondence to: Xing Li <li.xing2@huawei.c...
Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Yuxiao Qu * 1 Matthew Y. R. Yang * 1 Amrith Setlur 1 Lewis Tunstall 2 Edward Emanuel Beeching 2
Ruslan Salakhutdinov 1 Aviral Kumar 1
Abstract
Training models to effectively use test-time com-
pute is crucial for improving the reasoning per-
formance of LLMs. Current methods mostly do
so via fine-tuning on search traces or running RL
with 0/1 outcome reward, but do these approaches
efficiently utilize test-time compute? Would these
approaches continue to scale as the budget im-
proves? To answer these questions, in this paper,
we formalize the problem of optimizing test-time
compute as a meta-reinforcement learning (RL)
problem, which provides a principled perspective
on spending test-time compute. This perspective
enables us to view the long output stream from the
LLM as consisting of several episodes run at test
time and leads us to use a notion akin to cumula-
tive regret over output tokens as a way to measure
the efficacy of test-time compute. Akin to how RL
algorithms can best tradeoff exploration and ex-
ploitation over training, minimizing regret should
also provide the best balance between exploration
and exploitation in the token stream. While we
show that state-of-the-art models do not minimize
regret, one can do so by maximizing a dense re-
ward bonus in conjunction with the outcome 0/1
reward RL. This bonus is the “progress” made
by each subsequent block in the output stream,
quantified by the change in the likelihood of even-
tual success. Using these insights, we develop
Meta Reinforcment Fine-Tuning, or MRT, a new
class of fine-tuning methods for optimizing test-
time compute. MRT leads to a 2-3x relative gain
in performance and roughly a 1.5x gain in token
efficiency for math reasoning.
1. Introduction
Recent results in LLM reasoning (Snell et al., 2024) il-
lustrate the potential to improve reasoning capabilities by
*Equal contribution 1CMU 2Hugging Face. Correspondence to:
Yuxiao Qu <yu...
STAMP Your Content:
Proving Dataset Membership via Watermarked Rephrasings
Saksham Rastogi 1 Pratyush Maini 2 3 Danish Pruthi 1
Abstract
Given how large parts of publicly available text
are crawled to pretrain large language mod-
els (LLMs), data creators increasingly worry
about the inclusion of their proprietary data for
model training without attribution or licensing.
Their concerns are also shared by benchmark cu-
rators whose test-sets might be compromised. In
this paper, we present STAMP, a framework for de-
tecting dataset membership—i.e., determining the
inclusion of a dataset in the pretraining corpora of
LLMs. Given an original piece of content, our pro-
posal involves first generating multiple rephrases,
each embedding a watermark with a unique se-
cret key. One version is to be released publicly,
while others are to be kept private. Subsequently,
creators can compare model likelihoods between
public and private versions using paired statis-
tical tests to prove membership. We show that
our framework can successfully detect contami-
nation across four benchmarks which appear only
once in the training data and constitute less than
0.001% of the total tokens, outperforming sev-
eral contamination detection and dataset inference
baselines. We verify that STAMP preserves both
the semantic meaning and utility of the original
data. We apply STAMP to two real-world scenar-
ios to confirm the inclusion of paper abstracts and
blog articles in the pretraining corpora.
1. Introduction
To train large language models, much of the available text
from the internet is crawled, allegedly including copyrighted
material such as news articles and blogs (Grynbaum & Mac,
1Indian
Institute
of
Science
2Carnegie
Mellon
Uni-
versity
3DatologyAI.
Correspondence
to:
Saksham
Ras-
togi <iitdsaksham@gmail.com>, Pratyush Maini <pratyush-
maini.@cmu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author...
Organize the Web: Constructing Domains Enhances
Pre-Training Data Curation
Alexander Wettig 1 2 Kyle Lo 2 Sewon Min 3 2 Hannaneh Hajishirzi 2 4 Danqi Chen 1 Luca Soldaini 2
Abstract
Modern language models are trained on large,
unstructured datasets consisting of trillions of to-
kens and obtained by crawling the web.
The
unstructured nature makes it difficult to reason
about their contents and develop systematic ap-
proaches to data curation. In this paper, we un-
pack monolithic web corpora by developing tax-
onomies of their contents and organizing them
into domains. We introduce WebOrganizer, a
framework for organizing web pages in terms of
both their topic and format.
Using these two
complementary notions of domains, we automat-
ically annotate pre-training data by distilling an-
notations from a large language model into effi-
cient classifiers. This allows us to study how data
from different domains should be mixed to im-
prove models on downstream tasks, and we show
that we can combine insights about effective top-
ics and formats to further boost performance.
We demonstrate that our domain mixing also im-
proves existing methods that select data based on
quality. Furthermore, we study and compare how
quality-based methods will implicitly change the
domain mixture. Overall, our work demonstrates
that constructing and mixing domains provides a
valuable complement to quality-based data cura-
tion methods, opening new avenues for effective
and insightful pre-training data curation.
Website
weborganizer.allen.ai
Artifacts
hf.co/WebOrganizer
Code
CodeCreator/WebOrganizer
1Princeton Language and Intelligence, Princeton University
2Allen Institute for Artificial Intelligence 3University of Califor-
nia, Berkeley 4Paul G. Allen School of Computer Science & Engi-
neering, University of Washington. Correspondence to: Alexan-
der Wettig <awettig@cs.princeton.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. C...
Sequential Attention-based Sampling for
Histopathological Analysis
Tarun G
Indian Institute of Science, Bangalore, India
tarung@iisc.ac.in
Naman Malpani
Indian Institute of Science, Bangalore, India
namanmalpani@iisc.ac.in
Gugan Thoppe
Indian Institute of Science, Bangalore, India
gthoppe@iisc.ac.in
Devarajan Sridharan
Indian Institute of Science, Bangalore, India
sridhar@iisc.ac.in
Abstract
Deep neural networks are increasingly applied in automated histopathology. Yet,
whole-slide images (WSIs) are often acquired at gigapixel sizes, rendering them
computationally infeasible to analyze entirely at high resolution. Diagnostic labels
are largely available only at the slide-level, because expert annotation of images
at a finer (patch) level is both laborious and expensive. Moreover, regions with
diagnostic information typically occupy only a small fraction of the WSI, mak-
ing it inefficient to examine the entire slide at full resolution. Here, we propose
SASHA – Sequential Attention-based Sampling for Histopathological Analysis – a
deep reinforcement learning approach for efficient analysis of histopathological
images. First, SASHA learns informative features with a lightweight hierarchical,
attention-based multiple instance learning (MIL) model. Second, SASHA samples
intelligently and zooms selectively into a small fraction (10-20%) of high-resolution
patches to achieve reliable diagnoses. We show that SASHA matches state-of-the-
art methods that analyze the WSI fully at high resolution, albeit at a fraction of
their computational and memory costs. In addition, it significantly outperforms
competing, sparse sampling methods. We propose SASHA as an intelligent sam-
pling model for medical imaging challenges that involve automated diagnosis with
exceptionally large images containing sparsely informative features1.
1
Introduction
Deep learning models have emerged as a major frontier in automated medical imaging diagnosis. For
example, such models accurately identify tum...