AIS-Bench/AIS-Bench huggingface dataset:
Analyzing Similarity Metrics for Data Selection for
Language Model Pretraining
Dylan Sam∗
Carnegie Mellon University
Ayan Chakrabarti
Google Research
Afshin Rostamizadeh
Google Research
Srikumar Ramalingam
Google Research
Gui Citovsky
Google Research
Sanjiv Kumar
Google Research
Abstract
Measuring similarity between training examples is critical for curating high-quality
and diverse pretraining datasets for language models. However, similarity is
typically computed with a generic off-the-shelf embedding model that has been
trained for tasks such as retrieval. Whether these embedding-based similarity
metrics are well-suited for pretraining data selection remains largely unexplored.
In this paper, we propose a new framework to assess the suitability of a similarity
metric specifically for data curation in language model pretraining applications.
Our framework’s first evaluation criterion captures how well distances reflect
generalization in pretraining loss between different training examples. Next, we use
each embedding model to guide a standard diversity-based data curation algorithm
and measure its utility by pretraining a language model on the selected data and
evaluating downstream task performance. Finally, we evaluate the capabilities of
embeddings to distinguish between examples from different data sources. With
these evaluations, we demonstrate that standard off-the-shelf embedding models
are not well-suited for the pretraining data curation setting, underperforming even
remarkably simple embeddings that are extracted from models trained on the same
pretraining corpus. Our experiments are performed on the Pile, for pretraining
a 1.7B parameter language model on 200B tokens. We believe our analysis and
evaluation framework serves as a foundation for the future design of embeddings
that specifically reason about similarity in pretraining datasets.
1
Introduction
The recent success of language models [Brown et al., 2020, Chowdhery et al., 2023] is in no small
pa...
TabNAT: A Continuous-Discrete Joint Generative Framework for Tabular Data
Hengrui Zhang * 1 Liancheng Fang * 1 Qitian Wu 2 Philip S. Yu 1
Abstract
While autoregressive models dominate natural lan-
guage generation, their application to tabular data
remains limited due to two challenges: 1) tabular
data contains heterogeneous types, whereas au-
toregressive next-token (distribution) prediction
is designed for discrete data, and 2) tabular data
is column permutation-invariant, requiring flexi-
ble generation orders. Traditional autoregressive
models, with their fixed generation order, struggle
with tasks like missing data imputation, where the
target and conditioning columns vary. To address
these issues, we propose Diffusion-nested Non-
autoregressive Transformer (TabNAT), a hybrid
model combining diffusion processes and masked
generative modeling. For continuous columns,
TabNAT uses a diffusion model to parameterize
their conditional distributions, while for discrete
columns, it employs next-token prediction with
KL divergence minimization. A masked Trans-
former with bi-directional attention enables order-
agnostic generation, allowing it to learn the distri-
bution of target columns conditioned on arbitrary
observed columns. Extensive experiments on ten
datasets with diverse properties demonstrate Tab-
NAT’s superiority in both unconditional tabular
data generation and conditional missing data im-
putation tasks.
1. Introduction
The growing demand for synthetic tabular data in critical
applications such as privacy-preserving data sharing and
augmented machine learning pipelines (Fonseca & Bacao,
2023; Hernandez et al., 2022) has spurred significant inter-
est in deep generative models. While multiple architectures,
*Equal contribution
1Computer Science Department, Univer-
sity of Illinois at Chicago, Chicago, United States 2Eric and
Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cam-
bridge, United States. Correspondence to: Qitian Wu <wuqi-
tian@broadi...
Published as a conference paper at ICLR 2025
GAUSSIAN SPLATTING LUCAS-KANADE
Liuyue Xie1, Joel Julin1, Koichiro Niinuma2, L´aszl´o A. Jeni1
1Carnegie Mellon University, 2Fujitsu Research of America
{liuyuex,jjulin}@andrew.cmu.edu, laszlojeni@cmu.edu, kniinuma@fujitsu.com
ABSTRACT
Gaussian Splatting and its dynamic extensions are effective for reconstructing 3D
scenes from 2D images when there is significant camera movement to facilitate
motion parallax and when scene objects remain relatively static. However, in many
real-world scenarios, these conditions are not met. As a consequence, data-driven
semantic and geometric priors have been favored as regularizers, despite their bias
toward training data and their neglect of broader movement dynamics.
Departing from this practice, we propose a novel analytical approach that adapts
the classical Lucas-Kanade method to dynamic Gaussian splatting. By leveraging
the intrinsic properties of the forward warp field network, we derive an analytical
velocity field that, through time integration, facilitates accurate scene flow compu-
tation. This enables the precise enforcement of motion constraints on warp fields,
thus constraining both 2D motion and 3D positions of the Gaussians. Our method
excels in reconstructing highly dynamic scenes with minimal camera movement,
as demonstrated through experiments on both synthetic and real-world scenes. 1
Figure 1: Data-driven depth and optical flow supervisions produce inaccurate geometries. Instead,
we derive the analytical warp field of Gaussians to refine geometries and motions.
1
INTRODUCTION
The task of reconstructing dynamic 3D scenes from monocular video presents a significant challenge,
particularly when constrained by limited camera movement. This limitation results in a lack of
parallax and epipolar constraints, which are crucial for accurately estimating scene structures Gao
et al. (2022); Sch¨onberger & Frahm (2016). Implicit neural representation methods like NeRF have
shown...
Learning Utilities from Demonstrations in Markov Decision Processes
Filippo Lazzati 1 Alberto Maria Metelli 1
Abstract
Although it is well-known that humans com-
monly engage in risk-sensitive behaviors in the
presence of stochasticity, most Inverse Rein-
forcement Learning (IRL) models assume a risk-
neutral agent. As such, beyond piq introducing
model misspecification, piiq they do not permit
direct inference of the risk attitude of the ob-
served agent, which can be useful in many ap-
plications.
In this paper, we propose a novel
model of behavior to cope with these issues. By
allowing for risk sensitivity, our model allevi-
ates piq, and by explicitly representing risk at-
titudes through (learnable) utility functions, it
solves piiq.
Then, we characterize the partial
identifiability of an agent’s utility under the new
model and note that demonstrations from multi-
ple environments mitigate the problem. We de-
vise two provably-efficient algorithms for learn-
ing utilities in a finite-data regime, and we con-
clude with some proof-of-concept experiments to
validate both our model and our algorithms.
1. Introduction
The ultimate goal of Artificial Intelligence (AI) is to con-
struct artificial rational autonomous agents (Russell &
Norvig, 2010). Such agents will interact with each other
and with human beings to achieve the tasks that we assign
to them. In this vision, a crucial feature is being able to cor-
rectly model the observed behavior of other agents. This
allows a variety of applications: descriptive, to understand
the intent of the observed agent (Russell, 1998), predictive,
to anticipate the behavior of the observed agent (potentially
in new scenarios) (Arora & Doshi, 2021), and normative,
to imitate the observed agent because they are behaving in
the “right way” (Osa et al., 2018).
Nowadays, Inverse Reinforcement Learning (IRL) pro-
1Politecnico di Milano, Milan, Italy. Correspondence to: Fil-
ippo Lazzati <filippo.lazzati@polimi.it>.
Proceedings of th...
EVOREFUSE: Evolutionary Prompt Optimization for
Evaluation and Mitigation of LLM Over-Refusal to
Pseudo-Malicious Instructions
Xiaorui Wu1, Fei Li1, Xiaofeng Mao2, Xin Zhang3*, Li Zheng1,
Yuxiang Peng1, Chong Teng1, Donghong Ji1*, Zhuang Li4†
1 Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of
Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China
2 Ant Group 3 Ant International
4 School of Computing Technologies, Royal Melbourne Institute of Technology, Australia
1 {wuxiaorui, lifei_csnlp, zhengli, yxpeng, tengchong, dhji}@whu.edu.cn
2 mxf164419@antgroup.com, 3 evan.zx@ant-intl.com, 4 zhuang.li@rmit.edu.au
Abstract
Large language models (LLMs) frequently refuse to respond to pseudo-malicious
instructions: semantically harmless input queries triggering unnecessary LLM
refusals due to conservative safety alignment, signiﬁcantly impairing user ex-
perience. Collecting such instructions is crucial for evaluating and mitigating
over-refusals, but existing instruction curation methods, like manual creation
or instruction rewriting, either lack scalability or fail to produce sufﬁciently
diverse and effective refusal-inducing prompts.
To address these limitations,
we introduce EVOREFUSE, a prompt optimization approach that generates di-
verse pseudo-malicious instructions consistently eliciting conﬁdent refusals across
LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction
space in more diverse directions than existing methods via mutation strategies
and recombination, and iteratively evolves seed instructions to maximize evidence
lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel
datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions
that outperforms the next-best benchmark with 85.34% higher average refusal trig-
gering rate across 9 LLMs without a safety-prior system prompt, 34.86% greater
lexical diversity, and 40.03% improved LLM response co...
Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking
Through Options
Lakshmi Nair 1 Ian Trase 1 J. Mark Kim 1
Abstract
We present a novel reasoning approach called
Flow-of-Options (FoO), designed to address in-
trinsic biases in Large Language Models (LLMs).
Flow-of-Options enables LLMs to systematically
explore a diverse range of possibilities in their rea-
soning, as demonstrated by an FoO-based agentic
framework developed for autonomously solving
Machine Learning (ML) tasks.
FoO enforces
diversity in LLM solutions through compressed
and interpretable task representations, resulting in
improvements of 38.2% – 69.2% on standard data
science tasks, and 37.4% – 47.9% on therapeutic
chemistry tasks, as compared to state-of-the-art
baselines. With an overall operation cost under
$1 per task, our framework is well-suited for
cost-sensitive applications. Going beyond tabular
classification and regression, we show the broader
applicability of our FoO-based agentic system to
tasks such as reinforcement learning and image
generation. Our code is open-sourced at: https:
//github.com/flagshippioneering/
Flow-of-Options.
1. Introduction
Large Language Models (LLMs) have impacted automation
on a wide spectrum of tasks, motivating a growing body
of work in agentic system design, e.g., web-browsing (Yu
et al., 2025), idea generation (Li et al., 2024b), and data
science (Guo et al., 2024; Grosnit et al., 2024). Other ap-
proaches have focused on improving the inherent reasoning
capabilities of the LLMs, through Tree-of-Thoughts (ToT)
(Yao et al., 2024) or Chain-of-Thoughts (CoT) (Wei et al.,
2022). This paper seeks to bridge the two areas, exploring
how “thought-based” approaches can be extended to im-
prove agentic systems. Hence, we propose Flow-of-Options
1Flagship Pioneering, Cambridge MA, USA. Correspondence
to: Lakshmi Nair <lnair@flagshippioneering.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. ...
MAGELLAN: Metacognitive predictions of learning progress
guide autotelic LLM agents in large goal spaces
Loris Gaven 1 Thomas Carta 1 Cl´ement Romac 1 2 C´edric Colas 1 3 Sylvain Lamprier 4 Olivier Sigaud 5
Pierre-Yves Oudeyer 1
Abstract
Open-ended learning agents must efficiently prior-
itize goals in vast possibility spaces, focusing on
those that maximize learning progress (LP). When
such autotelic exploration is achieved by LLM
agents trained with online RL in high-dimensional
and evolving goal spaces, a key challenge for LP
prediction is modeling one’s own competence, a
form of metacognitive monitoring. Traditional
approaches either require extensive sampling or
rely on brittle expert-defined goal groupings. We
introduce MAGELLAN, a metacognitive frame-
work that lets LLM agents learn to predict their
competence and LP online. By capturing seman-
tic relationships between goals, MAGELLAN en-
ables sample-efficient LP estimation and dynamic
adaptation to evolving goal spaces through gener-
alization. In an interactive learning environment,
we show that MAGELLAN improves LP predic-
tion efficiency and goal prioritization, being the
only method allowing the agent to fully master
a large and evolving goal space. These results
demonstrate how augmenting LLM agents with
a metacognitive ability for LP predictions can ef-
fectively scale curriculum learning to open-ended
goal spaces.
1. Introduction
Humans are open-ended learners, continuously exploring
and developing new skills through their lifetime. A key
mechanism to enable this remarkable capacity is curiosity-
driven learning — the intrinsic motivation to explore for
1Inria (Flowers), University of Bordeaux, France 2Hugging
Face 3MIT, Computational Cognitive Science Lab, Cambridge,
MA, USA 4Univ Angers, LERIA, SFR MATHSTIC, F-49000
Angers, France 5Sorbonne Universit´e, ISIR, Paris, France. Corre-
spondence to: Loris Gaven <loris.gaven@inria.fr>.
Proceedings of the 42 nd International Conference on Machine
Learnin...
SD-KDE: Score-Debiased Kernel Density Estimation
Elliot L. Epstein1∗
Rajat Dwaraknath1∗
Thanawat Sornwanee1∗
John Winnicki1∗
Jerry Weihong Liu1∗
1Stanford University, Stanford, CA 94305, USA
{epsteine, rajatvd, tsornwanee, winnicki, jwl50}@stanford.edu
Abstract
We propose a novel method for density estimation that leverages an estimated score
function to debias kernel density estimation (SD-KDE). In our approach, each data
point is adjusted by taking a single step along the score function with a specific
choice of step size, followed by standard KDE with a modified bandwidth. The
step size and modified bandwidth are chosen to remove the leading order bias in the
KDE, improving the asymptotic convergence rate. Our experiments on synthetic
tasks in 1D, 2D and on MNIST, demonstrate that our proposed SD-KDE method
significantly reduces the mean integrated squared error compared to the standard
Silverman KDE, even with noisy estimates in the score function. These results
underscore the potential of integrating score-based corrections into nonparametric
density estimation.
1
Introduction
Kernel density estimation (KDE) (Rosenblatt, 1956; Parzen, 1962) is a widely used nonparametric
method for estimating an unknown probability density function from a finite set of data points.
The classical KDE effectively smooths the data by convolving with a kernel function, such as the
Gaussian kernel, and then normalizing the result to obtain a density estimate. KDE finds application in
diverse fields such as anomaly detection, clustering (Campello et al., 2013), data visualization (Scott,
2012), nonparametric statistical inference (Guerre et al., 2000; Zhang et al., 2008), and dynamical
systems (Hang et al., 2018).
The classical KDE suffers from a well-known bias-variance trade-off, controlled by the choice of ker-
nel bandwidth (Silverman, 1986). Larger bandwidths lead to smoother estimates with lower variance
but higher bias, while smaller bandwidths yield more variable estimates wi...
Published as a conference paper at ICLR 2025
OPTIMAL TRANSPORT FOR TIME-SERIES IMPUTATION
Hao Wang1
Zhengnan Li2
Haoxuan Li3
Xu Chen4
Mingming Gong5,6
Bin Chen7
Zhichao Chen1,∗
1Zhejiang University
2Communication University of China
3Peking University
4Renmin University of China
5The University of Melbourne
6Mohamed bin Zayed University of Artificial Intelligence
7University of Chinese Academy of Sciences
Ho-ward@outlook.com
fmlyd@cuc.edu.cn
justuschencse@zju.edu.cn
ABSTRACT
Missing data imputation through distribution alignment has demonstrated advan-
tages for non-temporal datasets but exhibits suboptimal performance in time-series
applications. The primary obstacle is crafting a discrepancy measure that simulta-
neously (1) captures temporal patterns—accounting for periodicity and temporal
dependencies inherent in time-series—and (2) accommodates non-stationarity, en-
suring robustness amidst multiple coexisting temporal patterns. In response to these
challenges, we introduce the Proximal Spectrum Wasserstein (PSW) discrepancy, a
novel discrepancy tailored for comparing two sets of time-series based on optimal
transport. It incorporates a pairwise spectral distance to encapsulate temporal
patterns, and a selective matching regularization to accommodate non-stationarity.
Subsequently, we develop the PSW for Imputation (PSW-I) framework, which iter-
atively refines imputation results by minimizing the PSW discrepancy. Extensive
experiments demonstrate that PSW-I effectively accommodates temporal patterns
and non-stationarity, outperforming prevailing time-series imputation methods.
Code is available at https://github.com/FMLYD/PSW-I.
1
INTRODUCTION
The incompleteness of time-series data is a widespread issue in many fundamental fields. For example,
in healthcare (Prosperi et al., 2020), users utilize monitoring devices to track their health data, but
these records may be incomplete due to occasional device failures or disconnections. Similarly, in
manufacturing and...
Kinetic Langevin Diffusion for Crystalline Materials Generation
Franc¸ois Cornet * 1 Federico Bergamin * 1 2
Arghya Bhowmik 1 Juan Maria Garcia Lastra 1 Jes Frellsen 1 2 Mikkel N. Schmidt 1 2
Abstract
Generative modeling of crystalline materials us-
ing diffusion models presents a series of chal-
lenges: the data distribution is characterized by in-
herent symmetries and involves multiple modali-
ties, with some defined on specific manifolds. No-
tably, the treatment of fractional coordinates rep-
resenting atomic positions in the unit cell requires
careful consideration, as they lie on a hypertorus.
In this work, we introduce Kinetic Langevin Dif-
fusion for Materials (KLDM), a novel diffusion
model for crystalline materials generation, where
the key innovation resides in the modeling of the
coordinates. Instead of resorting to Riemannian
diffusion on the hypertorus directly, we generalize
Trivialized Diffusion Model (TDM) to account for
the symmetries inherent to crystals. By coupling
coordinates with auxiliary Euclidean variables
representing velocities, the diffusion process is
now offset to a flat space. This allows us to effec-
tively perform diffusion on the hypertorus while
providing a training objective that accounts for
the periodic translation symmetry of the true data
distribution. We evaluate KLDM on both Crystal
Structure Prediction (CSP) and De-novo Genera-
tion (DNG) tasks, demonstrating its competitive
performance with current state-of-the-art models.
1. Introduction
The discovery of novel compounds with desired properties is
critical to several scientific fields, such as molecular discov-
ery (Bilodeau et al., 2022) and materials design (Merchant
et al., 2023; Zeni et al., 2025). In the case of crystalline
materials, the search space is vast, but only a fraction of it is
physically plausible. The main challenges are to efficiently
*Equal contribution
1Technical University of Denmark, Kon-
gens Lyngby, Denmark 2Pioneer Center for Artificial Intellig...
Fourier Token Merging: Understanding and
Capitalizing Frequency Domain for Efficient Image
Generation
Jiesong Liu
Department of Computer Science
North Carolina State University
Raleigh, NC 27606
jliu93@ncsu.edu
Xipeng (Gracen) Shen
Department of Computer Science
North Carolina State University
Raleigh, NC 27606
xshen5@ncsu.edu
Abstract
Image generation requires intensive computations and faces challenges due to long
latency. Exploiting redundancy in the input images and intermediate representa-
tions throughout the neural network pipeline is an effective way to accelerate image
generation. Token merging (ToMe) exploits similarities among input tokens by
clustering them and merges similar tokens into one, thus significantly reducing the
number of tokens that are fed into the transformer block. This work introduces
Fourier Token Merging, a new method for understanding and capitalizing frequency
domain for efficient image generation. By introducing frequency token merging, we
find that transforming the token into the frequency domain representation for clus-
tering can better exert the ability of clustering based on the underlying redundancy
after de-correlation. Through analytical and empirical studies, we demonstrate the
benefits of using Fourier clustering over the original time domain clustering. We
experimented Fourier Token Merging on the stable diffusion model, and the results
show up to 25% reduction in latency without impairing image quality. The code is
available at https://github.com/Fred1031/Fourier-Token-Merging.
1
Introduction
Image generation has achieved impressive results through advanced generative models such as stable
diffusion [22], but these models face significant computational challenges and long latency, especially
when deployed on resource-constrained devices [5, 12, 27]. A primary avenue for addressing
these issues involves exploiting redundancies within input images and intermediate neural network
representations to streamline computation. T...
LIMOPro: Reasoning Refinement for Efficient and
Effective Test-time Scaling
Yang Xiao1
Jiashuo Wang1
Ruifeng Yuan1
Chunpu Xu1
Kaishuai Xu1
Wenjie Li1†
Pengfei Liu2,3†
1The Hong Kong Polytechnic University
2Shanghai Jiao Tong University
3SII
yang-alan.xiao@connect.polyu.hk
csjwang@comp.polyu.edu.hk
Abstract
Large language models (LLMs) have demonstrated remarkable reasoning capa-
bilities through test-time scaling approaches, particularly when fine-tuned with
chain-of-thought (CoT) data distilled from more powerful large reasoning models
(LRMs). However, these reasoning chains often contain verbose elements that
mirror human problem-solving, categorized as progressive reasoning (the essential
solution development path) and functional elements (verification processes, alter-
native solution approaches, and error corrections). While progressive reasoning
is crucial, the functional elements significantly increase computational demands
during test-time inference. We introduce PIR (Perplexity-based Importance Re-
finement), a principled framework that quantitatively evaluates the importance of
each reasoning step based on its impact on answer prediction confidence. PIR sys-
tematically identifies and selectively prunes only low-importance functional steps
while preserving progressive reasoning components, creating optimized training
data that maintains the integrity of the core solution path while reducing verbosity.
Models fine-tuned on PIR-optimized data exhibit superior test-time scaling proper-
ties, generating more concise reasoning chains while achieving improved accuracy
(+0.9% to +6.6%) with significantly reduced token usage (-3% to -41%) across
challenging reasoning benchmarks (AIME, AMC, and GPQA Diamond). Our
approach demonstrates strong generalizability across different model sizes, data
sources, and token budgets, offering a practical solution for deploying reasoning-
capable LLMs in scenarios where efficient test-time scaling, response time, and
computationa...