AIS-Bench/AIS-Bench huggingface dataset:
Published as a conference paper at ICLR 2025
GREATER:
GRADIENTS OVER REASONING MAKES
SMALLER LANGUAGE MODELS STRONG PROMPT OP-
TIMIZERS
Sarkar Snigdha Sarathi Das†
Ryo Kamoi†
Bo Pang⋄
Yusen Zhang†
Caiming Xiong⋄
Rui Zhang†
†The Pennsylvania State University
⋄Salesforce Research
{sfd5525, rmz5227}@psu.edu
ABSTRACT
The effectiveness of large language models (LLMs) is closely tied to the design of
prompts, making prompt optimization essential for enhancing their performance
across a wide range of tasks. Many existing approaches to automating prompt
engineering rely exclusively on textual feedback, refining prompts based solely
on inference errors identified by large, computationally expensive LLMs. Unfor-
tunately, smaller models struggle to generate high-quality feedback, resulting in
complete dependence on large LLM judgment. Moreover, these methods fail to
leverage more direct and finer-grained information, such as gradients, due to op-
erating purely in text space. To this end, we introduce GREATER, a novel prompt
optimization technique that directly incorporates gradient information over task-
specific reasoning.
By utilizing task loss gradients, GREATER enables self-
optimization of prompts for open-source, lightweight language models without the
need for costly closed-source LLMs. This allows high-performance prompt opti-
mization without dependence on massive LLMs, closing the gap between smaller
models and the sophisticated reasoning often needed for prompt refinement. Ex-
tensive evaluations across diverse reasoning tasks including BBH, GSM8k, and
FOLIO demonstrate that GREATER consistently outperforms previous state-of-
the-art prompt optimization methods, even those reliant on powerful LLMs. Ad-
ditionally, GREATER-optimized prompts frequently exhibit better transferability
and, in some cases, boost task performance to levels comparable to or surpass-
ing those achieved by larger language models, highlighting the effectiveness of
prompt optimization guided ...
Compositional Generalization via Forced Rendering of Disentangled Latents
Qiyao Liang* 1 Daoyuan Qian* 2 Liu Ziyin 1 3 Ila Fiete 1
Abstract
Composition—the ability to generate myriad vari-
ations from finite means—is believed to underlie
powerful generalization. However, compositional
generalization remains a key challenge for deep
learning. A widely held assumption is that learn-
ing disentangled (factorized) representations nat-
urally supports this kind of extrapolation. Yet,
empirical results are mixed, with many genera-
tive models failing to recognize and compose fac-
tors to generate out-of-distribution (OOD) sam-
ples. In this work, we investigate a controlled 2D
Gaussian “bump” generation task with fully dis-
entangled (x, y) inputs, demonstrating that stan-
dard generative architectures still fail in OOD
regions when training with partial data, by re-
entangling latent representations in subsequent
layers. By examining the model’s learned kernels
and manifold geometry, we show that this failure
reflects a “memorization” strategy for generation
via data superposition rather than via composition
of the true factorized features. We show that when
models are forced—through architectural modi-
fications with regularization or curated training
data—to render the disentangled latents into the
full-dimensional representational (pixel) space,
they can be highly data-efficient and effective at
composing in OOD regions. These findings un-
derscore that disentangled latents in an abstract
representation are insufficient and show that if
models can represent disentangled factors directly
in the output representational space, it can achieve
robust compositional generalization.
*Equal contribution
1Massachusetts Institute of Technology,
Cambridge MA, USA 02139 2University of Cambridge, Cam-
bridge CB2 1EW, U.K 3NTT Research. Correspondence to: Qiyao
Liang <qiyao@mit.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, ...
Survival Analysis via Density Estimation
Hiroki Yanagisawa 1 Shunta Akiyama 1
Abstract
This paper introduces a framework for survival
analysis by reinterpreting it as a form of density
estimation. Our algorithm post-processes density
estimation outputs to derive survival functions,
enabling the application of any density estimation
model to effectively estimate survival functions.
This approach broadens the toolkit for survival
analysis and enhances the flexibility and applica-
bility of existing techniques for density estimation.
Our framework is versatile enough to handle var-
ious survival analysis scenarios, including com-
peting risk models for multiple event types. It
can also address dependent censoring when prior
knowledge of the dependency between event time
and censoring time is available in the form of
a copula. In the absence of such information,
our framework can estimate the upper and lower
bounds of survival functions, accounting for the
associated uncertainty.
1. Introduction
Survival analysis constitutes a statistical methodology ex-
tensively employed across diverse domains, including
medicine, engineering, and social sciences, to analyze and
predict the probability distribution of the time until the oc-
currence of an event of interest. In numerous real-world
scenarios, the phenomenon of competing risks emerges
when an individual is subject to multiple mutually exclu-
sive events, where the occurrence of one event precludes
the observation or alters the probability of the other events.
The field of survival analysis with competing risks has a
rich historical foundation, as extensively reviewed in survey
papers (Wang et al., 2019; Wiegrebe et al., 2024).
Survival analysis with K competing risks can be conceptual-
ized as a variant of density estimation, a subfield of machine
learning aimed at estimating the probability distribution of a
1CyberAgent, Tokyo, Japan. Correspondence to: Hiroki Yanagi-
sawa <yanagisawa hiroki@cyberagent.co.jp>.
Proceedin...
Conditional Diffusion Model with Nonlinear Data Transformation for
Time Series Forecasting
J Rishi * 1 GVS Mothish * 1 Deepak Subramani 1
Abstract
Time-series forecasting finds application across
domains such as finance, climate science, and
energy systems. We introduce the Conditional
Diffusion with Nonlinear Data Transformation
Model (CN-Diff), a generative framework that em-
ploys novel nonlinear transformations and learn-
able conditions in the forward process for time
series forecasting. A new loss formulation for
training is proposed, along with a detailed deriva-
tion of both forward and reverse process. The
new additions improve the diffusion model’s ca-
pacity to capture complex time series patterns,
thus simplifying the reverse process. Our novel
condition facilitates learning an efficient prior dis-
tribution. This also reduces the gap between the
true negative log-likelihood and its variational ap-
proximation. CN-Diff is shown to perform better
than other leading time series models on nine real-
world datasets. Ablation studies are conducted to
elucidate the role of each component of CN-Diff.
1. Introduction
Time series forecasting has become an essential element in
modern data-driven decision-making processes, spanning
a wide range of fields including financial pricing analy-
sis (Kim, 2003), economics (Henrique et al., 2019), trans-
portation planning (Huang et al., 2023), energy (Dumas
et al., 2022) and various other fields (Li et al., 2024a; Rasul
et al., 2022). There has been substantial advancement in
time series modeling, beginning with traditional statistical
techniques such as ARIMA and state-space models, which
initially dominated forecasting efforts. Later, various archi-
tectures of deep neural networks, such as recurrent neural
*Equal contribution 1Department of Computational and Data
Sciences, Indian Institute of Science, Bengaluru, India. Corre-
spondence to: J Rishi <rishij@iisc.ac.in>, GVS Mothish <moth-
ishg@iisc.ac.in>, Deepak Subrama...
GauSAM: Contour-Guided 2D Gaussian Fields for
Multi-Scale Medical Image Segmentation with
Segment Anything
Jinxuan Wu1∗
Jiange Wang1∗
Dongdong Zhang1†
1School of Computer Science and Technology
Tongji University
{2154146, 2252699, ddzhang}@tongji.edu.cn
Abstract
Effective multiscale medical image segmentation requires simultaneously preserv-
ing smooth spatial continuity and accurately delineating high-frequency bound-
aries, yet pixel-wise decoders often fail to maintain this balance consistently
across varying resolutions. We introduce GauSAM, which seamlessly integrates
contour-guided 2D Gaussian probability fields into the Segment Anything Model to
address these challenges. In our framework, segmentation masks are parameterized
as continuous probability fields of learnable 2D Gaussian primitives, enforcing
spatially smooth and structurally consistent. Contourlet transforms extract rich
multidirectional frequency information, notably edges and fine textures, which
dynamically guide the spatial distribution of Gaussian primitives to substantially
improve boundary fidelity in complex structures. The incorporation of these high-
frequency contour priors also enriches the expressive capacity of the SAM image
encoder. Extensive experiments on diverse 2D medical segmentation tasks con-
firm that GauSAM consistently delivers robust generalization and state-of-the-art
performance with only 1.2M trainable parameters. The official implementation of
GauSAM is publicly available at https://github.com/Quinten-Wu504/GauSAM.
1
Introduction
Medical image segmentation demands high precision and smooth boundary delineation, as anatom-
ical structures often exhibit ambiguous edges and segmentation errors can lead to serious clinical
consequences [1]. To address this, a wide range of methods have been proposed, including classical
architectures such as U-Net [2] and several fine-tuned variants of the Segment Anything Model
(SAM) [3]. These approaches typically rely on fixed-resoluti...
Average Sensitivity of Hierarchical k-Median Clustering
Shijie Li 1 Weiqiang He 1 Ruobing Bai 1 Pan Peng 1
Abstract
Hierarchical clustering is a widely used method
for unsupervised learning with numerous applica-
tions. However, in the application of modern algo-
rithms, the datasets studied are usually large and
dynamic. If the hierarchical clustering is sensitive
to small perturbations of the dataset, the usabil-
ity of the algorithm will be greatly reduced. In
this paper, we focus on the hierarchical k -median
clustering problem, which bridges hierarchical
and centroid-based clustering while offering the-
oretical appeal, practical utility, and improved
interpretability. We analyze the average sensitiv-
ity of algorithms for this problem by measuring
the expected change in the output when a random
data point is deleted. We propose an efficient al-
gorithm for hierarchical k-median clustering and
theoretically prove its low average sensitivity and
high clustering quality. Additionally, we show
that single linkage clustering and a deterministic
variant of the CLNSS algorithm exhibit high aver-
age sensitivity, making them less stable. Finally,
we validate the robustness and effectiveness of
our algorithm through experiments.
1. Introduction
Hierarchical clustering, as discussed in (McQuitty, 1957;
Hastie et al., 2009), is one of the most commonly used
clustering strategies. It produces a set of nested clusters,
organized into a tree-like structure known as a dendrogram.
In this structure, each leaf node typically encompasses only
a single object, whereas each internal node represents the
union of its child nodes. By exploring the dendrogram’s
multiple levels, one can identify clusters with varying levels
of granularity. Consequently, hierarchical clustering is adept
at detecting and analyzing complex data structures. Hier-
archical clustering has attracted great interest in computer
1School of Computer Science and Technology, University of
Science and Technology of...
Simultaneous Multi-Robot Motion Planning with Projected Diffusion Models
Jinhao Liang 1 Jacob K. Christopher 1 Sven Koenig 2 Ferdinando Fioretto 1
Abstract
Recent advances in diffusion models hold signifi-
cant potential in robotics, enabling the generation
of diverse and smooth trajectories directly from
raw representations of the environment. Despite
this promise, applying diffusion models to mo-
tion planning remains challenging due to their
difficulty in enforcing critical constraints, such
as collision avoidance and kinematic feasibility.
These limitations become even more pronounced
in Multi-Robot Motion Planning (MRMP), where
multiple robots must coordinate in shared spaces.
To address these challenges, this work proposes
Simultaneous MRMP Diffusion (SMD), a novel
approach integrating constrained optimization
into the diffusion sampling process to produce
collision-free, kinematically feasible trajectories.
Additionally, the paper introduces a comprehen-
sive MRMP benchmark to evaluate trajectory
planning algorithms across scenarios with vary-
ing robot densities, obstacle complexities, and
motion constraints. Experimental results show
SMD consistently outperforms classical and other
learning-based motion planners, achieving higher
success rates and efficiency in complex multi-
robot environments. The code and implementa-
tion are available at https://github.com/
RAISELab-atUVA/Diffusion-MRMP.
1. Introduction
Multi-Robot Motion Planning (MRMP) is a fundamental
problem in robotics and autonomous systems, where the
goal is to compute collision-free paths for multiple robots
navigating shared environments (Luo et al., 2024; Shaoul
et al., 2025). MRMP has widespread applications, from
autonomous vehicles to warehouse logistics and search-and-
1Department of Computer Science, University of Virginia, Char-
lottesville, VA 22903, USA 2Department of Computer Science,
University of California, Irvine, CA 92697, USA. Correspondence
to: Ferdinando Fioretto <fioretto@vir...
On Evaluating Policies for Robust POMDPs
Merlijn Krale∗
Radboud University
Nijmegen, The Netherlands
merlijn.krale@ru.nl
Eline M. Bovy∗
Radboud University
Nijmegen, The Netherlands
eline.bovy@ru.nl
Maris F. L. Galesloot∗
Radboud University
Nijmegen, The Netherlands
maris.galesloot@ru.nl
Thiago D. Simão
Eindhoven University of Technology
Eindhoven, The Netherlands
t.simao@tue.nl
Nils Jansen
Ruhr-University Bochum & Radboud University
Bochum, Germany & Nijmegen, The Netherlands
n.jansen@rub.de
Abstract
Robust partially observable Markov decision processes (RPOMDPs) model sequen-
tial decision-making problems under partial observability, where an agent must
be robust against a range of dynamics. RPOMDPs can be viewed as a two-player
game between an agent, who selects actions, and nature, who adversarially selects
the dynamics. Evaluating an agent policy requires finding an adversarial nature
policy, which is computationally challenging. In this paper, we advance the eval-
uation of agent policies for RPOMDPs in three ways. First, we discuss suitable
benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be
found by considering only subsets of nature policies, making them easier to solve.
We formalize this concept of solvability and construct three benchmarks that are
only solvable for expressive sets of nature policies. Second, we describe a new
method to evaluate agent policies for RPOMDPs by solving an equivalent MDP.
Third, we lift two well-known upper bounds from POMDPs to RPOMDPs, which
can be used to efficiently approximate the optimality gap of a policy and serve as
baselines. Our experimental evaluation shows that (1) our proposed benchmarks
cannot be solved by assuming naive nature policies, (2) our method of evaluating
policies is accurate, and (3) the upper bounds provide solid baselines for evaluation.
1
Introduction
Partially observable Markov decision processes [POMDPs; 25] are ubiquitous for representing
sequential decision-making prob...
Permutation-Free High-Order Interaction Tests
Zhaolu Liu 1 Robert L. Peach 2 3 Mauricio Barahona 1
Abstract
Kernel-based hypothesis tests offer a flexible, non-
parametric tool to detect high-order interactions in
multivariate data, beyond pairwise relationships.
Yet the scalability of such tests is limited by the
computationally demanding permutation schemes
used to generate null approximations. Here we
introduce a family of permutation-free high-order
tests for joint independence and partial factorisa-
tions of d variables. Our tests eliminate the need
for permutation-based approximations by lever-
aging V-statistics and a novel cross-centring tech-
nique to yield test statistics with a standard normal
limiting distribution under the null. We present
implementations of the tests and showcase their
efficacy and scalability through synthetic datasets.
We also show applications inspired by causal dis-
covery and feature selection, which highlight both
the importance of high-order interactions in data
and the need for efficient computational methods.
1. Introduction
Many complex real-world systems involve interactions be-
yond pairwise relationships, necessitating an explicit under-
standing of interactions among three or more entities (Bat-
tiston et al., 2020; 2021; Rosas et al., 2022; Thibeault et al.,
2024). Such high-order interactions amongst d > 2 vari-
ables have been shown to affect processes in social (Benson
et al., 2016), ecological (Grilli et al., 2017) and biological
systems (Luff et al., 2024; Arnaudon et al., 2022), often
in ways that cannot be reduced to combinations of pair-
wise effects (Schaub et al., 2020; Bick et al., 2023). Given
the relative scarcity of relational datasets that record high-
order phenomena, it is important to develop unsupervised,
mathematically grounded methods for their detection in ob-
1Department of Mathematics, Imperial College London,
United Kingdom 2Department of Neurology, University Hospi-
tal W¨urzburg, Germany 3Depar...
AAAR-1.0: Assessing AI’s Potential to Assist Research
Renze Lou 1 Hanzi Xu 2 Sijia Wang 3 Jiangshu Du 4 Ryo Kamoi 1 Xiaoxin Lu 1 Jian Xie 5 Yuxuan Sun 5
Yusen Zhang 1 Jihyun Janice Ahn 1 Hongchao Fang 1 Zhuoyang Zou 1 Wenchao Ma 1 Xi Li 6 Kai Zhang 7
Congying Xia 5 Lifu Huang 3 Wenpeng Yin 1
Abstract
Numerous studies have assessed the proficiency
of AI systems, particularly large language mod-
els (LLMs), in facilitating everyday tasks such
as email writing, question answering, and cre-
ative content generation. However, researchers
face unique challenges and opportunities in lever-
aging LLMs for their own work, such as brain-
storming research ideas, designing experiments,
and writing or reviewing papers. In this study,
we introduce AAAR-1.0, a benchmark dataset
designed to evaluate LLM performance in three
fundamental, expertise-intensive research tasks:
(i) EQUATIONINFERENCE, assessing the correct-
ness of equations based on the contextual infor-
mation in paper submissions; (ii) EXPERIMENT-
DESIGN, designing experiments to validate re-
search ideas and solutions; and (iii) PAPERWEAK-
NESS, identifying weaknesses in paper submis-
sions. AAAR-1.0 differs from prior benchmarks
in two key ways: first, it is explicitly research-
oriented, with tasks requiring deep domain exper-
tise; second, it is researcher-oriented, mirroring
the primary activities that researchers engage in
on a daily basis. An evaluation of both open-
source and closed-source LLMs reveals their po-
tential as well as limitations in conducting so-
phisticated research tasks. We will keep iterating
AAAR-1.0 to new versions. Project Webpage:
https://renzelou.github.io/AAAR-1.0/
Work done prior to Jiangshu joining Amazon. 1Pennsylvania
State University;
2Netflix;
3University of California, Davis;
4University
of
Illinois
Chicago;
5Individual
Researcher;
6University of Alabama at Birmingham; 7Ohio State University.
Correspondence to: Renze Lou <renze.lou@psu.edu>, Wenpeng
Yin <wenpeng@psu.edu>.
Procee...
Reinforcement Learning with Random Time Horizons
Enric Ribera Borrell * 1 2 Lorenz Richter * 1 3 Christof Sch¨utte 1 2
Abstract
We extend the standard reinforcement learning
framework to random time horizons. While the
classical setting typically assumes finite and deter-
ministic or infinite runtimes of trajectories, we ar-
gue that multiple real-world applications naturally
exhibit random (potentially trajectory-dependent)
stopping times. Since those stopping times typi-
cally depend on the policy, their randomness has
an effect on policy gradient formulas, which we
(mostly for the first time) derive rigorously in this
work both for stochastic and deterministic poli-
cies. We present two complementary perspectives,
trajectory or state-space based, and establish con-
nections to optimal control theory. Our numerical
experiments demonstrate that using the proposed
formulas can significantly improve optimization
convergence compared to traditional approaches.
1. Introduction
The goal in reinforcement learning is to identify policies π
that maximize the expected return
J(π) = Eπ
" N
X
n=0
rn(Sn, An)
#
,
(1)
where the states Sn and the actions An run until a time N.
While the classical framework considers the case where N
is either finite and deterministic or infinite, runtimes can as
well be random. To name two examples, one could consider
robots that aim to reach a target in a noisy environment or
one could play a game whose terminating state depends on
reaching a certain level. Furthermore, considering stochastic
policies (as commonly done in practice in order to balance
exploration and exploitation) leads to the fact that runtimes
*Equal contribution 1Zuse Institute Berlin, 14195 Berlin, Ger-
many 2Institute of Mathematics, Free University Berlin, 14195
Berlin, Germany 3dida Datenschmiede GmbH, 10827 Berlin,
Germany.
Correspondence to:
Enric Ribera Borrell <rib-
era.borrell@zib.de>, Lorenz Richter <richter@zib.de>.
Proceedings of the 42 nd International Conference ...
PAC Learning with Improvements
Idan Attias 1 2 Avrim Blum 2 Keziah Naggita 2 Donya Saless 2 Dravyansh Sharma 2 3 Matthew Walter 2
Abstract
One of the most basic lower bounds in machine
learning is that in nearly any nontrivial setting, it
takes at least 1/ϵ samples to learn to error ϵ (and
more, if the classifier being learned is complex).
However, suppose that data points are agents who
have the ability to improve by a small amount if
doing so will allow them to receive a (desired)
positive classification. In that case, we may ac-
tually be able to achieve zero error by just being
“close enough”. For example, imagine a hiring test
used to measure an agent’s skill at some job such
that for some threshold θ, agents who score above
θ will be successful and those who score below
θ will not (i.e., learning a threshold on the line).
Suppose also that by putting in effort, agents can
improve their skill level by some small amount r.
In that case, if we learn an approximation ˆθ of θ
such that θ ≤ˆθ ≤θ + r and use it for hiring, we
can actually achieve error zero, in the sense that
(a) any agent classified as positive is truly quali-
fied, and (b) any agent who truly is qualified can
be classified as positive by putting in effort. Thus,
the ability for agents to improve has the potential
to allow for a goal one could not hope to achieve
in standard models, namely zero error. In this
paper, we explore this phenomenon more broadly,
giving general results and examining under what
conditions the ability of agents to improve can
allow for a reduction in the sample complexity
of learning, or alternatively, can make learning
harder. We also examine both theoretically and
empirically what kinds of improvement-aware al-
gorithms can take into account agents who have
the ability to improve to a limited extent when it
is in their interest to do so.
Authors are listed alphabetically.
1University of Illinois at
Chicago 2Toyota Technological Institute at Chicago 3Northwestern
University...