AIS-Bench/AIS-Bench huggingface dataset:
Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations
Evan Frick∗1 Connor Chen∗1 Joseph Tennyson∗1 Tianle Li∗1 Wei-Lin Chiang∗1
Anastasios N. Angelopoulos∗1 Ion Stoica 1
Abstract
Large language model (LLM) evaluations typi-
cally rely on aggregated metrics like accuracy
or human preference, averaging across users
and prompts.
This averaging obscures user-
and prompt-specific variations in model perfor-
mance.
To address this, we propose Prompt-
to-Leaderboard (P2L), a method that produces
leaderboards specific to a prompt or set of
prompts. The core idea is to train an LLM taking
natural language prompts as input to output a vec-
tor of Bradley-Terry coefficients which are then
used to predict the human preference vote. The
resulting prompt-dependent leaderboards allow
for unsupervised task-specific evaluation, opti-
mal routing of queries to models, personalization,
and automated evaluation of model strengths and
weaknesses. Data from Chatbot Arena suggest
that P2L better captures the nuanced landscape of
language model performance than the averaged
leaderboard. Furthermore, our findings suggest
that P2L’s ability to produce prompt-specific eval-
uations follows a power law scaling similar to that
observed in LLMs themselves. In January 2025,
the router we trained based on this methodology
achieved the #1 spot on the Chatbot Arena leader-
board. Our code is available at this GitHub link:
https://github.com/lmarena/p2l.
1. Introduction
Evaluating the real-world performance of large language
models is an unresolved challenge. A growing suite of
benchmarks, including MMLU (Hendrycks et al., 2020),
MMLU-Pro (Wang et al., 2024), and GPQA (Rein et al.,
2023), seek to address the challenge by reporting task-
specific performance metrics, such as multiple-choice
*Equal contribution 1University of California, Berkeley. Corre-
spondence to: Evan Frick <evanfrick@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267...
Published as a conference paper at ICLR 2025
NARROWING INFORMATION BOTTLENECK THEORY
FOR MULTIMODAL IMAGE-TEXT REPRESENTATIONS
INTERPRETABILITY
Zhiyu Zhu1, Zhibo Jin1, Jiayu Zhang2, Nan Yang3, Jiahao Huang3 ,
Jianlong Zhou1 & Fang Chen1 ∗
University of Technology Sydney1, SuZhouYierqi2, University of Sydney3
ABSTRACT
The task of identifying multimodal image-text representations has garnered in-
creasing attention, particularly with models such as CLIP (Contrastive Language-
Image Pretraining), which demonstrate exceptional performance in learning com-
plex associations between images and text. Despite these advancements, ensuring
the interpretability of such models is paramount for their safe deployment in real-
world applications, such as healthcare. While numerous interpretability methods
have been developed for unimodal tasks, these approaches often fail to transfer ef-
fectively to multimodal contexts due to inherent differences in the representation
structures. Bottleneck methods, well-established in information theory, have been
applied to enhance CLIP’s interpretability. However, they are often hindered by
strong assumptions or intrinsic randomness. To overcome these challenges, we
propose the Narrowing Information Bottleneck Theory, a novel framework that
fundamentally redefines the traditional bottleneck approach. This theory is specif-
ically designed to satisfy contemporary attribution axioms, providing a more ro-
bust and reliable solution for improving the interpretability of multimodal models.
In our experiments, compared to state-of-the-art methods, our approach enhances
image interpretability by an average of 9%, text interpretability by an average of
58.83%, and accelerates processing speed by 63.95%. Our code is publicly acces-
sible at https://github.com/LMBTough/NIB.
1
INTRODUCTION
CLIP (Contrastive Language-Image Pretraining) has rapidly become a pivotal model in the field
of multimodal learning, especially excelling in its ability to connect th...
Modiﬁed K-means Algorithm with Local Optimality Guarantees
Mingyi Li 1 Michael R. Metel 2 Akiko Takeda 1 3
Abstract
The K-means algorithm is one of the most widely
studied clustering algorithms in machine learn-
ing.
While extensive research has focused on
its ability to achieve a globally optimal solution,
there still lacks a rigorous analysis of its local op-
timality guarantees. In this paper, we ﬁrst present
conditions under which the K-means algorithm
converges to a locally optimal solution. Based on
this, we propose simple modiﬁcations to the K-
means algorithm which ensure local optimality in
both the continuous and discrete sense, with the
same computational complexity as the original
K-means algorithm. As the dissimilarity mea-
sure, we consider a general Bregman divergence,
which is an extension of the squared Euclidean
distance often used in the K-means algorithm.
Numerical experiments conﬁrm that the K-means
algorithm does not always ﬁnd a locally optimal
solution in practice, while our proposed methods
provide improved locally optimal solutions with
reduced clustering loss. Our code is available at
https://github.com/lmingyi/LO-K-means.
1. Introduction
K-means clustering is one of the most widely recog-
nized methods for partitioning datasets into homogeneous
groups. The K-means clustering problem, formulated as
a nonconvex optimization problem in (P1), is NP-hard
(Aloise et al., 2009). The K-means algorithm, also known
as Lloyd’s algorithm, was ﬁrst proposed by Lloyd (1982) as
a quantization method for pulse-code modulation and has
since been widely applied and extensively studied across
various domains, including image recognition, text mining,
deep learning model weight quantization, healthcare, mar-
1Department of Mathematical Informatics, The University of
Tokyo, Tokyo, Japan 2Huawei Noah’s Ark Lab, Montr´eal, Canada
3Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan.
Correspondence to:
Michael R. Metel <michael.metel@h-
partners.com>.
...
Published as a conference paper at ICLR 2025
TOWARDS A LEARNING THEORY OF REPRESENTATION
ALIGNMENT
Francesco Insulla
Institute of Computational and Mathematical Engineering
Stanford University
Stanford, CA 94305, USA
franinsu@stanford.edu
Shuo Huang
Istituto Italiano di Tecnologia
Genoa, GE 16163, Italy
shuo.huang@iit.it
Lorenzo Rosasco
MaLGa Center, DIBRIS, Universit`a di Genova, Genoa, GE 16146, Italy
CBMM, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Istituto Italiano di Tecnologia, Genoa, GE 16163, Italy
lrosasco@mit.edu
ABSTRACT
It has recently been argued that AI models’ representations are becoming aligned
as their scale and performance increase. Empirical analyses have been designed to
support this idea and conjecture the possible alignment of different representations
toward a shared statistical model of reality. In this paper, we propose a learning-
theoretic perspective to representation alignment. First, we review and connect
different notions of alignment based on metric, probabilistic, and spectral ideas.
Then, we focus on stitching, a particular approach to understanding the interplay
between different representations in the context of a task. Our main contribution
here is to relate the properties of stitching to the kernel alignment of the underlying
representation. Our results can be seen as a first step toward casting representation
alignment as a learning-theoretic problem.
1
INTRODUCTION
In recent years, as AI systems have grown in scale and performance, attention has moved towards
universal models that share architecture across modalities. Examples of such systems include CLIP
(Radford et al., 2021), VinVL (Zhang et al., 2021), FLAVA (Singh et al., 2022), OpenAI’s GPT-4
(OpenAI, 2023), and Google’s Gemini (Google, 2023). These models are trained on diverse datasets
containing both images and text and yield embeddings that can be used for downstream tasks in
either modality or for tasks that require both modalities. The emergen...
Enforcing Idempotency in Neural Networks
Nikolaj Banke Jensen 1 Jamie Vicary 2
Abstract
In this work, we propose a new architecture-
agnostic method for training idempotent neu-
ral networks. An idempotent operator satisfies
f(x) = f(f(x)), meaning it can be applied itera-
tively with no effect beyond the first application.
Some neural networks used in data transformation
tasks, such as image generation and augmenta-
tion, can represent non-linear idempotent projec-
tions. Using methods from perturbation theory we
derive the recurrence relation K′ ←3K2 −2K3
for iteratively projecting a real-valued matrix K
onto the manifold of idempotent matrices. Our
analysis shows that for linear, single-layer MLP
networks this projection 1) has idempotent fixed
points, and 2) is attracting only around idempo-
tent points. We give an extension to non-linear
networks by considering our approach as a sub-
stitution of the gradient for the canonical loss
function, achieving an architecture-agnostic train-
ing scheme. We provide experimental results for
MLP- and CNN-based architectures with signif-
icant improvement in idempotent error over the
canonical gradient-based approach. Finally, we
demonstrate practical applications of the method
as we train generative networks on MNIST and
CelebA successfully using only a simple recon-
struction loss paired with our method.
1. Introduction
Using neural networks as data augmentation tools is becom-
ing more widespread in areas such as signal processing and
generative artificial intelligence. In particular, networks of
the form f : X →X, mapping data within the same space
X, are frequently used in image augmentation (Lu et al.,
1Department of Computer Science, University of Oxford, Ox-
ford, UK. 2Department of Computer Science and Technology,
University of Cambridge, Cambridge, UK. Correspondence to:
Nikolaj Banke Jensen <nikolaj.jensen@cs.ox.ac.uk>, Jamie Vi-
cary <jamie.vicary@cl.cam.ac.uk>.
Proceedings of the 42 nd International Conference ...
Published as a conference paper at ICLR 2025
EXPECTED RETURN SYMMETRIES
Darius Muglich∗
University of Oxford
darius@robots.ox.ac.uk
Johannes Forkel∗
University of Oxford
johannes.forkel@eng.ox.ac.uk
Elise van der Pol
Microsoft Research AI for Science
evanderpol@microsoft.com
Jakob Foerster
University of Oxford
jakob.foerster@eng.ox.ac.uk
ABSTRACT
Symmetry is an important inductive bias that can improve model robustness and
generalization across many deep learning domains. In multi-agent settings, a pri-
ori known symmetries have been shown to address a fundamental coordination
failure mode known as mutually incompatible symmetry breaking; e.g. in a game
where two independent agents can choose to move “left” or “right”, and where a
reward of +1 or −1 is received when the agents choose the same action or differ-
ent actions, respectively. However, the efficient and automatic discovery of envi-
ronment symmetries, in particular for decentralized partially observable Markov
decision processes, remains an open problem. Furthermore, environmental sym-
metry breaking constitutes only one type of coordination failure, which motivates
the search for a more accessible and broader symmetry class. In this paper, we
introduce such a broader group of previously unexplored symmetries, which we
call expected return symmetries, which contains environment symmetries as a sub-
group. We show that agents trained to be compatible under the group of expected
return symmetries achieve better zero-shot coordination results than those using
environment symmetries. As an additional benefit, our method makes minimal a
priori assumptions about the structure of their environment and does not require
access to ground truth symmetries.
1
INTRODUCTION
Incorporating the symmetries of an underlying problem into models has had demonstrable success in
improving generalization and accuracy across many different machine learning domains (Bronstein
et al., 2021; Krizhevsky et al., 2017; Cohen et al., 201...
Simple Path Structural Encoding for Graph Transformers
Louis Airale 1 Antonio Longa 1 Mattia Rigon 1 Andrea Passerini 1 Roberto Passerone 1
Abstract
Graph transformers extend global self-attention
to graph-structured data, achieving notable suc-
cess in graph learning. Recently, Relative Ran-
dom Walk Probabilities (RRWP) has been found
to further enhance their predictive power by en-
coding both structural and positional information
into the edge representation. However, RRWP
cannot always distinguish between edges that be-
long to different local graph patterns, which re-
duces its ability to capture the full structural com-
plexity of graphs. This work introduces Simple
Path Structural Encoding (SPSE), a novel method
that utilizes simple path counts for edge encod-
ing. We show theoretically and experimentally
that SPSE overcomes the limitations of RRWP,
providing a richer representation of graph struc-
tures, particularly for capturing local cyclic pat-
terns. To make SPSE computationally tractable,
we propose an efficient approximate algorithm
for simple path counting. SPSE demonstrates sig-
nificant performance improvements over RRWP
on various benchmarks, including molecular and
long-range graph datasets, achieving statistically
significant gains in discriminative tasks. These
results pose SPSE as a powerful edge encoding
alternative for enhancing the expressivity of graph
transformers.
1. Introduction
Graphs are pervasive across diverse domains, representing
complex relationships in areas such as social networks (Otte
& Rousseau, 2002), molecular structures (Quinn et al.,
2017), and citation graphs (Radicchi et al., 2011). Recent ad-
vances in graph neural networks (GNNs) have driven signif-
icant progress in learning from graph-structured data (Kipf
& Welling, 2017; Veliˇckovi´c et al., 2018; Bodnar et al.,
1University of Trento, Trento, Italy.
Correspondence
to: Louis Airale <louis.airale@unitn.it>, Roberto Passerone
<roberto.passerone@unitn.it>.
Proceedin...
Periodic Skill Discovery
Jonghae Park1 Daesol Cho2 Jusuk Lee1 Dongseok Shim1 Inkyu Jang1 H. Jin Kim1
1Seoul National University
2Georgia Institute of Technology
bdfire1234@snu.ac.kr
Abstract
Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse
behaviors without relying on external rewards. However, current methods often
overlook the periodic nature of learned skills, focusing instead on increasing the
mutual dependence between states and skills or maximizing the distance traveled
in latent space. Considering that many robotic tasks—particularly those involving
locomotion—require periodic behaviors across varying timescales, the ability
to discover diverse periodic skills is essential. Motivated by this, we propose
Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors
in an unsupervised manner. The key idea of PSD is to train an encoder that maps
states to a circular latent space, thereby naturally encoding periodicity in the latent
representation. By capturing temporal distance, PSD can effectively learn skills
with diverse periods in complex robotic tasks, even with pixel-based observations.
We further show that these learned skills achieve high performance on downstream
tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery
method offers more diverse behaviors, thus broadening the agent’s repertoire.
Our code and demos are available at https://jonghaepark.github.io/psd
1
Introduction
A fundamental observation in nature is that nearly all forms of locomotion are inherently periodic.
Rhythmic gaits of quadrupeds, the oscillatory motions of fish, and even human walking patterns
share a distinct periodic structure, which can be flexibly modulated across multiple timescales
[39, 28, 31]. This inherent periodicity not only enables energy-efficient movement [39, 82] but also
provides adaptability under varying conditions [28, 31, 82]. Motivated by this understanding, robotics
research has le...
SkipGPT: Each Token is One of a Kind
Anhao Zhao 1 2 Fanghua Ye 3 Yingqi Fan 1 Junlong Tong 1 4 Jing Xiong 5 Zhiwei Fei 6 Hui Su 7 Xiaoyu Shen 1
Abstract
Large language models (LLMs) achieve remark-
able performance across tasks but incur substan-
tial computational costs due to their deep, multi-
layered architectures. Layer pruning has emerged
as a strategy to alleviate these inefficiencies, but
conventional static pruning methods overlook two
critical dynamics inherent to LLM inference: (1)
horizontal dynamics, where token-level hetero-
geneity demands context-aware pruning decisions,
and (2) vertical dynamics, where the distinct func-
tional roles of MLP and self-attention layers ne-
cessitate component-specific pruning policies. We
introduce SkipGPT, a dynamic layer pruning
framework designed to optimize computational
resource allocation through two core innovations:
(1) global token-aware routing to prioritize crit-
ical tokens and (2) decoupled pruning policies
for MLP and self-attention components. To miti-
gate training instability, we propose a two-stage
optimization paradigm: first, a disentangled train-
ing phase that learns routing strategies via soft
parameterization to avoid premature pruning de-
cisions, followed by parameter-efficient LoRA
fine-tuning to restore performance impacted by
layer removal. Extensive experiments demon-
strate that SkipGPT reduces over 40% model pa-
rameters while matching or exceeding the perfor-
mance of the original dense model across bench-
marks. By harmonizing dynamic efficiency with
preserved expressivity, SkipGPT advances the
practical deployment of scalable, resource-aware
LLMs. Our code is publicly available at: https:
//github.com/EIT-NLP/SkipGPT.
1Ningbo Key Laboratory of Spatial Intelligence and Dig-
ital Derivative, Institute of Digital Twin, Eastern Institute of
Technology, Ningbo 2Southwest Jiaotong University 3Tencent
Inc.
4Shanghai Jiao Tong University 5The University of Hong
Kong
6Nanjing University
7Meitu...
AdaptiveStep: Automatically Dividing Reasoning Step through Model
Confidence
Yuliang Liu * 1 2 Junjie Lu * 3 Chaofeng Qu 4 Zhaoling Chen 1 Zefan Cai 5 Jason Klein Liu 4 Chonghan Liu 4
Yunhui Xia 4 Li Zhao 6 Jiang Bian 6 Chuheng Zhang 6 Wei Shen 4 Zhouhan Lin 2 7
Abstract
Current approaches for training Process Reward
Models (PRMs) often involve decomposing re-
sponses into multiple reasoning steps using rule-
based techniques, such as using predefined place-
holder tokens or setting the reasoning step’s
length to a fixed size. These approaches overlook
the fact that certain words don’t usually indicate
true decision points. To address this, we propose
AdaptiveStep, a method that divides reasoning
steps based on the model’s confidence in predict-
ing the next word, offering more information on
decision-making at each step, improving down-
stream tasks like reward model training. More-
over, our method requires no manual annotation.
Experiments with AdaptiveStep-trained PRMs in
mathematical reasoning and code generation show
that the outcome PRM achieves state-of-the-art
Best-of-N performance, surpassing greedy search
strategy with token-level value-guided decoding,
while also reducing construction costs by over
30% compared to existing open-source PRMs.
We also provide a thorough analysis and case
study on its performance, transferability, and gen-
eralization capabilities. We provide our code on
https://github.com/Lux0926/ASPRM.
1. Introduction
Large language models (LLMs) have demonstrated excep-
tional performance across various tasks. However, even
most advanced LLMs struggle to generate correct solutions
when facing complex reasoning problems, such as mathe-
*Equal contribution 1Nanjing University 2Shanghai Innovation
Institute
3University of Technology Sydney
4Independent
5UW-Madison 6MSRA 7Shanghai Jiaotong University. Corre-
spondence to: Chuheng Zhang <zhangchuheng123@live.com>,
Wei
Shen
<shenwei0917@126.com>,
Zhouhan
Lin
<lin.zhouhan@gmail.com>.
Proceedings...
Sample complexity of data-driven tuning of model
hyperparameters in neural networks with structured
parameter-dependent dual function
Maria-Florina Balcan
Carnegie Mellon University
ninamf@cs.cmu.edu
Anh Tuan Nguyen
Carnegie Mellon University
atnguyen@cs.cmu.edu
Dravyansh Sharma
Toyota Technological Institute at Chicago
dravy@ttic.edu
Abstract
Modern machine learning algorithms, especially deep learning-based techniques,
typically involve careful hyperparameter tuning to achieve the best performance.
Despite the surge of intense interest in practical techniques like Bayesian opti-
mization and random search-based approaches to automating this laborious and
compute-intensive task, the fundamental learning-theoretic complexity of tuning
hyperparameters for deep neural networks is poorly understood. Inspired by this
glaring gap, we initiate the formal study of hyperparameter tuning complexity in
deep learning through a recently introduced data-driven setting. We assume that we
have a series of learning tasks, and we have to tune hyperparameters to do well on
average over the distribution of tasks. A major difficulty is that the utility function
as a function of the hyperparameter is very volatile and furthermore, it is given
implicitly by an optimization problem over the model parameters. To tackle this
challenge, we introduce a new technique to characterize the discontinuities and
oscillations of the utility function on any fixed problem instance as we vary the
hyperparameter; our analysis relies on subtle concepts including tools from alge-
braic geometry, differential geometry and constrained optimization. We use this to
show that the learning-theoretic complexity of the corresponding family of utility
functions is bounded. We instantiate our results and provide sample complexity
bounds for concrete applications—tuning a hyperparameter that interpolates neural
activation functions and setting the kernel parameter in graph neural networks.
1
Introduction
Developing d...
The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language
Models via Visual Information Steering
Zhuowei Li 1 Haizhou Shi 1 Yunhe Gao 1 2 Di Liu 1 Zhenting Wang 1 Yuxiao Chen 1 Ting Liu 3 Long Zhao 3
Hao Wang 1 Dimitris N. Metaxas 1
Abstract
Large Vision-Language Models (LVLMs) can rea-
son effectively over both textual and visual in-
puts, but they tend to hallucinate syntactically
coherent yet visually ungrounded contents. In
this paper, we investigate the internal dynamics
of hallucination by examining the tokens logits
ranking throughout the generation process, re-
vealing three key patterns in how LVLMs pro-
cess information: (1) gradual visual informa-
tion loss – visually grounded tokens gradually
become less favored throughout generation, and
(2) early excitation – semantically meaningful to-
kens achieve peak activation in the layers earlier
than the final layer. (3) hidden genuine informa-
tion – visually grounded tokens though not be-
ing eventually decoded still retain relatively high
rankings at inference. Based on these insights,
we propose VISTA (Visual Information Steering
with Token-logit Augmentation), a training-free
inference-time intervention framework that re-
duces hallucination while promoting genuine in-
formation. VISTA works by combining two com-
plementary approaches: reinforcing visual infor-
mation in activation space and leveraging early
layer activations to promote semantically mean-
ingful decoding. Compared to existing methods,
VISTA requires no external supervision and is
applicable to various decoding strategies. Ex-
tensive experiments show that VISTA on aver-
age reduces hallucination by about 40% on eval-
uated open-ended generation task, and it con-
sistently outperforms existing methods on four
benchmarks across four architectures under three
decoding strategies. Code is available at https:
//github.com/LzVv123456/VISTA.
*Equal contribution
1Rutgers University 2Stanford Univer-
sity
3Google DeepMind.
Correspo...