AIS-Bench/AIS-Bench huggingface dataset:
Stochastically Dominant Peer Prediction
Yichi Zhang
DIMACS, Rutgers University
yz1636@dimacs.rutgers.edu
Shengwei Xu
University of Michigan, Ann Arbor
shengwei@umich.edu
David Pennock
DIMACS, Rutgers University
dpennock@dimacs.rutgers.edu
Grant Schoenebeck
University of Michigan, Ann Arbor
schoeneb@umich.edu
Abstract
Eliciting reliable human feedback is essential for many machine learning tasks,
such as learning from noisy labels and aligning AI systems with human prefer-
ences. Peer prediction mechanisms incentivize truthful reporting without ground
truth verification by scoring agents based on correlations with peers. Traditional
mechanisms, which ensure that truth-telling maximizes the expected scores in
equilibrium, can elicit honest information while assuming agents’ utilities are
linear functions of their scores. However, in practice, non-linear payment rules
are usually preferred, or agents’ utilities are inherently non-linear.
We propose stochastically dominant truthfulness (SD-truthfulness) as a stronger
guarantee: the score distribution of truth-telling stochastically dominates all other
strategies, incentivizing truthful reporting for a wide range of monotone utility func-
tions. Our first observation is that no existing peer prediction mechanism naturally
satisfies this criterion without strong assumptions. A simple solution—rounding
scores into binary lotteries—can enforce SD-truthfulness, but often degrades sensi-
tivity, a key property related to fairness and statistical efficiency. We demonstrate
how a more careful application of rounding can better preserve sensitivity. Further-
more, we introduce a new enforced agreement (EA) mechanism that is theoretically
guaranteed to be SD-truthful in binary-signal settings and, under mild assump-
tions, empirically achieves the highest sensitivity among all known SD-truthful
mechanisms.
1
Introduction
Information elicitation studies how to design mechanisms that incentivize honest and high-effort
human feedbac...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000057. Conference bucket: NeurIPS.
Private Evolution Converges
Tomas Gonzalez
Carnegie Mellon University
tcgonzal@andrew.cmu.edu
Giulia Fanti
Carnegie Mellon University
gfanti@andrew.cmu.edu
Aaditya Ramdas
Carnegie Mellon University
aramdas@cs.cmu.edu
Abstract
Private Evolution (PE) is a promising training-free method for differentially private
(DP) synthetic data generation. While it achieves strong performance in some
domains (e.g., images and text), its behavior in others (e.g., tabular data) is less
consistent. To date, the only theoretical analysis of the convergence of PE depends
on unrealistic assumptions about both the algorithm’s behavior and the structure
of the sensitive dataset. In this work, we develop a new theoretical framework
to understand PE’s practical behavior and identify sufficient conditions for its
convergence. For d-dimensional sensitive datasets with n data points from a
convex and compact domain, we prove that under the right hyperparameter settings
and given access to the Gaussian variation API proposed in [33], PE produces an
(ε, δ)-DP synthetic dataset with expected 1-Wasserstein distance ˜O(d(nε)−1/d)
from the original; this establishes worst-case convergence of the algorithm as
n →∞. Our analysis extends to general Banach spaces as well. We also connect
PE to the Private Signed Measure Mechanism, a method for DP synthetic data
generation that has thus far not seen much practical adoption. We demonstrate the
practical relevance of our theoretical findings in experiments.
1
Introduction
Many modern machine learning applications rely on user data, making data privacy protection a
central concern. In this regard, differential privacy (DP) [17, 19] has become a de facto standard for
safeguarding sensitive information of individuals. Given a private dataset, many problems—including
regression [40, 12], deep learning [1], and stochastic optimization [6, 2]—can be performed in a DP
manner. While it is possible to design a different DP algorithm for each specific task, a reason...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000058. Conference bucket: NeurIPS.
Uni-MuMER: Unified Multi-Task Fine-Tuning of
Vision-Language Model for Handwritten
Mathematical Expression Recognition
Yu Li∗
Jin Jiang∗
Jianhua Zhu
Shuai Peng
Baole Wei
Yuxuan Zhou
Liangcai GaoB
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
liyu@stu.pku.edu.cn
jiangjin@stu.pku.edu.cn
zhujianhuapku@pku.edu.cn
glc@pku.edu.cn
Abstract
Handwritten Mathematical Expression Recognition (HMER) remains a persistent
challenge in Optical Character Recognition (OCR) due to the inherent freedom
of symbol layouts and variability in handwriting styles. Prior methods have faced
performance bottlenecks by proposing isolated architectural modifications, making
them difficult to integrate coherently into a unified framework. Meanwhile, recent
advances in pretrained vision-language models (VLMs) have demonstrated strong
cross-task generalization, offering a promising foundation for developing unified
solutions. In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM
for the HMER task without modifying its architecture, effectively injecting domain-
specific knowledge into a generalist framework. Our method integrates three data-
driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial rea-
soning, Error-Driven Learning (EDL) for reducing confusion among visually simi-
lar characters, and Symbol Counting (SC) for improving recognition consistency
in long expressions. Experiments on the CROHME and HME100K datasets show
that Uni-MuMER achieves super state-of-the-art performance, outperforming the
best lightweight specialized model SSAN by 16.31% and the top-performing VLM
Gemini2.5-flash by 24.42% under zero-shot setting. Our datasets, models, and
code are open-sourced at: https://github.com/BFlameSwift/Uni-MuMER
1
Introduction
38.64
37.75
37.45
62.58
62.51
65.30 
72.62
64.26
69.14 
82.05
77.94
79.23
30
40
50
60
70
80
90
100
CROHME 2014
CROHME 2016
CROHME 2019
ExpRate(%)
SSAN(2025)
Qwen2.5VL Zero-Shot
Qwen2.5VL SFT(Vanilla)
Un...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000064. Conference bucket: NeurIPS.
Imagined Autocurricula
Ahmet H. Güzel∗
University College London AI Centre
Matthew T. Jackson
University of Oxford
Jarek L. Liesen
University of Oxford
Tim Rocktäschel
University College London AI Centre
Jakob N. Foerster
University of Oxford
Ilija Bogunovic
University College London AI Centre
Jack Parker-Holder
University College London AI Centre
Abstract
Training agents to act in embodied environments typically requires vast training
data or access to accurate simulation, neither of which exists for many cases in
the real world. Instead, world models are emerging as an alternative–leveraging
offline, passively collected data, they make it possible to generate diverse worlds
for training agents in simulation. In this work, we harness world models to gen-
erate “imagined” environments to train robust agents capable of generalizing to
novel task variations. One of the challenges in doing this is ensuring the agent
trains on useful generated data. We thus propose a novel approach IMAC (Imag-
ined Autocurricula) leveraging Unsupervised Environment Design (UED), which
induces an automatic curriculum over generated worlds. In a series of challeng-
ing, procedurally generated environments, we show it is possible to achieve strong
transfer performance on held-out environments having trained only inside a world
model learned from a narrower dataset. We believe this opens the path to utilizing
larger-scale, foundation world models for generally capable agents.
1
Introduction
Despite significant advancements in AI, we remain far from generally capable agents [1]. In the
2010s, progress towards this goal was driven by breakthroughs in deep reinforcement learning (RL,
[2]) where agents such as AlphaGo [3] showed it was possible to discover new knowledge beyond
human capabilities, like the famous move37. However, scaling RL in simulation to produce a more
general intelligence remains bottlenecked by the lack of sufficiently diverse and real-world simula-
tors [4, 5]. By contrast...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000066. Conference bucket: NeurIPS.
Brain network science modelling of sparse neural
networks enables Transformers and LLMs to perform
as fully connected
Yingtao Zhang1,2, Diego Cerretti1,2, Jialin Zhao1,2, Ziheng Liao1,2, Wenjing Wu1,2
Umberto Michieli4,5 & Carlo Vittorio Cannistraci1,2,3∗
1Center for Complex Network Intelligence (CCNI)†
2Dept. of Computer Science & Technology, 3School of Biomedical Engineering, Tsinghua University
4University of Padova, 5Canva Research
Abstract
This study aims to enlarge our current knowledge on the application of brain-
inspired network science principles for training artificial neural networks (ANNs)
with sparse connectivity. Dynamic sparse training (DST) emulates the synaptic
turnover of real brain networks, reducing the computational demands of train-
ing and inference in ANNs. However, existing DST methods face difficulties in
maintaining peak performance at high connectivity sparsity levels. The Cannistraci-
Hebb training (CHT) is a brain-inspired method that is used in DST for growing
synaptic connectivity in sparse neural networks. CHT leverages a gradient-free,
topology-driven link regrowth mechanism, which has been shown to achieve ultra-
sparse (1% connectivity or lower) advantage across various tasks compared to
fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time
complexity is O(N · d3)- N node network size, d node degree - hence it can be
efficiently applied only to ultra-sparse networks. (ii) it rigidly selects top link
prediction scores, which is inappropriate for the early training epochs, when the
network topology presents many unreliable connections. Here, we design the first
brain-inspired network model - termed bipartite receptive field (BRF) - to initialize
the connectivity of sparse artificial neural networks. Then, we propose a matrix
multiplication GPU-friendly approximation of the CH link predictor, which reduces
the computational complexity to O(N 3), enabling a fast implementation of link
prediction in large-scale m...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000068. Conference bucket: NeurIPS.
Published as a conference paper at ICLR 2025
LOCALITY ALIGNMENT IMPROVES VISION-LANGUAGE
MODELS
Ian Covert, Tony Sun, James Zou∗, Tatsunori Hashimoto∗
Stanford University
{icovert, suntony, jamesz, thashim}@stanford.edu
ABSTRACT
Vision language models (VLMs) have seen growing adoption in recent years,
but many still struggle with basic spatial reasoning errors. We hypothesize that
this is due to VLMs adopting pre-trained vision backbones, specifically vision
transformers (ViTs) trained with image-level supervision and minimal inductive
biases. Such models may fail to encode the class contents at each position in
the image, and our goal is to resolve this with a vision backbone that effectively
captures both local and global image semantics. Our main insight is that we do
not require new supervision to learn this capability – pre-trained models contain
significant knowledge of local semantics that we can extract and use for scalable
self-supervision. We propose a new efficient post-training stage for ViTs called
locality alignment and a novel fine-tuning procedure called MaskEmbed that uses
a masked reconstruction loss to learn semantic contributions for each image patch.
We first evaluate locality alignment with a vision-only benchmark, finding that it
improves a model’s performance at patch-level semantic segmentation, especially
for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We
then train a series of VLMs with and without locality alignment, and show that
locality-aligned backbones improve performance across a range of benchmarks,
particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref,
TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local
semantic extraction via a locality alignment stage, and that this procedure benefits
VLM training recipes that use off-the-shelf vision backbones.
1
INTRODUCTION
Auto-regressive VLMs are an exciting new type of model that emerged in the last couple ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000009. Conference bucket: ICLR.
Published as a conference paper at ICLR 2025
ROBUST SIMULATION-BASED INFERENCE
UNDER MISSING DATA VIA NEURAL PROCESSES
Yogesh Verma, Ayush Bharti
Department of Computer Science, Aalto University
{yogesh.verma, ayush.bharti}@aalto.fi
Vikas Garg
YaiYai Ltd and Aalto University
vgarg@csail.mit.edu
ABSTRACT
Simulation-based inference (SBI) methods typically require fully observed data to
infer parameters of models with intractable likelihood functions. However, datasets
often contain missing values due to incomplete observations, data corruptions
(common in astrophysics), or instrument limitations (e.g., in high-energy physics
applications). In such scenarios, missing data must be imputed before applying any
SBI method. We formalize the problem of missing data in SBI and demonstrate that
naive imputation methods can introduce bias in the estimation of SBI posterior. We
also introduce a novel amortized method that addresses this issue by jointly learning
the imputation model and the inference network within a neural posterior estimation
(NPE) framework. Extensive empirical results on SBI benchmarks show that our
approach provides robust inference outcomes compared to standard baselines
for varying levels of missing data. Moreover, we demonstrate the merits of our
imputation model on two real-world bioactivity datasets (Adrenergic and Kinase
assays). Code is available at https://github.com/Aalto-QuML/RISE.
1
INTRODUCTION
Mechanistic models for studying complex physical or biological phenomena have become indispens-
able tools in research fields as diverse as genetics (Riesselman et al., 2018), epidemiology (Kypraios
et al., 2017), gravitational wave astronomy (Dax et al., 2021), and radio propagation (Bharti et al.,
2022a). However, fitting such models to observational data can be challenging due to the intractabil-
ity of their likelihood functions, which renders standard Bayesian inference methods inapplicable.
Simulation-based inference (SBI) methods (Cranmer et al., 2...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000010. Conference bucket: ICLR.
Published as a conference paper at ICLR 2025
OPTIONZERO: PLANNING WITH LEARNED OPTIONS
Po-Wei Huang1,2, Pei-Chiun Peng1,2, Hung Guei1, Ti-Rong Wu1†
1Institute of Information Science, Academia Sinica, Taiwan
2Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
ABSTRACT
Planning with options – a sequence of primitive actions – has been shown ef-
fective in reinforcement learning within complex environments. Previous studies
have focused on planning with predefined options or learned options through ex-
pert demonstration data. Inspired by MuZero, which learns superhuman heuristics
without any human knowledge, we propose a novel approach, named OptionZero.
OptionZero incorporates an option network into MuZero, providing autonomous
discovery of options through self-play games. Furthermore, we modify the dy-
namics network to provide environment transitions when using options, allowing
searching deeper under the same simulation constraints. Empirical experiments
conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero,
achieving a 131.58% improvement in mean human-normalized score. Our behav-
ior analysis shows that OptionZero not only learns options but also acquires strate-
gic skills tailored to different game characteristics. Our findings show promising
directions for discovering and using options in planning. Our code is available at
https://rlg.iis.sinica.edu.tw/papers/optionzero.
1
INTRODUCTION
Reinforcement learning is a decision-making process in which an agent interacts with environments
by selecting actions at each step to maximize long-term rewards. Actions, commonly referred to as
primitive action, advance the state by one step each. While this granularity allows precise control
at each time step, it can lead to inefficiencies in scenarios where predictable sequences of actions
are beneficial. For example, in a maze navigation task, it is more efficient to choose a sequence
of actions – such as following a strai...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000018. Conference bucket: ICLR.
Published as a conference paper at ICLR 2025
CORESET SPECTRAL CLUSTERING
Ben Jourdan1, Gregory Schwartzman2, Peter Macgregor3, and He Sun1
1University of Edinburgh, UK
2Japan Advanced Institute of Science and Technology (JAIST), Japan
3University of St Andrews, UK
ben.jourdan@ed.ac.uk, greg@jaist.ac.jp, prm4@st-andrews.ac.uk,
h.sun@ed.ac.uk
ABSTRACT
Coresets have become an invaluable tool for solving k-means and kernel k-means clustering
problems on large datasets with small numbers of clusters. On the other hand, spectral clustering
works well on sparse graphs and has recently been extended to scale efficiently to large numbers
of clusters. We exploit the connection between kernel k-means and the normalised cut problem
to combine the benefits of both. Our main result is a coreset spectral clustering algorithm
for graphs that clusters a coreset graph to infer a good labelling of the original graph. We
prove that an α-approximation for the normalised cut problem on the coreset graph is an O(α)-
approximation on the original. We also improve the running time of the state-of-the-art coreset
algorithm for kernel k-means on sparse kernels, from ˜O(nk) to ˜O(n · min{k, davg}), where
davg is the average number of non-zero entries in each row of the n × n kernel matrix. Our
experiments confirm our coreset algorithm is asymptotically faster on large real-world graphs
with many clusters, and show that our clustering algorithm overcomes the main challenge faced
by coreset kernel k-means on sparse kernels which is getting stuck in local optima.
1
INTRODUCTION
Kernel k-means and spectral clustering are two popular algorithms which are capable of learning
non-linear decision boundaries between clusters. For this reason, they have been applied to many
practical problems in machine learning, including in the fields of medical research and network
science (Gönen & Margolin, 2014; Kuo et al., 2014; White & Smyth, 2005).
Given a data set X, both kernel k-means and spectral clustering ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000023. Conference bucket: ICLR.
Published as a conference paper at ICLR 2025
EMBEDLLM:
LEARNING
COMPACT
REPRESENTA-
TIONS OF LARGE LANGUAGE MODELS
Richard Zhuang∗Tianhao Wu∗
Zhaojin Wen
Andrew Li
Jiantao Jiao
Kannan Ramchandran
University of California, Berkeley
ABSTRACT
With hundreds of thousands of language models available on Huggingface to-
day, efficiently evaluating and utilizing these models across various downstream
tasks has become increasingly critical. Many existing methods repeatedly learn
task-specific representations of Large Language Models (LLMs), which leads to
inefficiencies in both time and computational resources. To address this, we pro-
pose EmbedLLM, a framework designed to learn compact vector representations
of LLMs that facilitate downstream applications involving many models, such as
model routing. We introduce an encoder-decoder approach for learning such em-
beddings, along with a systematic framework to evaluate their effectiveness. Em-
pirical results show that EmbedLLM outperforms prior methods in model routing
both in accuracy and latency. Additionally, we demonstrate that our method can
forecast a model’s performance on multiple benchmarks, without incurring ad-
ditional inference cost. Extensive probing experiments validate that the learned
embeddings capture key model characteristics, e.g. whether the model is special-
ized for coding tasks, even without being explicitly trained on them. We open
source our dataset, code and embedder to facilitate further research and applica-
tion: https://github.com/richardzhuang0412/EmbedLLM.
1
INTRODUCTION
Recent breakthroughs in Large Language Models (LLMs) (Vaswani et al., 2023) have led to the
creation of a vast array of models, each tailored for different use cases. These models, ranging from
small, specialized models to large, general-purpose systems (Hao et al., 2022), differ significantly
in their architecture, size, training data, and performance characteristics. For example, while some
models excel as conversational ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000029. Conference bucket: ICLR.
Published as a conference paper at ICLR 2025
CAKE: CASCADING
AND ADAPTIVE KV CACHE
EVICTION WITH LAYER PREFERENCES
Ziran Qin1,2∗, Yuchen Cao2, Mingbao Lin3†, Wen Hu2, Shixuan Fan2, Ke Cheng2,
Weiyao Lin1†, Jianguo Li2†
1Shanghai Jiao Tong University, 2Ant Group, 3Independent Researcher
ABSTRACT
Large language models (LLMs) excel at processing long sequences, boosting de-
mand for key-value (KV) caching. While recent efforts to evict KV cache have al-
leviated the inference burden, they often fail to allocate resources rationally across
layers with different attention patterns. In this paper, we introduce Cascading and
Adaptive KV cache Eviction (CAKE), a novel approach that frames KV cache
eviction as a “cake-slicing problem.” CAKE assesses layer-specific preferences
by considering attention dynamics in both spatial and temporal dimensions, allo-
cates rational cache size for layers accordingly, and manages memory constraints
in a cascading manner. This approach enables a global view of cache alloca-
tion, adaptively distributing resources across diverse attention mechanisms while
maintaining memory budgets. CAKE also employs a new eviction indicator that
considers the shifting importance of tokens over time, addressing limitations in
existing methods that overlook temporal dynamics. Comprehensive experiments
on LongBench and NeedleBench show that CAKE maintains model performance
with only 3.2% of the KV cache and consistently outperforms current baselines
across various models and memory constraints, particularly in low-memory set-
tings. Additionally, CAKE achieves over 10× speedup in decoding latency com-
pared to full cache when processing contexts of 128K tokens with FlashAttention-
2. Our code is available at https://github.com/antgroup/cakekv.
1
INTRODUCTION
Large language models (LLMs) (Zhao et al., 2023; Achiam et al., 2023; Dubey et al., 2024; An-
thropic, 2024; AI, 2024) have enhanced their long-text processing capabilities, improving perfor-
mance in mul...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000037. Conference bucket: ICLR.