Yancy567/AIS-Bench huggingface dataset:
Conference: ICML 2025

Sparse Autoencoders, Again?
Yin Lu 1 Xuening Zhu 1 Tong He 2 David Wipf 2
Abstract
Is there really much more to say about sparse au-
toencoders (SAEs)? Autoencoders in general, and
SAEs in particular, represent deep architectures
that are capable of modeling low-dimensional la-
tent structure in data. Such structure could re-
flect, among other things, correlation patterns
in large language model activations, or complex
natural image manifolds. And yet despite the
wide-ranging applicability, there have been rel-
atively few changes to SAEs beyond the origi-
nal recipe from decades ago, namely, standard
deep encoder/decoder layers trained with a classi-
cal/deterministic sparse regularizer applied within
the latent space. One possible exception is the
variational autoencoder (VAE), which adopts a
stochastic encoder module capable of producing
sparse representations when applied to manifold
data. In this work we formalize underappreciated
weaknesses with both canonical SAEs, as well as
analogous VAEs applied to similar tasks, and pro-
pose a hybrid alternative model that circumvents
these prior limitations. In terms of theoretical sup-
port, we prove that global minima of our proposed
model recover certain forms of structured data
spread across a union of manifolds. Meanwhile,
empirical evaluations on synthetic and real-world
datasets substantiate the efficacy of our approach
in accurately estimating underlying manifold di-
mensions and producing sparser latent representa-
tions without compromising reconstruction error.
In general, we are able to exceed the performance
of equivalent-capacity SAEs and VAEs, as well as
recent diffusion models where applicable, within
domains such as images and language model acti-
vation patterns. A link to the code is here.
1School of Data Science, Fudan University 2Amazon Web Services.
Correspondence
to:
Yin
Lu
<yinlu23@m.fudan.edu.cn>,
Xuening
Zhu
<xueningzhu@fudan.edu.cn>,
Tong
He
<htong@amazon.com>, David Wipf <davidwipf@gmail.com...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000002. Conference bucket: ICML.
Conference: ICML 2025

Function-Space Learning Rates
Edward Milsom 1 Ben Anson 1 Laurence Aitchison 1
Abstract
We consider layerwise function-space learning
rates, which measure the magnitude of the change
in a neural network’s output function in response
to an update to a parameter tensor. This con-
trasts with traditional learning rates, which de-
scribe the magnitude of changes in parameter
space. We develop efficient methods to measure
and set function-space learning rates in arbitrary
neural networks, requiring only minimal compu-
tational overhead through a few additional back-
ward passes that can be performed at the start
of, or periodically during, training. We demon-
strate two key applications: (1) analysing the dy-
namics of standard neural network optimisers in
function space, rather than parameter space, and
(2) introducing FLeRM (Function-space Learn-
ing Rate Matching), a novel approach to hyper-
parameter transfer across model scales. FLeRM
records function-space learning rates while train-
ing a small, cheap base model, then automatically
adjusts parameter-space layerwise learning rates
when training larger models to maintain consistent
function-space updates. FLeRM gives hyperpa-
rameter transfer across model width, depth, initial-
isation scale, and LoRA rank in various architec-
tures including MLPs with residual connections
and transformers with different layer normalisa-
tion schemes.
1. Introduction
The fundamental purpose of neural network training is to
learn a function that maps inputs to desired outputs. How-
ever, we typically understand optimisation methods as act-
ing in parameter space, e.g. traditional learning rates tell us
how much the parameters change during each step rather
than the functional impact of those changes. This raises
1University of Bristol.
Correspondence to: Edward Mil-
som <edward.milsom@bristol.ac.uk>, Laurence Aitchison <lau-
rence.aitchison@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000003. Conference bucket: ICML.
Conference: ICML 2025

Clients Collaborate: Flexible Differentially Private Federated Learning with
Guaranteed Improvement of Utility-Privacy Trade-off
Yuecheng Li 1 Lele Fu 1 Tong Wang 2 Jian Lou 1 Bin Chen 3 Lei Yang 1 Jian Shen 4 Zibin Zheng 1
Chuan Chen 1
Abstract
To defend against privacy leakage of user data,
differential privacy is widely used in federated
learning, but it is not free.
The addition of
noise randomly disrupts the semantic integrity
of the model and this disturbance accumulates
with increased communication rounds. In this
paper, we introduce a novel federated learning
framework with rigorous privacy guarantees,
named FedCEO, designed to strike a trade-off
between model utility and user privacy by
letting clients “Collaborate with Each Other”.
Specifically, we perform efficient tensor low-rank
proximal optimization on stacked local model
parameters at the server, demonstrating its
capability to flexibly truncate high-frequency
components in spectral space. This capability
implies that our FedCEO can effectively recover
the disrupted semantic information by smoothing
the global semantic space for different privacy
settings
and
continuous
training
processes.
Moreover, we improve the SOTA utility-privacy
trade-off bound by order of
√
d, where d is the
input dimension. We illustrate our theoretical
results
with
experiments
on
representative
datasets and observe significant performance
improvements and strict privacy guarantees under
different privacy settings. The code is available at
https://github.com/6lyc/FedCEO_
Collaborate-with-Each-Other.
1Sun Yat-sen University, Guangzhou, China 2Texas A&M Uni-
versity, Texas, USA 3Harbin Institute of Technology, Shenzhen,
China 4Zhejiang Sci-Tech University, China. Correspondence to:
Chuan Chen <chenchuan@mail.sysu.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
Federated learning (FL) (McMahan et al., 2017), a priva...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000004. Conference bucket: ICML.
Conference: ICML 2025

ML2-GCL: Manifold Learning Inspired Lightweight Graph Contrastive
Learning
Jianqing Liang 1 Zhiqiang Li 1 Xinkai Wei 1 Yuan Liu 1 Zhiqiang Wang 1
Abstract
Graph contrastive learning has attracted great
interest as a dominant and promising self-
supervised representation learning approach in
recent years. While existing works follow the ba-
sic principle of pulling positive pairs closer and
pushing negative pairs far away, they still suffer
from several critical problems, such as the under-
lying semantic disturbance brought by augmen-
tation strategies, the failure of GCN in capturing
long-range dependence, rigidness and inefficiency
of node sampling techniques. To address these
issues, we propose Manifold Learning Inspired
Lightweight Graph Contrastive Learning (ML2-
GCL), which inherits the merits of both manifold
learning and GCN. ML2-GCL avoids the poten-
tial risks of semantic disturbance with only one
single view. It achieves global nonlinear struc-
ture recovery from locally linear fits, which can
make up for the defects of GCN. The most amaz-
ing advantage is about the lightweight due to its
closed-form solution of positive pairs weights and
removal of pairwise distances calculation. The-
oretical analysis proves the existence of the op-
timal closed-form solution. Extensive empirical
results on various benchmarks and evaluation pro-
tocols demonstrate effectiveness and lightweight
of ML2-GCL. We release the code at https:
//github.com/a-hou/ML2-GCL.
1. Introduction
Graph representation learning, which aims to learn appro-
priate low-dimensional representations for graph structured
*Equal contribution 1Key Laboratory of Computational Intel-
ligence and Chinese Information Processing of Ministry of Edu-
cation, School of Computer and Information Technology, Shanxi
University, Taiyuan 030006, Shanxi, China. Correspondence to:
Jianqing Liang <liangjq@sxu.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267,...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000005. Conference bucket: ICML.
Conference: ICML 2025

Stochastic Encodings for Active Feature Acquisition
Alexander Norcliffe 1 Changhee Lee 2 Fergus Imrie 3 Mihaela van der Schaar 4 Pietro Liò 1
Abstract
Active Feature Acquisition is an instance-wise,
sequential decision making problem. The aim is
to dynamically select which feature to measure
based on current observations, independently for
each test instance. Common approaches either
use Reinforcement Learning, which experiences
training difficulties, or greedily maximize the con-
ditional mutual information of the label and unob-
served features, which makes myopic acquisitions.
To address these shortcomings, we introduce a la-
tent variable model, trained in a supervised man-
ner. Acquisitions are made by reasoning about
the features across many possible unobserved re-
alizations in a stochastic latent space. Extensive
evaluation on a large range of synthetic and real
datasets demonstrates that our approach reliably
outperforms a diverse set of baselines.
1. Introduction
The standard supervised learning paradigm is to learn a pre-
dictive model using a training dataset of features and labels,
such that the model can make accurate predictions on unseen
test inputs. A fundamental assumption is that, at test time,
all features are jointly available. However, this assumption
does not always hold. Consider the example of a doctor di-
agnosing a patient (Kachuee et al., 2019b;a). Initially, there
is little to no information available and, while there are many
tests that could be conducted, the doctor will choose which
ones to carry out based on their current understanding of
the specific patient’s condition. For instance, if a patient has
pain in their leg, and the doctor suspects a fracture, a leg X-
1Department of Computer Science and Technology, University
of Cambridge, Cambridge, United Kingdom 2Department of Artifi-
cial Intelligence, Korea University, Seoul, Korea 3Department
of Statistics, University of Oxford, Oxford, United Kingdom
4Department of Applied Mathem...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000006. Conference bucket: ICML.
Conference: ICML 2025

Graph Neural Network Generalization
with Gaussian Mixture Model Based Augmentation
Yassine Abbahaddou 1 * Fragkiskos D. Malliaros 2 Johannes F. Lutzeyer 1
Amine M. Aboussalah 3 Michalis Vazirgiannis 1 4
Abstract
Graph Neural Networks (GNNs) have shown
great promise in tasks like node and graph classi-
fication, but they often struggle to generalize, par-
ticularly to unseen or out-of-distribution (OOD)
data.
These challenges are exacerbated when
training data is limited in size or diversity. To
address these issues, we introduce a theoretical
framework using Rademacher complexity to com-
pute a regret bound on the generalization error
and then characterize the effect of data augmen-
tation. This framework informs the design of
GRATIN, an efficient graph data augmentation
algorithm leveraging the capability of Gaussian
Mixture Models (GMMs) to approximate any dis-
tribution. Our approach not only outperforms
existing augmentation techniques in terms of gen-
eralization but also offers improved time com-
plexity, making it highly suitable for real-world
applications. Our code is publicly available at:
https://github.com/abbahaddou/GRATIN.
1. Introduction
Graphs are a fundamental and ubiquitous structure for mod-
eling complex relationships and interactions. In biology,
graphs are employed to represent complex networks of pro-
tein interactions and in drug discovery by modeling molecu-
lar relationships (Gaudelet et al., 2021; Jagtap et al., 2022).
Similarly, social networks capture relationships and commu-
nity interactions (Aboussalah et al., 2023a; Zeng et al., 2022;
Malliaros & Vazirgiannis, 2013; Newman et al., 2002). To
address the unique challenges posed by graph-structured
*Work partially completed during a research visit to NYU.
1LIX, ´Ecole Polytechnique, IP Paris 2Universit´e Paris-Saclay,
CentraleSup´elec, Inria 3NYU Tandon School of Engineering
4MBZUAI. Correspondence to:
Yassine Abbahaddou <yas-
sine.abbahaddou@polytechnique.edu>.
Proceedings of the 42 nd...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000014. Conference bucket: ICML.
Conference: ICML 2025

Loss Functions and Operators Generated by f-Divergences
Vincent Roulet 1 Tianlin Liu 1 Nino Vieillard 1 Micha¨el E. Sander 1 Mathieu Blondel 1
Abstract
The logistic loss (a.k.a. cross-entropy loss) is one
of the most popular loss functions used for mul-
ticlass classification. It is also the loss function
of choice for next-token prediction in language
modeling. It is associated with the Kullback–
Leibler (KL) divergence and the softargmax oper-
ator. In this work, we build upon Fenchel-Young
losses to construct convex loss functions gener-
ated from f-divergences. Our loss functions gen-
eralize the logistic loss in two directions: i) by
replacing the KL divergence with f-divergences
and ii) by allowing non-uniform reference mea-
sures. We instantiate our framework for numerous
f-divergences, recovering existing losses and cre-
ating new ones. By analogy with the logistic loss,
the loss function generated by an f-divergence
is associated with an operator, that we dub f-
softargmax. We derive a novel parallelizable bi-
section algorithm for computing the f-softargmax
associated with any f-divergence. On the empiri-
cal side, one of the goals of this paper is to deter-
mine the effectiveness of loss functions beyond
the classical cross-entropy in a language model
setting, including on pre-training, post-training
(SFT) and distillation. We show that the loss
function generated by the α-divergence (which is
equivalent to Tsallis α-negentropy in the case of
unit reference measures) with α = 1.5 performs
well across several tasks.
1. Introduction
The logistic loss, a.k.a. cross-entropy loss, is widely used
for multiclass classification. It is associated with the sof-
targmax (a.k.a. softmax) operator, which turns logits into
class probabilities. The logistic loss and the softargmax
operators are also frequently used in the space of tokens for
language model pre-training, post-training and distillation.
1Google DeepMind. Correspondence to: Mathieu Blondel,
Vincent Roulet ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000015. Conference bucket: ICML.
Conference: ICML 2025

Zero-Inflated Bandits
Haoyu Wei * 1 Runzhe Wan * 2 Lei Shi 2 Rui Song 2
Abstract
Many real-world bandit applications are charac-
terized by sparse rewards, which can significantly
hinder learning efficiency. Leveraging problem-
specific structures for careful distribution model-
ing is recognized as essential for improving esti-
mation efficiency in statistics. However, this ap-
proach remains under-explored in the context of
bandits. To address this gap, we initiate the study
of zero-inflated bandits, where the reward is mod-
eled using a classic semi-parametric distribution
known as the zero-inflated distribution. We de-
velop algorithms based on the Upper Confidence
Bound and Thompson Sampling frameworks for
this specific structure. The superior empirical
performance of these methods is demonstrated
through extensive numerical studies.
1. Introduction
Bandit algorithms have been widely applied in areas such
as clinical trials (Durand et al., 2018), finance (Shen et al.,
2015), recommendation systems (Zhou et al., 2017), among
others. Accurate uncertainty quantification is key to ad-
dressing the exploration-exploitation trade-off and typically
requires on certain reward distribution assumptions. Exist-
ing assumptions can be roughly classified into two groups:
• Parametric: the reward distribution is assumed to be-
long to a parameterized family, such as Gaussian or
Bernoulli distributions (Audibert et al., 2010; Krause
& Ong, 2011; Agrawal & Goyal, 2012). The strong
assumption typically ensures clean algorithmic design
and nice theoretical results. However, when misspeci-
fied, it may lead to over- or under-exploration.
*Equal contribution 1Department of Economics, University of
California San Diego, La Jolla, USA 2Amazon, Seattle, USA.
This work does not relate to the positions at Amazon.
Cor-
respondence to: Haoyu Wei <h8wei@ucsd.edu>, Rui Song
<songray@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 2...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000016. Conference bucket: ICML.
Conference: ICML 2025

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth
Observation Applications
Ibrahim Fayad 1 2 Max Zimmer 3 Martin Schwartz 1 Fabian Gieseke 4 Philippe Ciais 1 Gabriel Belouze 1
Sarah Brood 5 Aurelien De Truchis 2 Alexandre d’Aspremont 5 2
Abstract
Significant efforts have been directed towards
adapting self-supervised multimodal learning for
Earth observation applications. However, most
current methods produce coarse patch-sized em-
beddings, limiting their effectiveness and inte-
gration with other modalities like LiDAR. To
close this gap, we present DUNIA, an ap-
proach to learn pixel-sized embeddings through
cross-modal alignment between images and full-
waveform LiDAR data. As the model is trained in
a contrastive manner, the embeddings can be di-
rectly leveraged in the context of a variety of envi-
ronmental monitoring tasks in a zero-shot setting.
In our experiments, we demonstrate the effec-
tiveness of the embeddings for seven such tasks:
canopy height mapping, fractional canopy cover,
land cover mapping, tree species identification,
plant area index, crop type classification, and per-
pixel waveform-based vertical structure mapping.
The results show that the embeddings, along with
zero-shot classifiers, often outperform specialized
supervised models, even in low-data regimes. In
the fine-tuning setting, we show strong perfor-
mances near or better than the state-of-the-art on
five out of six tasks.
1. Introduction
With the rapid expansion of Earth Observation (EO) satellite
missions, deep learning has emerged as a powerful solution
1Laboratoire des Sciences du Climat et de l’Environnement,
LSCE/IPSL, France
2Kayrros SAS, Paris 75009,
France
3Department for AI in Society, Science, and Technology, Zuse In-
stitute Berlin, Germany 4Department of Information Systems, Uni-
versity of M¨unster, Germany 5Department of Computer Science,
CNRS, INRIA & ´Ecole Normale Sup´erieure, Paris 75230, France.
Correspondence to: Ibrahim Fayad <ifayad@lsce.ipsl....

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000022. Conference bucket: ICML.
Conference: ICML 2025

Learning Joint Interventional Effects from Single-Variable Interventions
in Additive Models
Armin Keki´c 1 Sergio Hernan Garrido Mejia 1 2 Bernhard Sch¨olkopf 1 3 4
Abstract
Estimating causal effects of joint interventions on
multiple variables is crucial in many domains, but
obtaining data from such simultaneous interven-
tions can be challenging. Our study explores how
to learn joint interventional effects using only ob-
servational data and single-variable interventions.
We present an identifiability result for this prob-
lem, showing that for a class of nonlinear additive
outcome mechanisms, joint effects can be inferred
without access to joint interventional data. We
propose a practical estimator that decomposes the
causal effect into confounded and unconfounded
contributions for each intervention variable. Ex-
periments on synthetic data demonstrate that our
method achieves performance comparable to mod-
els trained directly on joint interventional data,
outperforming a purely observational estimator.
1. Introduction
Understanding the effects of interventions is fundamental
across many domains, from designing public health policies
to optimizing business operations and administering medical
treatments. Particularly challenging and important are joint
interventional effects, where simultaneous interventions on
multiple action variables influence a target outcome. Such
scenarios are common in fields like epidemiology (Keki´c
et al., 2023a), e-commerce (Kunz et al., 2023; Schultz et al.,
2023), and medicine (Prosperi et al., 2020).
Consider a company trying to optimize their marketing strat-
egy across multiple channels like social media, email cam-
paigns, and display advertising. While the ultimate goal is
to understand how these channels work together to drive
sales, running experiments for every possible combination
1Max Planck Institute for Intelligent Systems, T¨ubingen, Ger-
many 2Amazon Research, T¨ubingen, Germany 3T¨ubingen AI Cen-
ter, T¨ubingen, Germa...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000027. Conference bucket: ICML.
Conference: ICML 2025

Improved Learning via k-DTW: A Novel Dissimilarity Measure for Curves
Amer Krivoˇsija 1 Alexander Munteanu 1 2 Andr´e Nusser 3 Chris Schwiegelshohn 4
Abstract
This paper introduces k-Dynamic Time Warping
(k-DTW), a novel dissimilarity measure for polyg-
onal curves. k-DTW has stronger metric proper-
ties than Dynamic Time Warping (DTW) and is
more robust to outliers than the Fr´echet distance,
which are the two gold standards of dissimilarity
measures for polygonal curves. We show inter-
esting properties of k-DTW and give an exact
algorithm as well as a (1 + ε)-approximation al-
gorithm for k-DTW by a parametric search for
the k-th largest matched distance. We prove the
first dimension-free learning bounds for curves
and further learning theoretic results. k-DTW not
only admits smaller sample size than DTW for the
problem of learning the median of curves, where
some factors depending on the curves’ complexity
m are replaced by k, but we also show a surpris-
ing separation on the associated Rademacher and
Gaussian complexities: k-DTW admits strictly
smaller bounds than DTW, by a factor ˜Ω(√m)
when k ≪m. We complement our theoretical
findings with an experimental illustration of the
benefits of using k-DTW for clustering and near-
est neighbor classification.
1. Introduction
Handwriting, panel data, time series, sensor-generated tra-
jectories, and many more types of data are instances of
polygonal curves in Euclidean space. These curves are
input for learning processes, both supervised and unsuper-
vised. To compare and quantify the distance of two curves,
a suitable dissimilarity measure is needed that reflects the
fact that similar curves that represent, for instance, the same
1TU Dortmund, Germany 2University of Cologne, Germany
3Universit´e Cˆote d’Azur, CNRS, Inria, France 4Aarhus University,
Denmark. Correspondence to: A. Krivoˇsija <amer.krivosija@tu-
dortmund.de>,
A.
Munteanu
<alexander.munteanu@tu-
dortmund.de>,
A.
Nusser
<andre.nusser@cnrs.fr>,
C.
Schwiege...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000028. Conference bucket: ICML.