AIS-Bench/AIS-Bench huggingface dataset:
ADASPLASH: Adaptive Sparse Flash Attention
Nuno Gonc¸alves 1 Marcos Treviso 2 Andr´e F. T. Martins 1 2 3
Abstract
The computational cost of softmax-based atten-
tion in transformers limits their applicability to
long-context tasks. Adaptive sparsity, of which
α-entmax attention is an example, offers a flexi-
ble data-dependent alternative, but existing imple-
mentations are inefficient and do not leverage the
sparsity to obtain runtime and memory gains. In
this work, we propose ADASPLASH, which com-
bines the efficiency of GPU-optimized algorithms
with the sparsity benefits of α-entmax. We first
introduce a hybrid Halley-bisection algorithm, re-
sulting in a 7-fold reduction in the number of itera-
tions needed to compute the α-entmax transforma-
tion. Then, we implement custom Triton kernels
to efficiently handle adaptive sparsity. Experi-
ments with RoBERTa and ModernBERT for text
classification and single-vector retrieval, along
with GPT-2 for language modeling, show that
our method achieves substantial improvements in
runtime and memory efficiency compared to ex-
isting α-entmax implementations. It approaches—
and in some cases surpasses—the efficiency of
highly optimized softmax implementations like
FlashAttention-2, enabling long-context training
while maintaining strong task performance.1
1. Introduction
Central to the success of transformers (Vaswani et al., 2017)
lies the attention mechanism, where each token in a se-
quence attends directly to every other token. Attention prob-
abilities are computed through the softmax transformation,
which always assigns a nonzero probability to every token.
However, for long context inputs, the accumulation of small
probabilities can lead to dispersion (Veliˇckovi´c et al., 2025).
1Instituto Superior T´ecnico, Universidade de Lisboa, Portu-
gal 2Instituto de Telecomunicac¸˜oes, Lisbon, Portugal 3Unbabel,
Lisbon, Portugal.
Correspondence to:
Nuno Gonc¸alves
<nuno.m.goncalves@tecnico.ulisboa.pt>.
Proceedings of the 42 nd ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000032. Conference bucket: ICML.
Relational Conformal Prediction for Correlated Time Series
Andrea Cini 1 2 Alexander Jenkins 1 3 Danilo Mandic 3 Cesare Alippi 1 4 Filippo Maria Bianchi 5 6
Abstract
We address the problem of uncertainty quantifi-
cation in time series forecasting by exploiting
observations at correlated sequences. Relational
deep learning methods leveraging graph repre-
sentations are among the most effective tools for
obtaining point estimates from spatiotemporal
data and correlated time series. However, the
problem of exploiting relational structures to es-
timate the uncertainty of such predictions has
been largely overlooked in the same context. To
this end, we propose a novel distribution-free ap-
proach based on the conformal prediction frame-
work and quantile regression. Despite the recent
applications of conformal prediction to sequen-
tial data, existing methods operate independently
on each target time series and do not account
for relationships among them when constructing
the prediction interval. We fill this void by in-
troducing a novel conformal prediction method
based on graph deep learning operators.
Our
approach, named Conformal Relational Predic-
tion (COREL), does not require the relational
structure (graph) to be known a priori and can be
applied on top of any pre-trained predictor. Addi-
tionally, COREL includes an adaptive component
to handle non-exchangeable data and changes in
the input time series. Our approach provides ac-
curate coverage and achieves state-of-the-art un-
certainty quantification in relevant benchmarks.
1. Introduction
Many recent advancements in deep learning methods for
time series forecasting rely on learning from large collec-
tions of (related) time series (Benidis et al., 2022; Liang
1IDSIA USI-SUPSI, Universit`a della Svizzera italiana 2Swiss
National Science Foundation Postdoc Fellow 3Imperial College
London 4Politecnico di Milano 5UiT The Arctic University of Nor-
way 6NORCE Norwegian Research Centre AS. Correspondence
to: Andre...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000034. Conference bucket: ICML.
Subgroups Matter for Robust Bias Mitigation
Anissa Alloula 1 Charles Jones 2 Ben Glocker 2 Bartłomiej W. Papie˙z 1
Abstract
Despite the constant development of new bias mit-
igation methods for machine learning, no method
consistently succeeds, and a fundamental question
remains unanswered: when and why do bias miti-
gation techniques fail? In this paper, we hypothe-
sise that a key factor may be the often-overlooked
but crucial step shared by many bias mitigation
methods: the definition of subgroups. To investi-
gate this, we conduct a comprehensive evaluation
of state-of-the-art bias mitigation methods across
multiple vision and language classification tasks,
systematically varying subgroup definitions, in-
cluding coarse, fine-grained, intersectional, and
noisy subgroups. Our results reveal that subgroup
choice significantly impacts performance, with
certain groupings paradoxically leading to worse
outcomes than no mitigation at all. Our find-
ings suggest that observing a disparity between a
set of subgroups is not a sufficient reason to use
those subgroups for mitigation. Through theoret-
ical analysis, we explain these phenomena and
uncover a counter-intuitive insight that, in some
cases, improving fairness with respect to a par-
ticular set of subgroups is best achieved by using
a different set of subgroups for mitigation. Our
work highlights the importance of careful sub-
group definition in bias mitigation and presents it
as an alternative lever for improving the robust-
ness and fairness of machine learning models.
1. Introduction
A significant barrier to the wider deployment of machine
learning (ML) models is their tendency to fail when tested
on distributions that differ from their training data. One
particularly concerning manifestation of this issue is perfor-
mance degradation for population subgroups, often caused
1University of Oxford, UK 2Imperial College London, UK. Cor-
respondence to: Anissa Alloula <anissa.alloula@dtc.ox.ac.uk>.
Proceedings of ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000035. Conference bucket: ICML.
Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent
Santhosh Karnik * 1 Anna Veselovska * 2 3 Mark Iwen 4 5 Felix Krahmer 2 3
Abstract
We provide a rigorous analysis of implicit regu-
larization in an overparametrized tensor factoriza-
tion problem beyond the lazy training regime. For
matrix factorization problems, this phenomenon
has been studied in a number of works. A par-
ticular challenge has been to design universal ini-
tialization strategies which provably lead to im-
plicit regularization in gradient-descent methods.
At the same time, it has been argued by (Cohen
et al., 2016) that more general classes of neural
networks can be captured by considering tensor
factorizations. However, in the tensor case, im-
plicit regularization has only been rigorously es-
tablished for gradient flow or in the lazy training
regime. In this paper, we prove the first tensor
result of its kind for gradient descent rather than
gradient flow. We focus on the tubal tensor prod-
uct and the associated notion of low tubal rank,
encouraged by the relevance of this model for im-
age data. We establish that gradient descent in an
overparametrized tensor factorization model with
a small random initialization exhibits an implicit
bias towards solutions of low tubal rank. Our the-
oretical findings are illustrated in an extensive set
of numerical simulations show-casing the dynam-
ics predicted by our theory as well as the crucial
role of using a small random initialization.
1. Introduction
Analyzing implicit regularization during Neural Network
(NN) training is considered crucial for understanding why
*Equal contribution 1Department of Mathematics, Northeast-
ern University, Boston, USA 2Department of Mathematics and
Munich Data Science Institute, Technical University of Munich,
Munich, Germany 3Munich Center for Machine Learning, Mu-
nich, Germany 4Department of Mathematics, Michigan State
University, East Lansing, USA 5Department of Computational
Mathematics S...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000036. Conference bucket: ICML.
Strategic Planning: A Top-Down Approach to Option Generation
Max Ruiz Luyten * 1 Antonin Berthon * 1 Mihaela van der Schaar 1
Abstract
Real-world human decision making often relies
on strategic planning, where high-level goals
guide the formulation of sub-goals and subsequent
actions, as evidenced by domains such as health-
care, business, and urban policy. Despite notable
successes in controlled settings, conventional re-
inforcement learning (RL) follows a bottom-up
framework, which can struggle to adapt to real-
world complexities such as sparse rewards and
limited exploration budgets. While methods like
hierarchical RL and environment shaping provide
partial solutions, they frequently rely on either
ad hoc designs (e.g. choosing the set of high-
level actions) or purely data-driven discovery of
high-level actions that still requires significant ex-
ploration. In this paper, we introduce a top-down
RL framework that explicitly leverages human-
inspired strategy to reduce sample complexity,
guide exploration, and enable high-level decision
making. We first formalize the Strategy Problem,
which frames policy generation as finding distri-
butions over policies that balance specificity and
value. Building on this definition, we propose
the Strategist agent—an iterative framework that
leverages large language models to synthesize do-
main knowledge into a structured representation
of actionable strategies and sub-goals. We fur-
ther develop a reward shaping methodology that
translates these strategies expressed in natural lan-
guage into quantitative feedback for RL methods.
Empirically, we demonstrate that our framework
significantly enhances the performance of differ-
ent underlying RL algorithms, leading to faster
convergence and the discovery of more complex
behaviors. Taken together, our findings highlight
that top-down strategic exploration opens new
avenues to improve RL in real-world decision
problems.
*Equal contribution 1University of Cambridge. Corresponden...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000038. Conference bucket: ICML.
Aligning Multimodal Representations through an Information Bottleneck
Antonio Almud´evar 1 Jos´e Miguel Hern´andez-Lobato 2 Sameer Khurana 3 Ricard Marxer 4 Alfonso Ortega 1
Abstract
Contrastive losses have been extensively used
as a tool for multimodal representation learning.
However, it has been empirically observed that
their use is not effective to learn an aligned rep-
resentation space. In this paper, we argue that
this phenomenon is caused by the presence of
modality-specific information in the representa-
tion space. Although some of the most widely
used contrastive losses maximize the mutual in-
formation between representations of both modali-
ties, they are not designed to remove the modality-
specific information. We give a theoretical de-
scription of this problem through the lens of the
Information Bottleneck Principle. We also em-
pirically analyze how different hyperparameters
affect the emergence of this phenomenon in a con-
trolled experimental setup. Finally, we propose a
regularization term in the loss function that is de-
rived by means of a variational approximation and
aims to increase the representational alignment.
We analyze in a set of controlled experiments and
real-world applications the advantages of includ-
ing this regularization term.
1. Introduction
Multimodal Learning is an area of AI that is focused on pro-
cessing and integrating information from multiple modali-
ties (e.g., text, image or audio). It is becoming a pivotal topic
in the community because of multiple reasons, including,
but not limited to, (i) it permits to mimic human cognition
processes (Fei et al., 2022; Lee et al., 2023); (ii) it allows to
use a greater amount of training data from different modali-
ties, which tends to improve the performance of the models
1ViVoLab, Arag´on Institute for Engineering Research (I3A),
University of Zaragoza, Zaragoza, Spain 2University of Cam-
bridge, Cambridge, UK 3Mitsubishi Electric Research Labora-
tories (MERL), Cambridge, U...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000039. Conference bucket: ICML.
Graph World Model
Tao Feng 1 Yexin Wu 1 Guanyu Lin 1 Jiaxuan You 1
Abstract
World models (WMs) demonstrate strong capa-
bilities in prediction, generation, and planning
tasks. Existing WMs primarily focus on unstruc-
tured data while cannot leverage the ubiquitous
structured data, often represented as graphs, in
the digital world. While multiple graph foun-
dation models have been proposed, they focus
on graph learning tasks and cannot extend to
diverse multi-modal data and interdisciplinary
tasks. To address these challenges, we propose the
Graph World Model (GWM), a world model that
supports both unstructured and graph-structured
states with multi-modal information and repre-
sents diverse tasks as actions.
The core of a
GWM is a generic message-passing algorithm
to aggregate structured information, either over
a unified multi-modal token space by converting
multi-modal data into text (GWM-T) or a uni-
fied multi-modal embedding space by modality-
specific encoders (GWM-E). Notably, GWM in-
troduces action nodes to support diverse tasks,
where action nodes are linked to other nodes via
direct reference or similarity computation. Exten-
sive experiments on 6 tasks from diverse domains,
including multi-modal generation and matching,
recommendation, graph prediction, multi-agent,
retrieval-augmented generation, and planning and
optimization, show that the same GWM out-
performs or matches domain-specific baselines’
performance, benefits from multi-hop structures,
and demonstrate strong zero-shot/few-shot ca-
pabilities on unseen new tasks. Our codes for
GWM is released at https://github.com/
ulab-uiuc/GWM.
1Department of Computer Science, University of Illinois Ur-
bana Champaign Urbana, IL, USA. Correspondence to: Tao Feng
<taofeng2@illinois.edu>, Jiaxuan You <jiaxuan@illinois.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
A world model (WM) (Ha & Schmid...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000042. Conference bucket: ICML.
Is Complex Query Answering Really Complex?
Cosimo Gregucci 1 Bo Xiong 2 Daniel Hern´andez 1 Lorenzo Loconte 3
Pasquale Minervini 3 4
Steffen Staab 1 5
Antonio Vergari 3
Abstract
Complex query answering (CQA) on knowledge
graphs (KGs) is gaining momentum as a challeng-
ing reasoning task. In this paper, we show that
the current benchmarks for CQA might not be as
complex as we think, as the way they are built dis-
torts our perception of progress in this field. For
example, we find that in these benchmarks, most
queries (up to 98% for some query types) can be
reduced to simpler problems, e.g. link prediction,
where only one link needs to be predicted.The
performance of state-of-the-art CQA models de-
creases significantly when such models are eval-
uated on queries that cannot be reduced to easier
types. Thus, we propose a set of more challeng-
ing benchmarks composed of queries that require
models to reason over multiple hops and better
reflect the construction of real-world KGs. In a
systematic empirical investigation, the new bench-
marks show that current methods leave much to
be desired from current CQA methods.
1. Introduction
A crucial challenge in AI and ML is learning to perform
complex reasoning, i.e., solving tasks that involve several
intermediate steps and sub-goals to be completed. Complex
query answering (CQA; Hamilton et al., 2018; Zhang et al.,
2021; Arakelyan et al., 2021; Zhu et al., 2022) emerged as a
way to measure complex reasoning over external knowledge
bases, encoded as knowledge graphs (KGs; Hogan et al.,
2021). For instance, to answer the query:
“Which actor performed in a movie filmed in
New York City and distributed on Blue Ray?”
(q1)
Shared supervision 1Institute for Artificial Intelligence, Uni-
versity of Stuttgart, Germany 2Stanford University 3School of
Informatics University of Edinburgh Edinburgh, UK 4Miniml.AI
5University of Southampton, UK. Correspondence to: Cosimo
Gregucci <cosimo.gregucci@ki.uni-stuttgart.de>.
Proceedings of th...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000043. Conference bucket: ICML.
A Bregman Proximal Viewpoint on Neural Operators
Abdel-Rahim Mezidi 1 * Jordan Patracone 1 * Saverio Salzo 2 * Amaury Habrard 1 3 Massimiliano Pontil 4 5
Remi Emonet 1 3 Marc Sebban 1
Abstract
We present several advances on neural operators
by viewing the action of operator layers as the
minimizers of Bregman regularized optimization
problems over Banach function spaces. The pro-
posed framework allows interpreting the activa-
tion operators as Bregman proximity operators
from dual to primal space. This novel viewpoint
is general enough to recover classical neural op-
erators as well as a new variant, coined Bregman
neural operators, which includes the inverse acti-
vation operator and features the same expressivity
of standard neural operators. Numerical experi-
ments support the added benefits of the Bregman
variant of Fourier neural operators for training
deeper and more accurate models.
1. Introduction
Neural operators (Kovachki et al., 2021; 2023), a recent
extension of neural networks, have emerged as a versatile
framework for learning mappings between function spaces.
These operators have shown great potential in solving par-
tial differential equations (PDEs) and simulating complex
dynamical systems. The exploration of neural architectures
for the approximation and learning of operators has led to
the development of a variety of models.
One influential contribution is the Fourier Neural Opera-
tor (FNO) (Li et al., 2021a), sketched in Figure 1, which
transforms encoded input data into frequency components
in order to learn intricate relationships in the frequency
domain. More recently, the Group-Equivariant FNO (G-
FNO) (Helwig et al., 2023) additionally leverages symme-
*Equal contribution
1Université Jean Monnet Saint-Etienne,
CNRS, Institut d’Optique Graduate School, Inria, Laboratoire
Hubert Curien UMR 5516, F-42023, SAINT-ETIENNE, France
2DIAG, Sapienza University of Rome, 00185 Rome, Italy 3Institut
Universitaire de France (IUF) 4Computational Statisti...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000046. Conference bucket: ICML.
FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for
Enabling Fair LLM-Based Recommender Systems
Arya Fayyazi 1 Mehdi Kamal 1 Massoud Pedram 1
Abstract
We propose FACTER, a fairness-aware frame-
work for LLM-based recommendation systems
that integrates conformal prediction with dynamic
prompt engineering.
By introducing an adap-
tive semantic variance threshold and a violation-
triggered mechanism, FACTER automatically
tightens fairness constraints whenever biased pat-
terns emerge. We further develop an adversar-
ial prompt generator that leverages historical vi-
olations to reduce repeated demographic biases
without retraining the LLM. Empirical results
on MovieLens and Amazon show that FACTER
substantially reduces fairness violations (up to
95.5%) while maintaining strong recommenda-
tion accuracy, revealing semantic variance as a
potent proxy of bias.
1. Introduction
Large Language Models (LLMs) have significantly ad-
vanced natural language processing (NLP), demonstrating
robust generative capabilities across tasks including summa-
rization, dialogue, code completion, and creative composi-
tion. Representative models such as GPT-3 (Brown et al.,
2020), BERT (Devlin et al., 2019), Llama-2 (Touvron et al.,
2023), Llama-3 (Dubey et al., 2024), and Mistral-7B (Jiang
et al., 2023) leverage massive corpora and complex architec-
tures to produce remarkably fluent text, often approaching
or matching human performance in various linguistic bench-
marks. Yet a growing body of work reveals that these models
can inadvertently perpetuate or even amplify biases related
to sensitive attributes such as race, gender, or age (Sheng
et al., 2019; Blodgett et al., 2020; Bary et al., 2021). Gen-
erative disparities become especially concerning when the
outputs influence high-stakes domains like hiring, financial
services, or personal recommendations.
1University of Southern California, Los Angeles, California,
USA. Correspondence to: Arya Fayyazi <afayyazi@usc.e...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000048. Conference bucket: ICML.
PANDAS: Improving Many-shot Jailbreaking via
Positive Affirmation, Negative Demonstration, and Adaptive Sampling
Avery Ma 1 Yangchen Pan 2 Amir-massoud Farahmand 3
Abstract
Many-shot jailbreaking circumvents the safety
alignment of LLMs by exploiting their ability
to process long input sequences.
To achieve
this, the malicious target prompt is prefixed with
hundreds of fabricated conversational exchanges
between the user and the model. These exchanges
are randomly sampled from a pool of unsafe
question-answer pairs, making it appear as
though the model has already complied with
harmful instructions. In this paper, we present
PANDAS: a hybrid technique that improves
many-shot jailbreaking by modifying these
fabricated dialogues with Positive Affirmations,
Negative Demonstrations, and an optimized
Adaptive Sampling method tailored to the target
prompt’s topic. We also introduce ManyHarm,
a dataset of harmful question–answer pairs, and
demonstrate through extensive experiments that
PANDAS significantly outperforms baseline
methods in long-context scenarios.
Through
attention analysis, we provide insights into how
long-context vulnerabilities are exploited and
show how PANDAS further improves upon many-
shot jailbreaking. Our source code is available at
https://github.com/averyma/pandas.
Warning: This paper contains model behavior
that can be offensive or harmful in nature.
1. Introduction
The growing length of context windows in large lan-
guage models (LLMs) unlocks applications, such as agentic
LLMs (Park et al., 2023), that were previously impractical
or severely limited (Team et al., 2024; Ding et al., 2024; Jin
et al., 2024; Wu et al., 2024; Dong et al., 2024).
1University of Toronto, Vector Institute 2University of Oxford
3Polytechnique Montr´eal, Mila - Quebec AI Institute, University of
Toronto. Correspondence to: Avery Ma <ama@cs.toronto.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000051. Conference bucket: ICML.