AIS-Bench/AIS-Bench huggingface dataset:
scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data
Olga Ovcharenko * 1 Florian Barkmann * 2 Philip Toma * 2 Imant Daunhawer 2 Julia E. Vogt 2
Sebastian Schelter † 1 Valentina Boeva † 2 3 4
Abstract
Self-supervised learning (SSL) has proven to be
a powerful approach for extracting biologically
meaningful representations from single-cell data.
To advance our understanding of SSL methods
applied to single-cell data, we present scSSL-
Bench, a comprehensive benchmark that evalu-
ates nineteen SSL methods. Our evaluation spans
nine datasets and focuses on three common down-
stream tasks: batch correction, cell type anno-
tation, and missing modality prediction.
Fur-
thermore, we systematically assess various data
augmentation strategies.
Our analysis reveals
task-specific trade-offs: the specialized single-cell
frameworks, scVI, CLAIRE, and the finetuned
scGPT excel at uni-modal batch correction, while
generic SSL methods, such as VICReg and Sim-
CLR, demonstrate superior performance in cell
typing and multi-modal data integration. Random
masking emerges as the most effective augmenta-
tion technique across all tasks, surpassing domain-
specific augmentations. Notably, our results indi-
cate the need for a specialized single-cell multi-
modal data integration framework. scSSL-Bench
provides a standardized evaluation platform and
concrete recommendations for applying SSL to
single-cell analysis, advancing the convergence
of deep learning and single-cell genomics.
1. Introduction
Recent progress in single-cell RNA sequencing (scRNA-
seq) and multi-omics sequencing technologies has trans-
formed our understanding of cellular heterogeneity by en-
*† Equal contribution 1BIFOLD & TU Berlin, Berlin, Ger-
many 2Department of Computer Science, ETH Zurich, Zurich,
Switzerland
3Swiss Institute of Bioinformatics,
Lausanne,
Switzerland 4Paris Cit´e University, Cochin Institute, INSERM
U1016, Paris, France. Correspondence to: Olga Ovcharenko
<ovcharenko@tu-berlin....

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000072. Conference bucket: ICML.
Neural Guided Diffusion Bridges
Gefan Yang 1 Frank van der Meulen 2 Stefan Sommer 1
Abstract
We propose a novel method for simulating condi-
tioned diffusion processes (diffusion bridges) in
Euclidean spaces. By training a neural network
to approximate bridge dynamics, our approach
eliminates the need for computationally intensive
Markov Chain Monte Carlo (MCMC) methods
or score modeling. Compared to existing meth-
ods, it offers greater robustness across various
diffusion specifications and conditioning scenar-
ios. This applies in particular to rare events and
multimodal distributions, which pose challenges
for score-learning- and MCMC-based approaches.
We introduce a flexible variational family, par-
tially specified by a neural network, for approxi-
mating the diffusion bridge path measure. Once
trained, it enables efficient sampling of indepen-
dent bridges at a cost comparable to sampling the
unconditioned (forward) process.
1. Introduction
Diffusion processes play a fundamental role in various fields
such as mathematics, physics, evolutionary biology, and, re-
cently, generative models. In particular, diffusion processes
conditioned to hit a specific point at a fixed future time,
which are often referred to as diffusion bridges, are of great
interest in situations where observations constrain the dy-
namics of a stochastic process. For example, in generative
modeling, stochastic imputation between two given images,
also known as the image translation task, uses diffusion
bridges to model dynamics (Zhou et al., 2024; Zheng et al.,
2025). In the area of stochastic shape analysis and computa-
tional anatomy, random evolutions of biological shapes of
organisms are modeled as non-linear diffusion bridges, and
simulating such bridges is critical to solving inference and
1Department of Computer Science, University of Copenhagen,
Universitetsparken 1, 2100 København, Denmark 2Department of
Mathematics, Vrije Universiteit Amsterdam, De Boelelaan 1111,
1081HV Amsterdam...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000073. Conference bucket: ICML.
A Sample-Efficient Conditional Independence Test
in the Presence of Discretization
Boyang Sun 1 Yu Yao 2 Xinshuai Dong 3 Zongfang Liu 1 Tongliang Liu 2 1 Yumou Qiu 4 Kun Zhang 3 1
Abstract
In many real-world scenarios, interested variables
are often represented as discretized values due
to measurement limitations.
Applying Condi-
tional Independence (CI) tests directly to such
discretized data, however, can lead to incorrect
conclusions. To address this, recent advancements
have sought to infer the correct CI relationship
between the latent variables through binarizing
observed data. However, this process inevitably
results in a loss of information, which degrades
the test’s performance. Motivated by this, this
paper introduces a sample-efficient CI test that
does not rely on the binarization process. We
find that the independence relationships of latent
continuous variables can be established by ad-
dressing an over-identifying restriction problem
with Generalized Method of Moments (GMM).
Based on this insight, we derive an appropriate
test statistic and establish its asymptotic distribu-
tion correctly reflecting CI by leveraging node-
wise regression. Theoretical findings and Empir-
ical results across various datasets demonstrate
that the superiority and effectiveness of our pro-
posed test. Our code implementation is provided
in https://github.com/boyangaaaaa/DCT.
1. Introduction
Conditional independence tests for discrete variables are
fundamental in statistical analysis and widely applied across
various disciplines. Traditional methods including the chi-
squared test (F.R.S., 2009), the G-test (likelihood ratio test)
(McDonald, 2009), and measures based on conditional mu-
tual information (Kubkowski et al., 2021) are well estab-
lished and extensively used. However, a critical yet often
1Mohamed bin Zayed University of Artificial Intelligence
2Sydney AI Centre, The University of Sydney 3Carnegie Mellon
University 4Peking University. Correspondence to: Yumou Q...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000074. Conference bucket: ICML.
Graph World Model
Tao Feng 1 Yexin Wu 1 Guanyu Lin 1 Jiaxuan You 1
Abstract
World models (WMs) demonstrate strong capa-
bilities in prediction, generation, and planning
tasks. Existing WMs primarily focus on unstruc-
tured data while cannot leverage the ubiquitous
structured data, often represented as graphs, in
the digital world. While multiple graph foun-
dation models have been proposed, they focus
on graph learning tasks and cannot extend to
diverse multi-modal data and interdisciplinary
tasks. To address these challenges, we propose the
Graph World Model (GWM), a world model that
supports both unstructured and graph-structured
states with multi-modal information and repre-
sents diverse tasks as actions.
The core of a
GWM is a generic message-passing algorithm
to aggregate structured information, either over
a unified multi-modal token space by converting
multi-modal data into text (GWM-T) or a uni-
fied multi-modal embedding space by modality-
specific encoders (GWM-E). Notably, GWM in-
troduces action nodes to support diverse tasks,
where action nodes are linked to other nodes via
direct reference or similarity computation. Exten-
sive experiments on 6 tasks from diverse domains,
including multi-modal generation and matching,
recommendation, graph prediction, multi-agent,
retrieval-augmented generation, and planning and
optimization, show that the same GWM out-
performs or matches domain-specific baselines’
performance, benefits from multi-hop structures,
and demonstrate strong zero-shot/few-shot ca-
pabilities on unseen new tasks. Our codes for
GWM is released at https://github.com/
ulab-uiuc/GWM.
1Department of Computer Science, University of Illinois Ur-
bana Champaign Urbana, IL, USA. Correspondence to: Tao Feng
<taofeng2@illinois.edu>, Jiaxuan You <jiaxuan@illinois.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
A world model (WM) (Ha & Schmid...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000075. Conference bucket: ICML.
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Prasanna Mayilvahanan * 1 2 3 4 Thadd¨aus Wiedemer * 1 2 3 4 Sayak Mallick 1 4 Matthias Bethge 2 3 4
Wieland Brendel 1 2 3
Abstract
Scaling laws guide the development of large lan-
guage models (LLMs) by offering estimates for
the optimal balance of model size, tokens, and
compute. More recently, loss-to-loss scaling laws
that relate losses across pretraining datasets and
downstream tasks have emerged as a powerful
tool for understanding and improving LLM per-
formance and generalization. In this work, we
investigate which factors most strongly influence
loss-to-loss scaling. Our experiments reveal that
the pretraining data determines the scaling trend.
In contrast, model size, optimization hyperparam-
eters, tokenizer and even significant architectural
differences, such as between transformer-based
models like Llama and state-space models like
Mamba, generally have limited impact. Conse-
quently, practitioners should carefully curate pre-
training datasets for optimal downstream perfor-
mance, while architectures and other settings can
be freely optimized for training efficiency.
1. Introduction
Scaling laws have long guided Large Language Model
(LLM) pretraining, determining model and data size un-
der a fixed compute budget (Kaplan et al., 2020; Hoffmann
et al., 2022; Grattafiori et al., 2024). Typically, scaling laws
relate model performance, usually measured as training or
validation loss, to total compute measured in floating point
operations (FLOPs). FLOPs account for both parameter
count and the number of training tokens. While useful for
pretraining, scaling laws do not capture how well a model ul-
timately performs on downstream tasks (Gadre et al., 2024;
Schaeffer et al., 2024; Du et al., 2025). Consequently, mul-
tiple works have begun to investigate downstream scaling
*Equal contribution 1Max Planck Institute for Intelligent Sys-
tems 2ELLIS Institute T¨ubingen 3T¨ubingen AI Center 4University
...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000076. Conference bucket: ICML.
Sparse Autoencoders, Again?
Yin Lu 1 Xuening Zhu 1 Tong He 2 David Wipf 2
Abstract
Is there really much more to say about sparse au-
toencoders (SAEs)? Autoencoders in general, and
SAEs in particular, represent deep architectures
that are capable of modeling low-dimensional la-
tent structure in data. Such structure could re-
flect, among other things, correlation patterns
in large language model activations, or complex
natural image manifolds. And yet despite the
wide-ranging applicability, there have been rel-
atively few changes to SAEs beyond the origi-
nal recipe from decades ago, namely, standard
deep encoder/decoder layers trained with a classi-
cal/deterministic sparse regularizer applied within
the latent space. One possible exception is the
variational autoencoder (VAE), which adopts a
stochastic encoder module capable of producing
sparse representations when applied to manifold
data. In this work we formalize underappreciated
weaknesses with both canonical SAEs, as well as
analogous VAEs applied to similar tasks, and pro-
pose a hybrid alternative model that circumvents
these prior limitations. In terms of theoretical sup-
port, we prove that global minima of our proposed
model recover certain forms of structured data
spread across a union of manifolds. Meanwhile,
empirical evaluations on synthetic and real-world
datasets substantiate the efficacy of our approach
in accurately estimating underlying manifold di-
mensions and producing sparser latent representa-
tions without compromising reconstruction error.
In general, we are able to exceed the performance
of equivalent-capacity SAEs and VAEs, as well as
recent diffusion models where applicable, within
domains such as images and language model acti-
vation patterns. A link to the code is here.
1School of Data Science, Fudan University 2Amazon Web Services.
Correspondence
to:
Yin
Lu
<yinlu23@m.fudan.edu.cn>,
Xuening
Zhu
<xueningzhu@fudan.edu.cn>,
Tong
He
<htong@amazon.com>, David Wipf <davidwipf@gmail.com...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000077. Conference bucket: ICML.
Sliding Puzzles Gym: A Scalable Benchmark for
State Representation in Visual Reinforcement Learning
Bryan L. M. de Oliveira 1 2 Luana G. B. Martins 1 Bruno Brand˜ao 1 2 Murilo L. da Luz 1 2
Telma W. de L. Soares 1 2 Luckeciano C. Melo 1 3
Abstract
Effective visual representation learning is crucial
for reinforcement learning (RL) agents to extract
task-relevant information from raw sensory in-
puts and generalize across diverse environments.
However, existing RL benchmarks lack the ability
to systematically evaluate representation learning
capabilities in isolation from other learning chal-
lenges. To address this gap, we introduce the Slid-
ing Puzzles Gym (SPGym), a novel benchmark
that transforms the classic 8-tile puzzle into a vi-
sual RL task with images drawn from arbitrarily
large datasets. SPGym’s key innovation lies in its
ability to precisely control representation learning
complexity through adjustable grid sizes and im-
age pools, while maintaining fixed environment
dynamics, observation, and action spaces. This
design enables researchers to isolate and scale the
visual representation challenge independently of
other learning components. Through extensive ex-
periments with model-free and model-based RL
algorithms, we uncover fundamental limitations
in current methods’ ability to handle visual diver-
sity. As we increase the pool of possible images,
all algorithms exhibit in- and out-of-distribution
performance degradation, with sophisticated rep-
resentation learning techniques often underper-
forming simpler approaches like data augmen-
tation. These findings highlight critical gaps in
visual representation learning for RL and estab-
lish SPGym as a valuable tool for driving progress
in robust, generalizable decision-making systems.
1Advanced Knowledge Center for Immersive Technologies
– AKCIT, Brazil 2Institute of Informatics, Federal University of
Goi´as, Goiˆania, Brazil 3OATML, University of Oxford, United
Kingdom. Correspondence to: Bryan L. M. d...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000080. Conference bucket: ICML.
Modular Duality in Deep Learning
Jeremy Bernstein 1 Laker Newhouse 1
Abstract
An old idea in optimization theory says that
since the gradient is a dual vector it may not be
subtracted from the weights without first being
mapped to the primal space where the weights
reside. We take this idea seriously in this paper
and construct such a duality map for general neu-
ral networks. Our map, which we call modular
dualization, forms a unifying theoretical basis for
training algorithms that are a) fast and b) scalable.
Modular dualization involves first assigning op-
erator norms to layers based on the semantics of
each layer, and then using these layerwise norms
to recursively induce a duality map on the weight
space of the full neural architecture. We derive
GPU-friendly algorithms for dualizing Embed,
Linear and Conv2D layers—the latter two meth-
ods are based on a Newton-Schulz iteration. We
conclude with small experiments demonstrating
the speed, scalability and novel numerical prop-
erties of duality-based optimizers. Our methods
were used in the Muon optimizer, which recently
set speed records for training NanoGPT and was
scaled up to a 1.5 billion parameter transformer.
1
Introduction
This paper pursues a rigorous and first-principles theoretical
framework for designing neural network training algorithms.
We hope that such a framework will facilitate the design of
a next generation of fast and scalable optimizers that are
automatically tailored to different neural architectures.
While gradient descent is the workhorse of modern machine
learning, the most vanilla form of the algorithm does not,
in our view, pass a basic type check. For a gradient update
to type check, we insist that the gradient must be passed
through a duality map before being multiplied by a learning
rate and applied to the weights:
1MIT CSAIL, United States. Correspondence to: Jeremy Bern-
stein <jbernstein@mit.edu>, Laker Newhouse <lakern@mit.edu>.
Proceedings of the 42 nd International Conferenc...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000083. Conference bucket: ICML.
Guarantees of a Preconditioned Subgradient Algorithm for
Overparameterized Asymmetric Low-rank Matrix Recovery
Paris Giampouras 1 HanQin Cai 2 Ren´e Vidal 3
Abstract
In this paper, we focus on a matrix factorization-
based approach to recover low-rank asymmet-
ric matrices from corrupted measurements. We
propose an Overparameterized Preconditioned
Subgradient Algorithm (OPSA) and provide, for
the first time in the literature, linear convergence
rates independent of the rank of the sought asym-
metric matrix in the presence of gross corrup-
tions. Our work goes beyond existing results in
preconditioned-type approaches addressing their
current limitation, i.e., the lack of convergence
guarantees in the case of asymmetric matrices
of unknown rank. By applying our approach to
(robust) matrix sensing, we highlight its merits
when the measurement operator satisfies a mixed-
norm restricted isometry property. Lastly, we
present extensive numerical experiments that val-
idate our theoretical results and demonstrate the
effectiveness of our approach for different levels
of overparameterization and outlier corruptions.
1. Introduction
Low-rank matrix recovery has been a ubiquitous problem
showing up in numerous applications in the fields of sig-
nal/image processing, machine learning, and data science
(Recht et al., 2010; Chen et al., 2013; Davenport & Romberg,
2016; Cai et al., 2021a; Smith et al., 2024; Wang et al., 2024).
For instance, problems such as matrix sensing (Jain et al.,
2013; Wei et al., 2016; Park et al., 2017; Li et al., 2020),
matrix completion (Candes & Recht, 2008; Nie et al., 2012;
1Department of Computer Science, University of Warwick,
Coventry, England, UK 2Department of Statistics and Data Sci-
ence and Department of Computer Science, University of Central
Florida, Orlando, FL, USA 3Center for Innovation in Data Engi-
neering and Science (IDEAS), Departments of ESE and Radiology,
University of Pennsylvania, Philadelphia, PA, USA. Correspon-
dence to: Re...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000085. Conference bucket: ICML.
Progressive Tempering Sampler with Diffusion
Severi Rissanen * 1 RuiKang OuYang * 2 Jiajun He 2 Wenlin Chen 2 3 Markus Heinonen 1 Arno Solin 1
Jos´e Miguel Hern´andez-Lobato 2
Abstract
Recent research has focused on designing neu-
ral samplers that amortize the process of sam-
pling from unnormalized densities. However, de-
spite significant advancements, they still fall short
of the state-of-the-art MCMC approach, Parallel
Tempering (PT), when it comes to the efficiency
of target evaluations. On the other hand, unlike a
well-trained neural sampler, PT yields only depen-
dent samples and needs to be rerun—at consider-
able computational cost—whenever new samples
are required. To address these weaknesses, we
propose the Progressive Tempering Sampler with
Diffusion (PTSD), which trains diffusion models
sequentially across temperatures, leveraging the
advantages of PT to improve the training of neu-
ral samplers. We also introduce a novel method
to combine high-temperature diffusion models to
generate approximate lower-temperature samples,
which are minimally refined using MCMC and
used to train the next diffusion model. PTSD en-
ables efficient reuse of sample information across
temperature levels while generating well-mixed,
uncorrelated samples. Our method significantly
improves target evaluation efficiency, outperform-
ing diffusion-based neural samplers.
1. Introduction
Sampling from probability distributions is a fundamental
problem in many fields of science, including Bayesian infer-
ence (Gelman et al., 2013; Welling & Teh, 2011), statistical
physics (Von Toussaint, 2011) and molecular simulations
*Equal contribution
1Department of Computer Science,
Aalto University, Finland 2Department of Engineering, Univer-
sity of Cambridge, United Kingdom 3Department of Empir-
ical Inference, Max Planck Institute for Intelligent Systems,
T¨ubingen, Germany. Correspondence to: Severi Rissanen <sev-
eri.rissanen@aalto.fi>, RuiKang OuYang <ro352@cam.ac.uk>,
Jiajun He <jh2383@...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000088. Conference bucket: ICML.
Generating Hypotheses of Dynamic Causal Graphs in Neuroscience:
Leveraging Generative Factor Models of Observed Time Series
Zachary C. Brown 1 David Carlson 1 2
Abstract
The field of hypothesis generation promises to re-
duce costs in neuroscience by narrowing the range
of interventional studies needed to study various
phenomena. Existing machine learning methods
can generate scientific hypotheses from complex
datasets, but many approaches assume causal rela-
tionships are static over time, limiting their appli-
cability to systems with dynamic, state-dependent
behavior, such as the brain. While some tech-
niques attempt dynamic causal discovery through
factor models, they often restrict relationships to
linear patterns or impose other simplifying as-
sumptions. We propose a novel method that mod-
els dynamic graphs as a conditionally weighted
superposition of static graphs, where each static
graph can capture nonlinear relationships. This
approach enables the detection of complex, time-
varying interactions between variables beyond lin-
ear limitations. Our method improves f1-scores
of predicted dynamic causal patterns by roughly
22-28% on average over baselines in some of our
experiments, with some improvements reaching
well over 60%. A case study on real brain data
demonstrates our method’s ability to uncover re-
lationships linked to specific behavioral states,
offering valuable insights into neural dynamics.
1. Introduction
Causal discovery methods applied to brain dynamics hold
great promise for designing targeted interventions in var-
ious neurological diseases. A typical approach is to gen-
erate hypotheses about causal relationships between brain
1Department of Electrical and Computer Engineering, Duke
University, Durham, NC 15213, USA 2Departments of Civil and
Environmental Engineering; Biostatistics and Bioinformatics; and
Computer Science, Duke University, Durham, NC 15213, USA.
Correspondence to: Zachary C. Brown <zac.brown@duke.edu>,
David Carlson <dav...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000089. Conference bucket: ICML.