Yancy567/AIS-Bench huggingface dataset:
Conference: NeurIPS 2025

Brain network science modelling of sparse neural
networks enables Transformers and LLMs to perform
as fully connected
Yingtao Zhang1,2, Diego Cerretti1,2, Jialin Zhao1,2, Ziheng Liao1,2, Wenjing Wu1,2
Umberto Michieli4,5 & Carlo Vittorio Cannistraci1,2,3∗
1Center for Complex Network Intelligence (CCNI)†
2Dept. of Computer Science & Technology, 3School of Biomedical Engineering, Tsinghua University
4University of Padova, 5Canva Research
Abstract
This study aims to enlarge our current knowledge on the application of brain-
inspired network science principles for training artificial neural networks (ANNs)
with sparse connectivity. Dynamic sparse training (DST) emulates the synaptic
turnover of real brain networks, reducing the computational demands of train-
ing and inference in ANNs. However, existing DST methods face difficulties in
maintaining peak performance at high connectivity sparsity levels. The Cannistraci-
Hebb training (CHT) is a brain-inspired method that is used in DST for growing
synaptic connectivity in sparse neural networks. CHT leverages a gradient-free,
topology-driven link regrowth mechanism, which has been shown to achieve ultra-
sparse (1% connectivity or lower) advantage across various tasks compared to
fully connected networks. Yet, CHT suffers two main drawbacks: (i) its time
complexity is O(N · d3)- N node network size, d node degree - hence it can be
efficiently applied only to ultra-sparse networks. (ii) it rigidly selects top link
prediction scores, which is inappropriate for the early training epochs, when the
network topology presents many unreliable connections. Here, we design the first
brain-inspired network model - termed bipartite receptive field (BRF) - to initialize
the connectivity of sparse artificial neural networks. Then, we propose a matrix
multiplication GPU-friendly approximation of the CH link predictor, which reduces
the computational complexity to O(N 3), enabling a fast implementation of link
prediction in large-scale m...
Conference: NeurIPS 2025

Adaptive Cannistraci-Hebb Network Automata
Modelling of Complex Networks for Path-based Link
Prediction
Jialin Zhao1,2
Alessandro Muscoloni1,3,4
Umberto Michieli5,6
Yingtao Zhang1,2
Carlo Vittorio Cannistraci1,2,3,4 ∗
1Center for Complex Network Intelligence (CCNI) †
2Dept. of Computer Science & Technology, 3School of Biomedical Engineering, Tsinghua University
4Biomedical Cybernetics Group, Technische Universität Dresden, Germany ‡
5University of Padova, Italy, 6Canva Research
Abstract
Many complex networks have partially observed or evolving connectivity, making
link prediction a fundamental task. Topological link prediction infers missing
links using only network topology, with applications in social, biological, and
technological systems. The Cannistraci-Hebb (CH) theory provides a topological
formulation of Hebbian learning, grounded on two pillars: (1) the minimization
of external links within local communities, and (2) the path-based definition
of local communities that capture homophilic (similarity-driven) interactions via
paths of length 2 and synergetic (diversity-driven) interactions via paths of length
3. Building on this, we introduce the Cannistraci-Hebb Adaptive (CHA) network
automata, an adaptive learning machine that automatically selects the optimal CH
rule and path length to model each network. CHA unifies theoretical interpretability
and data-driven adaptivity, bridging physics-inspired network science and machine
intelligence. Across 1,269 networks from 14 domains, CHA consistently surpasses
state-of-the-art methods—including SPM, SBM, graph embedding methods, and
message-passing graph neural networks—while revealing the mechanistic princi-
ples governing link formation. Our code is available at https://github.com/
biomedical-cybernetics/Cannistraci_Hebb_network_automata.
1
Introduction
Many complex networks have a connectivity that might be only partially detected or that tends to grow
over time, hence the prediction of non-observed links is a...
Conference: ICML 2025

Gradient-based Explanations for Deep Learning Survival Models
Sophie Hanna Langbein * 1 2 Niklas Koenen * 1 2 Marvin N. Wright 1 2 3
Abstract
Deep learning survival models often outperform
classical methods in time-to-event predictions,
particularly in personalized medicine, but their
“black box” nature hinders broader adoption. We
propose a framework for gradient-based explana-
tion methods tailored to survival neural networks,
extending their use beyond regression and clas-
sification. We analyze the implications of their
theoretical assumptions for time-dependent ex-
planations in the survival setting and propose ef-
fective visualizations incorporating the temporal
dimension. Experiments on synthetic data show
that gradient-based methods capture the magni-
tude and direction of local and global feature
effects, including time dependencies. We intro-
duce GradSHAP(t), a gradient-based counterpart
to SurvSHAP(t), which outperforms SurvSHAP(t)
and SurvLIME in a computational speed vs. accu-
racy trade-off. Finally, we apply these methods to
medical data with multi-modal inputs, revealing
relevant tabular features and visual patterns, as
well as their temporal dynamics.
1. Introduction
As medical databases expand to include patients’ detailed
medical history and genetic information, healthcare is shift-
ing from population-based models and traditional statistical
approaches targeting the “average” patient to more complex
personalized medicine, which tailors diagnoses to individ-
ual patient characteristics. Survival analysis is fundamental
to medical data analysis, modeling time-to-event outcomes
while accounting for censoring, enabling personalized risk
predictions, and assessing treatment effects to advance clini-
*Equal contribution 1Leibniz Institute for Prevention Research
and Epidemiology – BIPS, Bremen, Germany 2Faculty of Math-
ematics and Computer Science, University of Bremen, Germany
3Department of Public Health, University of Copenhagen, Den-
mark. Corre...
Conference: ICML 2025

Enhancing Adversarial Robustness with Conformal Prediction: A Framework
for Guaranteed Model Reliability
Jie Bao 1 2 Chuangyin Dang 3 Rui Luo† 2 3 Hanwei Zhang 4 Zhixin Zhou 5
Abstract
As deep learning models are increasingly de-
ployed in high-risk applications, robust defenses
against adversarial attacks and reliable perfor-
mance guarantees become paramount. Moreover,
accuracy alone does not provide sufficient assur-
ance or reliable uncertainty estimates for these
models. This study advances adversarial train-
ing by leveraging principles from Conformal Pre-
diction. Specifically, we develop an adversarial
attack method, termed OPSA (OPtimal Size At-
tack), designed to reduce the efficiency of con-
formal prediction at any significance level by
maximizing model uncertainty without requiring
coverage guarantees. Correspondingly, we intro-
duce OPSA-AT (Adversarial Training), a defense
strategy that integrates OPSA within a novel con-
formal training paradigm. Experimental evalua-
tions demonstrate that our OPSA attack method
induces greater uncertainty compared to baseline
approaches for various defenses. Conversely, our
OPSA-AT defensive model significantly enhances
robustness not only against OPSA but also other
adversarial attacks, and maintains reliable pre-
diction. Our findings highlight the effectiveness
of this integrated approach for developing trust-
worthy and resilient deep learning models for
safety-critical domains. Our code is available
at https://github.com/bjbbbb/Enha
ncing-Adversarial-Robustness-wit
h-Conformal-Prediction.
1Huaiyin Institute of Technology, Huai’an, China 2Chengdu
Research Institute, City University of Hong Kong, Chengdu, China
3City University of Hong Kong, Hong Kong, China 4Saarland Uni-
versity, Saarbr¨ucken, Germany 5Alpha Benito Research, Los Ange-
les, USA. Correspondence to: Rui Luo <ruiluo@cityu.edu.hk>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 202...
Conference: ICML 2025

scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data
Olga Ovcharenko * 1 Florian Barkmann * 2 Philip Toma * 2 Imant Daunhawer 2 Julia E. Vogt 2
Sebastian Schelter † 1 Valentina Boeva † 2 3 4
Abstract
Self-supervised learning (SSL) has proven to be
a powerful approach for extracting biologically
meaningful representations from single-cell data.
To advance our understanding of SSL methods
applied to single-cell data, we present scSSL-
Bench, a comprehensive benchmark that evalu-
ates nineteen SSL methods. Our evaluation spans
nine datasets and focuses on three common down-
stream tasks: batch correction, cell type anno-
tation, and missing modality prediction.
Fur-
thermore, we systematically assess various data
augmentation strategies.
Our analysis reveals
task-specific trade-offs: the specialized single-cell
frameworks, scVI, CLAIRE, and the finetuned
scGPT excel at uni-modal batch correction, while
generic SSL methods, such as VICReg and Sim-
CLR, demonstrate superior performance in cell
typing and multi-modal data integration. Random
masking emerges as the most effective augmenta-
tion technique across all tasks, surpassing domain-
specific augmentations. Notably, our results indi-
cate the need for a specialized single-cell multi-
modal data integration framework. scSSL-Bench
provides a standardized evaluation platform and
concrete recommendations for applying SSL to
single-cell analysis, advancing the convergence
of deep learning and single-cell genomics.
1. Introduction
Recent progress in single-cell RNA sequencing (scRNA-
seq) and multi-omics sequencing technologies has trans-
formed our understanding of cellular heterogeneity by en-
*† Equal contribution 1BIFOLD & TU Berlin, Berlin, Ger-
many 2Department of Computer Science, ETH Zurich, Zurich,
Switzerland
3Swiss Institute of Bioinformatics,
Lausanne,
Switzerland 4Paris Cit´e University, Cochin Institute, INSERM
U1016, Paris, France. Correspondence to: Olga Ovcharenko
<ovcharenko@tu-berlin....
Conference: ICML 2025

Neural Guided Diffusion Bridges
Gefan Yang 1 Frank van der Meulen 2 Stefan Sommer 1
Abstract
We propose a novel method for simulating condi-
tioned diffusion processes (diffusion bridges) in
Euclidean spaces. By training a neural network
to approximate bridge dynamics, our approach
eliminates the need for computationally intensive
Markov Chain Monte Carlo (MCMC) methods
or score modeling. Compared to existing meth-
ods, it offers greater robustness across various
diffusion specifications and conditioning scenar-
ios. This applies in particular to rare events and
multimodal distributions, which pose challenges
for score-learning- and MCMC-based approaches.
We introduce a flexible variational family, par-
tially specified by a neural network, for approxi-
mating the diffusion bridge path measure. Once
trained, it enables efficient sampling of indepen-
dent bridges at a cost comparable to sampling the
unconditioned (forward) process.
1. Introduction
Diffusion processes play a fundamental role in various fields
such as mathematics, physics, evolutionary biology, and, re-
cently, generative models. In particular, diffusion processes
conditioned to hit a specific point at a fixed future time,
which are often referred to as diffusion bridges, are of great
interest in situations where observations constrain the dy-
namics of a stochastic process. For example, in generative
modeling, stochastic imputation between two given images,
also known as the image translation task, uses diffusion
bridges to model dynamics (Zhou et al., 2024; Zheng et al.,
2025). In the area of stochastic shape analysis and computa-
tional anatomy, random evolutions of biological shapes of
organisms are modeled as non-linear diffusion bridges, and
simulating such bridges is critical to solving inference and
1Department of Computer Science, University of Copenhagen,
Universitetsparken 1, 2100 København, Denmark 2Department of
Mathematics, Vrije Universiteit Amsterdam, De Boelelaan 1111,
1081HV Amsterdam...
Conference: ICML 2025

A Sample-Efficient Conditional Independence Test
in the Presence of Discretization
Boyang Sun 1 Yu Yao 2 Xinshuai Dong 3 Zongfang Liu 1 Tongliang Liu 2 1 Yumou Qiu 4 Kun Zhang 3 1
Abstract
In many real-world scenarios, interested variables
are often represented as discretized values due
to measurement limitations.
Applying Condi-
tional Independence (CI) tests directly to such
discretized data, however, can lead to incorrect
conclusions. To address this, recent advancements
have sought to infer the correct CI relationship
between the latent variables through binarizing
observed data. However, this process inevitably
results in a loss of information, which degrades
the test’s performance. Motivated by this, this
paper introduces a sample-efficient CI test that
does not rely on the binarization process. We
find that the independence relationships of latent
continuous variables can be established by ad-
dressing an over-identifying restriction problem
with Generalized Method of Moments (GMM).
Based on this insight, we derive an appropriate
test statistic and establish its asymptotic distribu-
tion correctly reflecting CI by leveraging node-
wise regression. Theoretical findings and Empir-
ical results across various datasets demonstrate
that the superiority and effectiveness of our pro-
posed test. Our code implementation is provided
in https://github.com/boyangaaaaa/DCT.
1. Introduction
Conditional independence tests for discrete variables are
fundamental in statistical analysis and widely applied across
various disciplines. Traditional methods including the chi-
squared test (F.R.S., 2009), the G-test (likelihood ratio test)
(McDonald, 2009), and measures based on conditional mu-
tual information (Kubkowski et al., 2021) are well estab-
lished and extensively used. However, a critical yet often
1Mohamed bin Zayed University of Artificial Intelligence
2Sydney AI Centre, The University of Sydney 3Carnegie Mellon
University 4Peking University. Correspondence to: Yumou Q...
Conference: ICML 2025

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Prasanna Mayilvahanan * 1 2 3 4 Thadd¨aus Wiedemer * 1 2 3 4 Sayak Mallick 1 4 Matthias Bethge 2 3 4
Wieland Brendel 1 2 3
Abstract
Scaling laws guide the development of large lan-
guage models (LLMs) by offering estimates for
the optimal balance of model size, tokens, and
compute. More recently, loss-to-loss scaling laws
that relate losses across pretraining datasets and
downstream tasks have emerged as a powerful
tool for understanding and improving LLM per-
formance and generalization. In this work, we
investigate which factors most strongly influence
loss-to-loss scaling. Our experiments reveal that
the pretraining data determines the scaling trend.
In contrast, model size, optimization hyperparam-
eters, tokenizer and even significant architectural
differences, such as between transformer-based
models like Llama and state-space models like
Mamba, generally have limited impact. Conse-
quently, practitioners should carefully curate pre-
training datasets for optimal downstream perfor-
mance, while architectures and other settings can
be freely optimized for training efficiency.
1. Introduction
Scaling laws have long guided Large Language Model
(LLM) pretraining, determining model and data size un-
der a fixed compute budget (Kaplan et al., 2020; Hoffmann
et al., 2022; Grattafiori et al., 2024). Typically, scaling laws
relate model performance, usually measured as training or
validation loss, to total compute measured in floating point
operations (FLOPs). FLOPs account for both parameter
count and the number of training tokens. While useful for
pretraining, scaling laws do not capture how well a model ul-
timately performs on downstream tasks (Gadre et al., 2024;
Schaeffer et al., 2024; Du et al., 2025). Consequently, mul-
tiple works have begun to investigate downstream scaling
*Equal contribution 1Max Planck Institute for Intelligent Sys-
tems 2ELLIS Institute T¨ubingen 3T¨ubingen AI Center 4University
...
Conference: NeurIPS 2025

Brain-tuning Improves Generalizability and Efficiency
of Brain Alignment in Speech Models
Omer Moussa
Max Planck Institute for Software Systems
Saarbrücken, Germany
omoussa@mpi-sws.org
Mariya Toneva
Max Planck Institute for Software Systems
Saarbrücken, Germany
mtoneva@mpi-sws.org
Abstract
Pretrained language models are remarkably effective in aligning with human brain
responses elicited by natural language stimuli, positioning them as promising
model organisms for studying language processing in the brain. However, existing
approaches for both estimating and improving this brain alignment are participant-
dependent and highly affected by the amount of data available per participant,
hindering both generalization to new participants and population-level analyses.
In this work, we address these limitations by introducing a scalable, generalizable
brain-tuning method, in which we fine-tune pretrained speech language models
to jointly predict fMRI responses from multiple participants. We demonstrate
that the resulting brain-tuned models exhibit strong individual brain alignment
while generalizing across participants. Specifically, our method leads to 1) a
5-fold decrease in the amount of fMRI data needed to predict brain data from
new participants, 2) up to a 50% increase in the overall brain alignment, and 3)
strong generalization to new unseen datasets. Furthermore, this multi-participant
brain-tuning additionally improves downstream performance on semantic tasks,
suggesting that training using brain data from multiple participants leads to more
generalizable semantic representations. Taken together, these findings demon-
strate a bidirectional benefit between neuroscience and AI, helping bridge the
gap between the two fields. We make our code and models publicly available at
https://github.com/bridge-ai-neuro/multi-brain-tuning.
1
Introduction
Language models (LMs) substantially align with human brain responses elicited during natural
language processing, such as fu...
Conference: ICML 2025

Sliding Puzzles Gym: A Scalable Benchmark for
State Representation in Visual Reinforcement Learning
Bryan L. M. de Oliveira 1 2 Luana G. B. Martins 1 Bruno Brand˜ao 1 2 Murilo L. da Luz 1 2
Telma W. de L. Soares 1 2 Luckeciano C. Melo 1 3
Abstract
Effective visual representation learning is crucial
for reinforcement learning (RL) agents to extract
task-relevant information from raw sensory in-
puts and generalize across diverse environments.
However, existing RL benchmarks lack the ability
to systematically evaluate representation learning
capabilities in isolation from other learning chal-
lenges. To address this gap, we introduce the Slid-
ing Puzzles Gym (SPGym), a novel benchmark
that transforms the classic 8-tile puzzle into a vi-
sual RL task with images drawn from arbitrarily
large datasets. SPGym’s key innovation lies in its
ability to precisely control representation learning
complexity through adjustable grid sizes and im-
age pools, while maintaining fixed environment
dynamics, observation, and action spaces. This
design enables researchers to isolate and scale the
visual representation challenge independently of
other learning components. Through extensive ex-
periments with model-free and model-based RL
algorithms, we uncover fundamental limitations
in current methods’ ability to handle visual diver-
sity. As we increase the pool of possible images,
all algorithms exhibit in- and out-of-distribution
performance degradation, with sophisticated rep-
resentation learning techniques often underper-
forming simpler approaches like data augmen-
tation. These findings highlight critical gaps in
visual representation learning for RL and estab-
lish SPGym as a valuable tool for driving progress
in robust, generalizable decision-making systems.
1Advanced Knowledge Center for Immersive Technologies
– AKCIT, Brazil 2Institute of Informatics, Federal University of
Goi´as, Goiˆania, Brazil 3OATML, University of Oxford, United
Kingdom. Correspondence to: Bryan L. M. d...
Conference: NeurIPS 2025

HybridNorm: Towards Stable and Efficient
Transformer Training via Hybrid Normalization
Zhijian Zhuo1, 2
Yutao Zeng2, †
Ya Wang2, †
Sijun Zhang2
Jian Yang3
Xiaoqing Li4
Xun Zhou2
Jinwen Ma1
1 School of Mathematical Sciences, Peking University
2 ByteDance Seed
3 Beihang University
4 Capital University of Economics and Business
Abstract
Transformers have become the de facto architecture for a wide range of machine
learning tasks, particularly in large language models (LLMs). Despite their remark-
able performance, many challenges remain in training deep transformer networks,
especially regarding the position of the layer normalization. While Pre-Norm
structures facilitate more stable training owing to their stronger identity path, they
often lead to suboptimal performance compared to Post-Norm. In this paper,
we propose HybridNorm, a simple yet effective hybrid normalization strategy
that integrates the advantages of both Pre-Norm and Post-Norm. Specifically,
HybridNorm employs QKV normalization within the attention mechanism and
Post-Norm in the feed-forward network (FFN) of each transformer block. We
provide both theoretical insights and empirical evidence to demonstrate that Hy-
bridNorm improves the gradient flow and the model robustness. Extensive ex-
periments on large-scale transformer models, including both dense and sparse
variants, show that HybridNorm consistently outperforms both Pre-Norm and
Post-Norm approaches across multiple benchmarks. These findings highlight the
potential of HybridNorm as a more stable and effective technique for improving
the training and performance of deep transformer models. Code is available at
https://github.com/BryceZhuo/HybridNorm.
1
Introduction
Transformers have become the backbone of large language models (LLMs) and a wide range of
machine learning applications. These architectures are capable of modeling long-range dependencies
through self-attention mechanisms, which have made them the preferred choice for a variety of
t...
Conference: ICML 2025

Modular Duality in Deep Learning
Jeremy Bernstein 1 Laker Newhouse 1
Abstract
An old idea in optimization theory says that
since the gradient is a dual vector it may not be
subtracted from the weights without first being
mapped to the primal space where the weights
reside. We take this idea seriously in this paper
and construct such a duality map for general neu-
ral networks. Our map, which we call modular
dualization, forms a unifying theoretical basis for
training algorithms that are a) fast and b) scalable.
Modular dualization involves first assigning op-
erator norms to layers based on the semantics of
each layer, and then using these layerwise norms
to recursively induce a duality map on the weight
space of the full neural architecture. We derive
GPU-friendly algorithms for dualizing Embed,
Linear and Conv2D layers—the latter two meth-
ods are based on a Newton-Schulz iteration. We
conclude with small experiments demonstrating
the speed, scalability and novel numerical prop-
erties of duality-based optimizers. Our methods
were used in the Muon optimizer, which recently
set speed records for training NanoGPT and was
scaled up to a 1.5 billion parameter transformer.
1
Introduction
This paper pursues a rigorous and first-principles theoretical
framework for designing neural network training algorithms.
We hope that such a framework will facilitate the design of
a next generation of fast and scalable optimizers that are
automatically tailored to different neural architectures.
While gradient descent is the workhorse of modern machine
learning, the most vanilla form of the algorithm does not,
in our view, pass a basic type check. For a gradient update
to type check, we insist that the gradient must be passed
through a duality map before being multiplied by a learning
rate and applied to the weights:
1MIT CSAIL, United States. Correspondence to: Jeremy Bern-
stein <jbernstein@mit.edu>, Laker Newhouse <lakern@mit.edu>.
Proceedings of the 42 nd International Conferenc...