dataset-review/dataset-review huggingface dataset:
A Sample Efficient Conditional Independence Test in the Presence of Discretization
ICML 2025

A Sample-Efficient Conditional Independence Test
in the Presence of Discretization
Boyang Sun 1 Yu Yao 2 Xinshuai Dong 3 Zongfang Liu 1 Tongliang Liu 2 1 Yumou Qiu 4 Kun Zhang 3 1
Abstract
In many real-world scenarios, interested variables
are often represented as discretized values due
to measurement limitations.
Applying Condi-
tional Independence (CI) tests directly to such
discretized data, however, can lead to incorrect
conclusions. To address this, recent advancements
have sought to infer the correct CI relationship
between the latent variables through binarizing
observed data. However, this process inevitably
results in a loss of information, which degrades
the test’s performance. Motivated by this, this
paper introduces a sample-efficient CI test that
does not rely on the binarization process. We
find that the independence relationships of latent
continuous variables can be established by ad-
dressing an over-identifying restriction problem
with Generalized Method of Moments (GMM).
Based on this insight, we derive an appropriate
test statistic and establish its asymptotic distribu-
tion correctly reflecting CI by leveraging node-
wise regression. Theoretical findings and Empir-
ical results across various datasets demonstrate
that the superiority and effectiveness of our pro-
posed test. Our code implementation is provided
in https://github.com/boyangaaaaa/DCT.
1. Introduction
Conditional independence tests for discrete variables are
fundamental in statistical analysis and widely applied across
various disciplines. Traditional methods including the chi-
squared test (F.R.S., 2009), the G-test (likelihood ratio test)
(McDonald, 2009), and measures based on conditional mu-
tual information (Kubkowski et al., 2021) are well estab-
lished and extensively used. However, a critical yet often
1Mohamed bin Zayed University of Artificial Intelligence
2Sydney AI Centre, The University of Sydney 3Carnegie Mellon
University 4Peking University. Correspondence to: Yumou Q...
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
ICML 2025

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Prasanna Mayilvahanan * 1 2 3 4 Thadd¨aus Wiedemer * 1 2 3 4 Sayak Mallick 1 4 Matthias Bethge 2 3 4
Wieland Brendel 1 2 3
Abstract
Scaling laws guide the development of large lan-
guage models (LLMs) by offering estimates for
the optimal balance of model size, tokens, and
compute. More recently, loss-to-loss scaling laws
that relate losses across pretraining datasets and
downstream tasks have emerged as a powerful
tool for understanding and improving LLM per-
formance and generalization. In this work, we
investigate which factors most strongly influence
loss-to-loss scaling. Our experiments reveal that
the pretraining data determines the scaling trend.
In contrast, model size, optimization hyperparam-
eters, tokenizer and even significant architectural
differences, such as between transformer-based
models like Llama and state-space models like
Mamba, generally have limited impact. Conse-
quently, practitioners should carefully curate pre-
training datasets for optimal downstream perfor-
mance, while architectures and other settings can
be freely optimized for training efficiency.
1. Introduction
Scaling laws have long guided Large Language Model
(LLM) pretraining, determining model and data size un-
der a fixed compute budget (Kaplan et al., 2020; Hoffmann
et al., 2022; Grattafiori et al., 2024). Typically, scaling laws
relate model performance, usually measured as training or
validation loss, to total compute measured in floating point
operations (FLOPs). FLOPs account for both parameter
count and the number of training tokens. While useful for
pretraining, scaling laws do not capture how well a model ul-
timately performs on downstream tasks (Gadre et al., 2024;
Schaeffer et al., 2024; Du et al., 2025). Consequently, mul-
tiple works have begun to investigate downstream scaling
*Equal contribution 1Max Planck Institute for Intelligent Sys-
tems 2ELLIS Institute T¨ubingen 3T¨ubingen AI Center 4University
...
Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
NeurIPS 2025

Brain-tuning Improves Generalizability and Efficiency
of Brain Alignment in Speech Models
Omer Moussa
Max Planck Institute for Software Systems
Saarbrücken, Germany
omoussa@mpi-sws.org
Mariya Toneva
Max Planck Institute for Software Systems
Saarbrücken, Germany
mtoneva@mpi-sws.org
Abstract
Pretrained language models are remarkably effective in aligning with human brain
responses elicited by natural language stimuli, positioning them as promising
model organisms for studying language processing in the brain. However, existing
approaches for both estimating and improving this brain alignment are participant-
dependent and highly affected by the amount of data available per participant,
hindering both generalization to new participants and population-level analyses.
In this work, we address these limitations by introducing a scalable, generalizable
brain-tuning method, in which we fine-tune pretrained speech language models
to jointly predict fMRI responses from multiple participants. We demonstrate
that the resulting brain-tuned models exhibit strong individual brain alignment
while generalizing across participants. Specifically, our method leads to 1) a
5-fold decrease in the amount of fMRI data needed to predict brain data from
new participants, 2) up to a 50% increase in the overall brain alignment, and 3)
strong generalization to new unseen datasets. Furthermore, this multi-participant
brain-tuning additionally improves downstream performance on semantic tasks,
suggesting that training using brain data from multiple participants leads to more
generalizable semantic representations. Taken together, these findings demon-
strate a bidirectional benefit between neuroscience and AI, helping bridge the
gap between the two fields. We make our code and models publicly available at
https://github.com/bridge-ai-neuro/multi-brain-tuning.
1
Introduction
Language models (LMs) substantially align with human brain responses elicited during natural
language processing, such as fu...
Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
ICML 2025

Sliding Puzzles Gym: A Scalable Benchmark for
State Representation in Visual Reinforcement Learning
Bryan L. M. de Oliveira 1 2 Luana G. B. Martins 1 Bruno Brand˜ao 1 2 Murilo L. da Luz 1 2
Telma W. de L. Soares 1 2 Luckeciano C. Melo 1 3
Abstract
Effective visual representation learning is crucial
for reinforcement learning (RL) agents to extract
task-relevant information from raw sensory in-
puts and generalize across diverse environments.
However, existing RL benchmarks lack the ability
to systematically evaluate representation learning
capabilities in isolation from other learning chal-
lenges. To address this gap, we introduce the Slid-
ing Puzzles Gym (SPGym), a novel benchmark
that transforms the classic 8-tile puzzle into a vi-
sual RL task with images drawn from arbitrarily
large datasets. SPGym’s key innovation lies in its
ability to precisely control representation learning
complexity through adjustable grid sizes and im-
age pools, while maintaining fixed environment
dynamics, observation, and action spaces. This
design enables researchers to isolate and scale the
visual representation challenge independently of
other learning components. Through extensive ex-
periments with model-free and model-based RL
algorithms, we uncover fundamental limitations
in current methods’ ability to handle visual diver-
sity. As we increase the pool of possible images,
all algorithms exhibit in- and out-of-distribution
performance degradation, with sophisticated rep-
resentation learning techniques often underper-
forming simpler approaches like data augmen-
tation. These findings highlight critical gaps in
visual representation learning for RL and estab-
lish SPGym as a valuable tool for driving progress
in robust, generalizable decision-making systems.
1Advanced Knowledge Center for Immersive Technologies
– AKCIT, Brazil 2Institute of Informatics, Federal University of
Goi´as, Goiˆania, Brazil 3OATML, University of Oxford, United
Kingdom. Correspondence to: Bryan L. M. d...
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
NeurIPS 2025

HybridNorm: Towards Stable and Efficient
Transformer Training via Hybrid Normalization
Zhijian Zhuo1, 2
Yutao Zeng2, †
Ya Wang2, †
Sijun Zhang2
Jian Yang3
Xiaoqing Li4
Xun Zhou2
Jinwen Ma1
1 School of Mathematical Sciences, Peking University
2 ByteDance Seed
3 Beihang University
4 Capital University of Economics and Business
Abstract
Transformers have become the de facto architecture for a wide range of machine
learning tasks, particularly in large language models (LLMs). Despite their remark-
able performance, many challenges remain in training deep transformer networks,
especially regarding the position of the layer normalization. While Pre-Norm
structures facilitate more stable training owing to their stronger identity path, they
often lead to suboptimal performance compared to Post-Norm. In this paper,
we propose HybridNorm, a simple yet effective hybrid normalization strategy
that integrates the advantages of both Pre-Norm and Post-Norm. Specifically,
HybridNorm employs QKV normalization within the attention mechanism and
Post-Norm in the feed-forward network (FFN) of each transformer block. We
provide both theoretical insights and empirical evidence to demonstrate that Hy-
bridNorm improves the gradient flow and the model robustness. Extensive ex-
periments on large-scale transformer models, including both dense and sparse
variants, show that HybridNorm consistently outperforms both Pre-Norm and
Post-Norm approaches across multiple benchmarks. These findings highlight the
potential of HybridNorm as a more stable and effective technique for improving
the training and performance of deep transformer models. Code is available at
https://github.com/BryceZhuo/HybridNorm.
1
Introduction
Transformers have become the backbone of large language models (LLMs) and a wide range of
machine learning applications. These architectures are capable of modeling long-range dependencies
through self-attention mechanisms, which have made them the preferred choice for a variety of
t...
Modular Duality in Deep Learning
ICML 2025

Modular Duality in Deep Learning
Jeremy Bernstein 1 Laker Newhouse 1
Abstract
An old idea in optimization theory says that
since the gradient is a dual vector it may not be
subtracted from the weights without first being
mapped to the primal space where the weights
reside. We take this idea seriously in this paper
and construct such a duality map for general neu-
ral networks. Our map, which we call modular
dualization, forms a unifying theoretical basis for
training algorithms that are a) fast and b) scalable.
Modular dualization involves first assigning op-
erator norms to layers based on the semantics of
each layer, and then using these layerwise norms
to recursively induce a duality map on the weight
space of the full neural architecture. We derive
GPU-friendly algorithms for dualizing Embed,
Linear and Conv2D layers—the latter two meth-
ods are based on a Newton-Schulz iteration. We
conclude with small experiments demonstrating
the speed, scalability and novel numerical prop-
erties of duality-based optimizers. Our methods
were used in the Muon optimizer, which recently
set speed records for training NanoGPT and was
scaled up to a 1.5 billion parameter transformer.
1
Introduction
This paper pursues a rigorous and first-principles theoretical
framework for designing neural network training algorithms.
We hope that such a framework will facilitate the design of
a next generation of fast and scalable optimizers that are
automatically tailored to different neural architectures.
While gradient descent is the workhorse of modern machine
learning, the most vanilla form of the algorithm does not,
in our view, pass a basic type check. For a gradient update
to type check, we insist that the gradient must be passed
through a duality map before being multiplied by a learning
rate and applied to the weights:
1MIT CSAIL, United States. Correspondence to: Jeremy Bern-
stein <jbernstein@mit.edu>, Laker Newhouse <lakern@mit.edu>.
Proceedings of the 42 nd International Conferenc...
Guarantees of a Preconditioned Subgradient Algorithm for Overparameterized Asymmetric Low-rank Matrix Recovery
ICML 2025

Guarantees of a Preconditioned Subgradient Algorithm for
Overparameterized Asymmetric Low-rank Matrix Recovery
Paris Giampouras 1 HanQin Cai 2 Ren´e Vidal 3
Abstract
In this paper, we focus on a matrix factorization-
based approach to recover low-rank asymmet-
ric matrices from corrupted measurements. We
propose an Overparameterized Preconditioned
Subgradient Algorithm (OPSA) and provide, for
the first time in the literature, linear convergence
rates independent of the rank of the sought asym-
metric matrix in the presence of gross corrup-
tions. Our work goes beyond existing results in
preconditioned-type approaches addressing their
current limitation, i.e., the lack of convergence
guarantees in the case of asymmetric matrices
of unknown rank. By applying our approach to
(robust) matrix sensing, we highlight its merits
when the measurement operator satisfies a mixed-
norm restricted isometry property. Lastly, we
present extensive numerical experiments that val-
idate our theoretical results and demonstrate the
effectiveness of our approach for different levels
of overparameterization and outlier corruptions.
1. Introduction
Low-rank matrix recovery has been a ubiquitous problem
showing up in numerous applications in the fields of sig-
nal/image processing, machine learning, and data science
(Recht et al., 2010; Chen et al., 2013; Davenport & Romberg,
2016; Cai et al., 2021a; Smith et al., 2024; Wang et al., 2024).
For instance, problems such as matrix sensing (Jain et al.,
2013; Wei et al., 2016; Park et al., 2017; Li et al., 2020),
matrix completion (Candes & Recht, 2008; Nie et al., 2012;
1Department of Computer Science, University of Warwick,
Coventry, England, UK 2Department of Statistics and Data Sci-
ence and Department of Computer Science, University of Central
Florida, Orlando, FL, USA 3Center for Innovation in Data Engi-
neering and Science (IDEAS), Departments of ESE and Radiology,
University of Pennsylvania, Philadelphia, PA, USA. Correspon-
dence to: Re...
LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch
ICLR 2025

Published as a conference paper at ICLR 2025
LLMOPT: LEARNING TO DEFINE AND SOLVE GEN-
ERAL OPTIMIZATION PROBLEMS FROM SCRATCH
Caigao Jiang♥∗,
Xiang Shu♣∗,
Hong Qian♣†,
Xingyu Lu♥†,
Jun Zhou♥,
Aimin Zhou♣,
Yang Yu♦
♣East China Normal University, China
♥Ant Group, China
♦Nanjing University, China
shux@stu.ecnu.edu.cn,
{hqian,amzhou}@cs.ecnu.edu.cn,
{caigao.jcg,sing.lxy,jun.zhoujun}@antgroup.com,
yuy@nju.edu.cn
ABSTRACT
Optimization problems are prevalent across various scenarios. Formulating and
then solving optimization problems described by natural language often requires
highly specialized human expertise, which could block the widespread applica-
tion of optimization-based decision making. To automate problem formulation
and solving, leveraging large language models (LLMs) has emerged as a potential
way. However, this kind of approach suffers from the issue of optimization gener-
alization. Namely, the accuracy of most current LLM-based methods and the gen-
erality of optimization problem types that they can model are still limited. In this
paper, we propose a unified learning-based framework called LLMOPT to boost
optimization generalization. Starting from the natural language descriptions of op-
timization problems and a pre-trained LLM, LLMOPT constructs the introduced
five-element formulation as a universal model for learning to define diverse op-
timization problem types. Then, LLMOPT employs the multi-instruction tuning
to enhance both problem formalization and solver code generation accuracy and
generality. After that, to prevent hallucinations in LLMs, such as sacrificing solv-
ing accuracy to avoid execution errors, the model alignment and self-correction
mechanism are adopted in LLMOPT. We evaluate the optimization generalization
ability of LLMOPT and compared methods across six real-world datasets cov-
ering roughly 20 fields such as health, environment, energy and manufacturing,
etc. Extensive experiment results show that LLMOPT is able to model vario...
GOttack: Universal Adversarial Attacks on Graph Neural Networks via Graph Orbits Learning
ICLR 2025

Published as a conference paper at ICLR 2025
GOTTACK: UNIVERSAL ADVERSARIAL ATTACKS ON
GRAPH NEURAL NETWORKS VIA GRAPH ORBITS
LEARNING
Zulfikar Alom1,
Tran Gia Bao Ngo1,
Murat Kantarcioglu2,
Cuneyt Gurcan Akcora3
1Department of Computer Science, University of Manitoba, Canada
2Department of Computer Science, Virginia Tech, USA
3AI Initiative - University of Central Florida, USA
md.alom@umanitoba.ca,
ngot1@myumanitoba.ca,
muratk@vt.edu
cuneyt.akcora@ucf.edu
ABSTRACT
Graph Neural Networks (GNNs) have demonstrated superior performance in node
classification tasks across diverse applications. However, their vulnerability to
adversarial attacks, where minor perturbations can mislead model predictions,
poses significant challenges. This study introduces GOttack, a novel adversarial
attack framework that exploits the topological structure of graphs to undermine the
integrity of GNN predictions systematically.
By defining a topology-aware method to manipulate graph orbits, our approach
generates adversarial modifications that are both subtle and effective, posing a
severe test to the robustness of GNNs. We evaluate the efficacy of GOttack
across multiple prominent GNN architectures using standard benchmark datasets.
Our results show that GOttack outperforms existing state-of-the-art adversarial
techniques and completes training in approximately 55% of the time required by
the fastest competing model, achieving the highest average misclassification rate
in 155 tasks. This work not only sheds light on the susceptibility of GNNs to
structured adversarial attacks but also shows that certain topological patterns may
play a significant role in the underlying robustness of the GNNs. Our Python
implementation is shared at https://github.com/cakcora/GOttack.
1
INTRODUCTION
Recent advances in Graph Neural Networks (GNNs) have brought significant progress in node classifi-
cation tasks, utilizing the power of graph topology and node features to generate insightful inferences
across va...
Progressive Tempering Sampler with Diffusion
ICML 2025

Progressive Tempering Sampler with Diffusion
Severi Rissanen * 1 RuiKang OuYang * 2 Jiajun He 2 Wenlin Chen 2 3 Markus Heinonen 1 Arno Solin 1
Jos´e Miguel Hern´andez-Lobato 2
Abstract
Recent research has focused on designing neu-
ral samplers that amortize the process of sam-
pling from unnormalized densities. However, de-
spite significant advancements, they still fall short
of the state-of-the-art MCMC approach, Parallel
Tempering (PT), when it comes to the efficiency
of target evaluations. On the other hand, unlike a
well-trained neural sampler, PT yields only depen-
dent samples and needs to be rerun—at consider-
able computational cost—whenever new samples
are required. To address these weaknesses, we
propose the Progressive Tempering Sampler with
Diffusion (PTSD), which trains diffusion models
sequentially across temperatures, leveraging the
advantages of PT to improve the training of neu-
ral samplers. We also introduce a novel method
to combine high-temperature diffusion models to
generate approximate lower-temperature samples,
which are minimally refined using MCMC and
used to train the next diffusion model. PTSD en-
ables efficient reuse of sample information across
temperature levels while generating well-mixed,
uncorrelated samples. Our method significantly
improves target evaluation efficiency, outperform-
ing diffusion-based neural samplers.
1. Introduction
Sampling from probability distributions is a fundamental
problem in many fields of science, including Bayesian infer-
ence (Gelman et al., 2013; Welling & Teh, 2011), statistical
physics (Von Toussaint, 2011) and molecular simulations
*Equal contribution
1Department of Computer Science,
Aalto University, Finland 2Department of Engineering, Univer-
sity of Cambridge, United Kingdom 3Department of Empir-
ical Inference, Max Planck Institute for Intelligent Systems,
T¨ubingen, Germany. Correspondence to: Severi Rissanen <sev-
eri.rissanen@aalto.fi>, RuiKang OuYang <ro352@cam.ac.uk>,
Jiajun He <jh2383@...
Generating Hypotheses of Dynamic Causal Graphs in Neuroscience: Leveraging Generative Factor Models of Observed Time Series
ICML 2025

Generating Hypotheses of Dynamic Causal Graphs in Neuroscience:
Leveraging Generative Factor Models of Observed Time Series
Zachary C. Brown 1 David Carlson 1 2
Abstract
The field of hypothesis generation promises to re-
duce costs in neuroscience by narrowing the range
of interventional studies needed to study various
phenomena. Existing machine learning methods
can generate scientific hypotheses from complex
datasets, but many approaches assume causal rela-
tionships are static over time, limiting their appli-
cability to systems with dynamic, state-dependent
behavior, such as the brain. While some tech-
niques attempt dynamic causal discovery through
factor models, they often restrict relationships to
linear patterns or impose other simplifying as-
sumptions. We propose a novel method that mod-
els dynamic graphs as a conditionally weighted
superposition of static graphs, where each static
graph can capture nonlinear relationships. This
approach enables the detection of complex, time-
varying interactions between variables beyond lin-
ear limitations. Our method improves f1-scores
of predicted dynamic causal patterns by roughly
22-28% on average over baselines in some of our
experiments, with some improvements reaching
well over 60%. A case study on real brain data
demonstrates our method’s ability to uncover re-
lationships linked to specific behavioral states,
offering valuable insights into neural dynamics.
1. Introduction
Causal discovery methods applied to brain dynamics hold
great promise for designing targeted interventions in var-
ious neurological diseases. A typical approach is to gen-
erate hypotheses about causal relationships between brain
1Department of Electrical and Computer Engineering, Duke
University, Durham, NC 15213, USA 2Departments of Civil and
Environmental Engineering; Biostatistics and Bioinformatics; and
Computer Science, Duke University, Durham, NC 15213, USA.
Correspondence to: Zachary C. Brown <zac.brown@duke.edu>,
David Carlson <dav...