dataset-review/dataset-review huggingface dataset:
Hardware and Software Platform Inference
ICML 2025

Hardware and Software Platform Inference
Cheng Zhang * 1 Hanna Foerster * 2 Robert D. Mullins 2 Yiren Zhao 1 Ilia Shumailov 3
Abstract
It is now a common business practice to buy ac-
cess to large language model (LLM) inference
rather than self-host, because of significant up-
front hardware infrastructure and energy costs.
However, as a buyer, there is no mechanism
to verify the authenticity of the advertised ser-
vice including the serving hardware platform, e.g.
that it is actually being served using an NVIDIA
H100. Furthermore, there are reports suggest-
ing that model providers may deliver models that
differ slightly from the advertised ones, often
to make them run on less expensive hardware.
That way, a client pays premium for a capable
model access on more expensive hardware, yet
ends up being served by a (potentially less capa-
ble) cheaper model on cheaper hardware. In this
paper we introduce hardware and software plat-
form inference (HSPI) – a method for identify-
ing the underlying GPU architecture and software
stack of a (black-box) machine learning model
solely based on its input-output behavior. Our
method leverages the inherent differences of var-
ious GPU architectures and compilers to distin-
guish between different GPU types and software
stacks. We evaluate HSPI against models served
on different real hardware and find that in a white-
box setting we can distinguish between differ-
ent GPUs with between 83.9% and 100% accu-
racy. Even in a black-box setting we achieve
results that are up to 3× higher than random
guess accuracy. Our code is available at https:
//github.com/ChengZhang-98/HSPI.
*Equal contribution
1Department of Computing, Impe-
rial College London, London, United Kingdom 2Department
of Computer Science and Technology, University of Cam-
bridge, Cambridge, United Kingdom 3Google DeepMind, Lon-
don, United Kingdom.
Correspondence to:
Cheng Zhang
<cheng.zhang122@imperial.ac.uk>.
Proceedings of the 42 nd International Conference on Mach...
RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
ICML 2025

RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for
Large Language Models
Quan Wei * 1 Chung-Yiu Yau * 2 Hoi-To Wai 2 Yang (Katie) Zhao 1 Dongyeop Kang 3 Youngsuk Park 4
Mingyi Hong 1
Abstract
Supervised fine-tuning is a standard method
for adapting pre-trained large language models
(LLMs) to downstream tasks. Quantization has
been recently studied as a post-training technique
for efficient LLM deployment. To obtain quan-
tized fine-tuned LLMs, conventional pipelines
would first fine-tune the pre-trained models, fol-
lowed by post-training quantization. This often
yields suboptimal performance as it fails to lever-
age the synergy between fine-tuning and quanti-
zation. To effectively realize low-bit quantization
of weights, activations and KV caches in LLMs,
we propose an algorithm named Rotated Straight-
Through-Estimator (RoSTE), which combines
quantization-aware supervised fine-tuning (QA-
SFT) with an adaptive rotation strategy that iden-
tifies an effective rotation configuration to reduce
activation outliers.
We provide theoretical in-
sights on RoSTE by analyzing its prediction error
when applied to an overparameterized least square
quantized training problem. Our findings reveal
that the prediction error is directly proportional to
the quantization error of the converged weights,
which can be effectively managed through an op-
timized rotation configuration. Experiments on
Pythia, Qwen and Llama models of different sizes
demonstrate the effectiveness of RoSTE. Com-
pared to existing post-SFT quantization baselines,
our method consistently achieves superior perfor-
mances across various tasks and different LLM
architectures. Our code is available at https:
//github.com/OptimAI-Lab/RoSTE.
*Equal contribution 1Department of Electrical and Computer
Engineering, University of Minnesota, USA. 2Department of Sys-
tems Engineering and Engineering Management, The Chinese Uni-
versity of Hong Kong, Hong Kong SAR of China. 3Department
of Compute...
FlashMoE: Fast Distributed MoE in a Single Kernel
NeurIPS 2025

FlashMoE: Fast Distributed MoE in a Single Kernel
Osayamen Jonathan Aimuyo
Cornell University
oja7@cornell.edu
Byungsoo Oh
Cornell University
bo239@cornell.edu
Rachee Singh
Cornell University
rs2293@cornell.edu
Abstract
The computational sparsity of Mixture-of-Experts (MoE) models enables sub-
linear growth in compute cost as model size increases, thus offering a scalable
path to training massive neural networks. However, existing implementations
suffer from low GPU utilization, significant latency overhead, and a fundamen-
tal inability to leverage task locality, primarily due to CPU-managed scheduling,
host-initiated communication, and frequent kernel launches. To overcome these
limitations, we develop FlashMoE, a fully GPU-resident MoE operator that fuses
expert computation and inter-GPU communication into a single persistent GPU ker-
nel. FlashMoE enables fine-grained pipelining of dispatch, compute, and combine
phases, eliminating launch overheads and reducing idle gaps. Unlike existing work,
FlashMoE obviates bulk-synchronous collectives for one-sided, device-initiated,
inter-GPU (R)DMA transfers, thus unlocking payload efficiency, where we elim-
inate bloated or redundant network payloads in sparsely activated layers. When
evaluated on an 8-H100 GPU node with MoE models having up to 128 experts
and 16K token sequences, FlashMoE achieves up to 9× higher GPU utilization,
6× lower latency, 5.7× higher throughput, and 4× better overlap efficiency com-
pared to state-of-the-art baselines—despite using FP32 while baselines use FP16.
FlashMoE shows that principled GPU kernel-hardware co-design is key to unlock-
ing the performance ceiling of large-scale distributed ML. We provide code at
https://github.com/osayamenja/FlashMoE.
FlashMoE
Comet
FasterMoE
Megatron
DeepEP
0
20
40
60
80
100
Average SM Util (%)
93.17%
42.31%
9.67%
59.11%
13.55%
Forward Pass | E = 64 | k = 2 | 2 A100s | 
 is better
(a) GPU SM Utilization
4K
8K
16K
Number of Tokens
0
10
20
30
40
Runtime (ms)...
Progressively Label Enhancement for Large Language Model Alignment
ICML 2025

Progressively Label Enhancement for Large Language Model Alignment
Biao Liu 1 2 Ning Xu 1 2 Xin Geng 1 2
Abstract
Large Language Models (LLM) alignment aims to
prevent models from producing content that mis-
aligns with human expectations, which can lead
to ethical and legal concerns. In the last few years,
Reinforcement Learning from Human Feedback
(RLHF) has been the most prominent method for
achieving alignment. Due to challenges in sta-
bility and scalability with RLHF stages, which
arise from the complex interactions between mul-
tiple models, researchers are exploring alternative
methods to achieve effects comparable to those
of RLHF. However, these methods often rely on
large high-quality datasets. Despite some meth-
ods considering the generation of additional data
to expand datasets, they often treat model training
and data generation as separate and static pro-
cesses, overlooking the fact that these processes
are highly interdependent, leading to inefficient
utilization of the generated data. To deal with
this problem, we propose PLE, i.e., Progressively
Label Enhancement for LLM Alignment, a frame-
work that dynamically adjusts the model’s train-
ing process based on the evolving quality of the
generated data. Specifically, we prompt the model
to generate responses for both the original query
and a set of carefully designed principle-guided
query, and then utilize a dynamic threshold to
determine the appropriate training approach for
both responses based on their corresponding re-
ward scores. Experimental results demonstrate the
effectiveness of PLE compared to existing LLM
alignment methods.
1School of Computer Science and Engineering, South-
east University, Nanjing, China.
E-mail: {liubiao01, xning,
xgeng}@seu.edu.cn. 2Key Laboratory of New Generation Artifi-
cial Intelligence Technology and Its Interdisciplinary Applications
(Southeast University), Ministry of Education, China. Correspon-
dence to: Ning Xu <xning@seu.edu.cn>.
Proceedings of the 42 n...
Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization
ICML 2025

Robust Offline Reinforcement Learning with Linearly Structured
f-Divergence Regularization
Cheng Tang * 1 Zhishuai Liu * 2 Pan Xu 2
Abstract
The Robust Regularized Markov Decision Pro-
cess (RRMDP) is proposed to learn policies ro-
bust to dynamics shifts by adding regularization
to the transition dynamics in the value function.
Existing methods mostly use unstructured reg-
ularization, potentially leading to conservative
policies under unrealistic transitions. To address
this limitation, we propose a novel framework,
the d-rectangular linear RRMDP (d-RRMDP),
which introduces latent structures into both tran-
sition kernels and regularization. We focus on
offline reinforcement learning, where an agent
learns policies from a precollected dataset in the
nominal environment. We develop the Robust
Regularized Pessimistic Value Iteration (R2PVI)
algorithm that employs linear function approxi-
mation for robust policy learning in d-RRMDPs
with f-divergence based regularization terms on
transition kernels. We provide instance-dependent
upper bounds on the suboptimality gap of R2PVI
policies, demonstrating that these bounds are in-
fluenced by how well the dataset covers state-
action spaces visited by the optimal robust policy
under robustly admissible transitions. We estab-
lish information-theoretic lower bounds to verify
that our algorithm is near-optimal. Finally, numer-
ical experiments validate that R2PVI learns robust
policies and exhibits superior computational effi-
ciency compared to baseline methods.
1. Introduction
Offline reinforcement learning (RL) (Levine et al., 2020)
facilitates policy learning from fixed datasets, eliminating
the need for direct interaction with the environment. When
*Equal contribution 1 University of Illinois Urbana-Champaign,
work was done when Cheng was in Tsinghua University 2Duke
University. Correspondence to: Pan Xu <pan.xu@duke.edu>.
Proceedings of the 42 st International Conference on Machine
Learning, Vancouver, Canada. PMLR 267...
Tensor Product Neural Networks for Functional ANOVA Model
ICML 2025

Tensor Product Neural Networks for Functional ANOVA Model
Seokhun Park 1 Insung Kong 2 Yongchan Choi 3 Chanmoo Park 1 4 Yongdai Kim 1
Abstract
Interpretability for machine learning models is
becoming more and more important as machine
learning models become more complex. The func-
tional ANOVA model, which decomposes a high-
dimensional function into a sum of lower dimen-
sional functions (commonly referred to as compo-
nents), is one of the most popular tools for inter-
pretable AI, and recently, various neural networks
have been developed for estimating each compo-
nent in the functional ANOVA model. However,
such neural networks are highly unstable when
estimating each component since the components
themselves are not uniquely defined. That is, there
are multiple functional ANOVA decompositions
for a given function. In this paper, we propose a
novel neural network which guarantees a unique
functional ANOVA decomposition and thus is
able to estimate each component stably. We call
our proposed neural network ANOVA Tensor Prod-
uct Neural Network (ANOVA-TPNN) since it is
motivated by the tensor product basis expansion.
Theoretically, we prove that ANOVA-TPNN can
approximate any smooth function well. Empiri-
cally, we show that ANOVA-TPNN provide much
more stable estimation of each component and
thus much more stable interpretation when train-
ing data and initial values of the model parameters
vary than existing neural networks do. Our source
code is released at https://github.com/
ParkSeokhun/ANOVA-TPNN
1. Introduction
Interpretability has become more important as artificial in-
telligence (AI) models have become more sophisticated and
complicated in recent years. Various methods such as LIME
1Department of Statistics, Seoul National University, Seoul, Re-
public of Korea 2Department of Applied Mathematics, University
of Twente, Enschede, Netherlands 3Toss Bank, Seoul, Republic of
Korea 4Samsung Electronics Co., Ltd, Republic of Korea. Corre-
spondence to: Yongdai ...
Sparse Autoencoders for Hypothesis Generation
ICML 2025

Sparse Autoencoders for Hypothesis Generation
Rajiv Movva * 1 Kenny Peng * 2 3 Nikhil Garg 3 Jon Kleinberg 2 Emma Pierson 1
Abstract
We describe HYPOTHESAES, a general method
to hypothesize interpretable relationships between
text data (e.g., headlines) and a target variable
(e.g., clicks). HYPOTHESAES has three steps:
(1) train a sparse autoencoder on text embeddings
to produce interpretable features describing the
data distribution, (2) select features that predict
the target variable, and (3) generate a natural lan-
guage interpretation of each feature (e.g., men-
tions being surprised or shocked) using an LLM.
Each interpretation serves as a hypothesis about
what predicts the target variable. Compared to
baselines, our method better identifies reference
hypotheses on synthetic datasets (at least +0.06
in F1) and produces more predictive hypotheses
on real datasets (∼twice as many significant find-
ings), despite requiring 1-2 orders of magnitude
less compute than recent LLM-based methods.
HYPOTHESAES also produces novel discoveries
on two well-studied tasks: explaining partisan dif-
ferences in Congressional speeches and identify-
ing drivers of engagement with online headlines.
1. Introduction
Large language models (LLMs) show promise as a tool for
hypothesis generation. Discovering relationships between
text data and a target variable is an important and funda-
mental task with diverse applications in economics, political
science, sociology, medicine, and business (Grimmer, 2010;
Rathje et al., 2021; Sun et al., 2022; Gentzkow & Shapiro,
2010; Monroe et al., 2009; Nelson, 2020; Ranard et al.,
2016; Ting et al., 2017). What features of a restaurant re-
view predict a low rating? What features of a social media
post predict whether it will go viral? What features of a
patient’s clinical notes predict if they will develop cancer?
*Equal contribution 1UC Berkeley 2Cornell University 3Cornell
Tech. Correspondence to: Rajiv Movva <rmovva@berkeley.edu>,
Kenny Peng <k...
A Machine Learning Approach to Duality in Statistical Physics
ICML 2025

A Machine Learning Approach to Duality in Statistical Physics
Prateek Gupta 1 Andrea E. V. Ferrari 2 3 Nabil Iqbal 4 5
Abstract
The notion of duality – that a given physical sys-
tem can have two different mathematical descrip-
tions – is a key idea in modern theoretical physics.
Establishing a duality in lattice statistical mechan-
ics models requires the construction of a dual
Hamiltonian and a map from the original to the
dual observables. By using neural networks to
parameterize these maps and introducing a loss
function that penalises the difference between cor-
relation functions in original and dual models, we
formulate the process of duality discovery as an
optimization problem. We numerically solve this
problem and show that our framework can redis-
cover the celebrated Kramers-Wannier duality for
the 2d Ising model, numerically reconstructing
the known mapping of temperatures. We further
investigate the 2d Ising model deformed by a pla-
quette coupling and find families of “approximate
duals”. We discuss future directions and prospects
for discovering new dualities within this frame-
work. 1.
1. Background
A key concept in physics is duality, i.e. the idea that the
same physical system can have two different mathematical
descriptions. Duality sits at the heart of modern theoretical
physics. In this work we seek to formalize the notion of
duality in statistical physics in a manner that allows modern
machine learning techniques to be used to systematically
search for dualities.
*Equal contribution 1Max Planck Institute for Human Develop-
ment, Berlin, Germany 2Deutsches Elektronen-Synchrotron DESY,
Germany 3School of Mathematics, The University of Edinburgh
4Department of Mathematical Sciences, Durham University, UK
5Amsterdam Machine Learning Laboratory, University of Ams-
terdam, Netherlands. Correspondence to: Prateek Gupta <con-
tact@pgupta.info>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Co...
ParFam -- (Neural Guided) Symbolic Regression via Continuous Global Optimization
ICLR 2025

Published as a conference paper at ICLR 2025
PARFAM—(NEURAL GUIDED) SYMBOLIC REGRES-
SION VIA CONTINUOUS GLOBAL OPTIMIZATION
Philipp Scholl, Katharina Bieker, Hillary Hauger & Gitta Kutyniok∗
Mathematical Institute
LMU Munich
Munich, Germany
scholl@math.lmu.de
ABSTRACT
The problem of symbolic regression (SR) arises in many different applications,
such as identifying physical laws or deriving mathematical equations describ-
ing the behavior of financial markets from given data. In this paper, we present
our new approach ParFam that utilizes parametric families of suitable symbolic
functions to translate the discrete symbolic regression problem into a continuous
one, resulting in a more straightforward setup compared to current state-of-the-
art methods. In combination with a global optimizer, this approach results in a
highly effective method to tackle the problem of SR. We theoretically analyze the
expressivity of ParFam and demonstrate its performance with extensive numerical
experiments based on the common SR benchmark suit SRBench, showing that we
achieve state-of-the-art results. Moreover, we present an extension incorporating
a pre-trained transformer network (DL-ParFam) to guide ParFam, accelerating the
optimization process by up to two magnitudes. Our code and results can be found
at https://github.com/Philipp238/parfam.
1
INTRODUCTION
Symbolic regression (SR) aims to discover concise and interpretable mathematical functions that
accurately model input-output relationships. This focus on simplicity is crucial for applications
requiring model analysis and trustworthiness, such as in physical or chemical sciences (Quade et al.,
2016; Angelis et al., 2023; Wang et al., 2019). SR finds broad application in diverse fields, including
ecosystem dynamics (Chen et al., 2019), solar power forecasting (Quade et al., 2016), financial
market analysis (Liu & Guo, 2023), materials science (He & Zhang, 2021), and robotics (Oplatkova
& Zelinka, 2007). The growing body of SR r...
Equivariant Neural Tangent Kernels
ICML 2025

Equivariant Neural Tangent Kernels
Philipp Misof 1 Pan Kessel 2 Jan E. Gerken 1
Abstract
Little is known about the training dynamics of
equivariant neural networks, in particular how
it compares to data augmented training of their
non-equivariant counterparts. Recently, neural
tangent kernels (NTKs) have emerged as a pow-
erful tool to analytically study the training dy-
namics of wide neural networks. In this work,
we take an important step towards a theoretical
understanding of training dynamics of equivari-
ant models by deriving neural tangent kernels for
a broad class of equivariant architectures based
on group convolutions. As a demonstration of
the capabilities of our framework, we show an in-
teresting relationship between data augmentation
and group convolutional networks. Specifically,
we prove that they share the same expected pre-
diction over initializations at all training times
and even off the data manifold. In this sense, they
have the same training dynamics. We demonstrate
in numerical experiments that this still holds ap-
proximately for finite-width ensembles. By imple-
menting equivariant NTKs for roto-translations
in the plane (G = Cn ⋉R2) and 3d rotations
(G = SO(3)), we show that equivariant NTKs
outperform their non-equivariant counterparts as
kernel predictors for histological image classifica-
tion and quantum mechanical property prediction.
1. Introduction
Equivariant neural networks (Weiler et al., 2023; Gerken
et al., 2023) are widely used in many applications of great
practical importance, for example in medical image analysis
in two and three dimensions (Bekkers et al., 2018; Winkels
& Cohen, 2019; M¨uller et al., 2021; Pang et al., 2023) and in
quantum chemistry (Duval et al., 2023; Batzner et al., 2022;
1Department of Mathematical Sciences, Chalmers University of
Technology and the University of Gothenburg, SE-412 96 Gothen-
burg, Sweden 2Prescient Design, Genentech Roche, Basel, Switzer-
land. Correspondence to: Jan Gerken <gerken@...
Uncertainty Herding: One Active Learning Method for All Label Budgets
ICLR 2025

Published as a conference paper at ICLR 2025
UNCERTAINTY HERDING:
ONE ACTIVE LEARNING
METHOD FOR ALL LABEL BUDGETS
Wonho Bae
University of British Columbia & Borealis AI
whbae@cs.ubc.ca
Danica J. Sutherland∗
University of British Columbia & Amii
dsuth@cs.ubc.ca
Gabriel L. Oliveira∗
Borealis AI
gabriel.oliveira@borealisai.com
ABSTRACT
Most active learning research has focused on methods which perform well when
many labels are available, but can be dramatically worse than random selection
when label budgets are small. Other methods have focused on the low-budget
regime, but do poorly as label budgets increase. As the line between “low” and
“high” budgets varies by problem, this is a serious issue in practice. We propose
uncertainty coverage, an objective which generalizes a variety of low- and high-
budget objectives, as well as natural, hyperparameter-light methods to smoothly
interpolate between low- and high-budget regimes. We call greedy optimization of
the estimate Uncertainty Herding; this simple method is computationally fast, and
we prove that it nearly optimizes the distribution-level coverage. In experimental
validation across a variety of active learning tasks, our proposal matches or beats
state-of-the-art performance in essentially all cases; it is the only method of which
we are aware that reliably works well in both low- and high-budget settings.
1
INTRODUCTION
In active learning, rather than being provided a dataset of input-output pairs as in passive learning,
a model strategically requests annotations for specific unlabeled inputs. The aim is to learn a good
model while minimizing the number of required output annotations. This procedure is generally
iterative: a model is initially trained on a small, labeled dataset, then selects the most “informative”
data points from an unlabeled pool to annotate. This is particularly useful when labeling is expensive
or time-consuming. For example, manual annotations of medical imaging by radiologists or pathol-
...