dataset-review/dataset-review huggingface dataset:
Connecting Neural Models Latent Geometries with Relative Geodesic Representations
NeurIPS 2025

Connecting Neural Models Latent Geometries with
Relative Geodesic Representations
Hanlin Yu1
University of Helsinki
Berfin Inal
University of Amsterdam
Georgios Arvanitidis
DTU
Søren Hauberg
DTU
Francesco Locatello
IST Austria
Marco Fumero1
IST Austria
Abstract
Neural models learn representations of high-dimensional data on low-dimensional
manifolds. Multiple factors, including stochasticities in the training process, model
architectures, and additional inductive biases, may induce different representations,
even when learning the same task on the same data. However, it has recently been
shown that when a latent structure is shared between distinct latent spaces, relative
distances between representations can be preserved, up to distortions. Building
on this idea, we demonstrate that exploiting the differential-geometric structure of
latent spaces of neural models, it is possible to capture precisely the transformations
between representational spaces trained on similar data distributions. Specifically,
we assume that distinct neural models parametrize approximately the same un-
derlying manifold, and introduce a representation based on the pullback metric
that captures the intrinsic structure of the latent space, while scaling efficiently
to large models. We validate experimentally our method on model stitching and
retrieval tasks, covering autoencoders and vision foundation discriminative models,
across diverse architectures, datasets, pretraining schemes and modalities. Code is
available at https://github.com/marc0git/RelativeGeodesics.
1
Introduction
Figure 1:
Neural models trained on similar
data learn parametrizations of the same mani-
fold. NNs learn parametrizations (D1, D2) of the
same underlying manifold Y up to isometries T.
Pulling back the metric from Y makes relative
geodesic representations invariant to transforma-
tions T between latent spaces Z1 and Z2.
Neural models learn meaningful representations
of high-dimensional data generalizing to many
task...
Potemkin Understanding in Large Language Models
ICML 2025

Potemkin Understanding in Large Language Models
Marina Mancoridis 1 Bec Weeks 2 Keyon Vafa 3 Sendhil Mullainathan 4
Abstract
Large language models (LLMs) are regularly eval-
uated using benchmark datasets. But what justi-
fies making inferences about an LLM’s capabili-
ties based on its answers to a curated set of ques-
tions? This paper first introduces a formal frame-
work to address this question. The key is to note
that the benchmarks used to test LLMs—such
as AP exams—are also those used to test people.
However, this raises an implication: such bench-
marks are only valid tests if LLMs misunderstand
concepts in ways that mirror human misunder-
standings. Otherwise, success on benchmarks
only demonstrates potemkin understanding: the
illusion of understanding driven by answers irrec-
oncilable with how any human would interpret
a concept. We present two procedures for quan-
tifying the existence of potemkins: one using a
specially designed benchmark in three domains,
the other using a general procedure that provides
a lower-bound on their prevalence. We find that
potemkins are ubiquitous across models, tasks,
and domains. We also find that these failures re-
flect not just incorrect understanding, but deeper
internal incoherence in concept representations.
1. Introduction
There has been a marked change in how we interpret ma-
chine learning evaluations. Today, large language models
(LLMs) are evaluated on benchmark datasets: curated ques-
tions with rubrics for grading. Success on these questions
is treated as evidence of broader conceptual understanding.
Previously, while benchmarks were also used to evaluate
supervised learning models, success was interpreted more
narrowly. A pathology classifier that performs well on X-
ray classification is not credited with an understanding of
1Massachusetts Institute of Technology 2University of Chicago
3Harvard University 4Massachusetts Institute of Technology. Cor-
respondence to: Marina Mancoridis <marinam@mit.edu>.
Proc...
Latent Variable Causal Discovery under Selection Bias
ICML 2025

Latent Variable Causal Discovery under Selection Bias
Haoyue Dai 1 2 Yiwen Qiu 1 Ignavier Ng 1 Xinshuai Dong 1 Peter Spirtes 1 Kun Zhang 1 2
Abstract
Addressing selection bias in latent variable causal
discovery is important yet underexplored, largely
due to a lack of suitable statistical tools: While
various tools beyond basic conditional indepen-
dencies have been developed to handle latent vari-
ables, none have been adapted for selection bias.
We make an attempt by studying rank constraints,
which, as a generalization to conditional inde-
pendence constraints, exploits the ranks of co-
variance submatrices in linear Gaussian models.
We show that although selection can significantly
complicate the joint distribution, interestingly, the
ranks in the biased covariance matrices still pre-
serve meaningful information about both causal
structures and selection mechanisms. We provide
a graph-theoretic characterization of such rank
constraints. Using this tool, we demonstrate that
the one-factor model, a classical latent variable
model, can be identified under selection bias. Sim-
ulations and real-world experiments confirm the
effectiveness of using our rank constraints.
1
Introduction
At the core of understanding complex systems lies causal
discovery, the identification of causal relations from obser-
vational data (Spirtes et al., 2000; Pearl, 2009). In many
real-world scenarios, the variables of interest are latent
constructs that cannot be directly observed or quantized,
while the observed variables serve merely as indirect
measurements. For instance, in psychological or political-
economic surveys, measured responses serve as proxies for
latent personality traits or political orientations. Recovering
the causal structure among these latent variables—referred
to as latent variable causal discovery—is essential for un-
derstanding and reasoning, yet remains a challenging task.
Furthermore, a typical assumption in causal discovery,
1Carnegie Mellon University 2Moham...
Counterfactual Realizability
ICLR 2025

Published as a conference paper at ICLR 2025
COUNTERFACTUAL REALIZABILITY
Arvind Raghavan and Elias Bareinboim
Causal Artificial Intelligence Lab
Columbia University
{ar, eb}@cs.columbia.edu
ABSTRACT
It is commonly believed that, in a real-world environment, samples can only be
drawn from observational and interventional distributions, corresponding to Layers
1 and 2 of the Pearl Causal Hierarchy. Layer 3, representing counterfactual
distributions, is believed to be inaccessible by definition. However, Bareinboim,
Forney, and Pearl (2015) introduced a procedure that allows an agent to sample
directly from a counterfactual distribution, leaving open the question of what other
counterfactual quantities can be estimated directly via physical experimentation.
We resolve this by introducing a formal definition of realizability, the ability to draw
samples from a distribution, and then developing a complete algorithm to determine
whether an arbitrary counterfactual distribution is realizable given fundamental
physical constraints, such as the inability to go back in time and subject the same
unit to a different experimental condition. We illustrate the implications of this
new framework for counterfactual data collection using motivating examples from
causal fairness and causal reinforcement learning. While the baseline approach
in these motivating settings typically follows an interventional or observational
strategy, we show that a counterfactual strategy provably dominates both.
1
INTRODUCTION
The Pearl Causal Hierarchy, or PCH, is an important recent milestone in our understanding of
causality (Pearl & Mackenzie, 2018; Bareinboim et al., 2022). The three layers of the PCH represent
the distinct regimes of seeing, doing, and imagining, with regard to an environment. Consider
an environment involving a decision variable X and an outcome Y . Layer 1 (L1) represents
observational distributions, such as P(Y | x). Layer 2 (L2) represents interventional distributions,
such a...
Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts
ICML 2025

Feynman-Kac Correctors in Diffusion:
Annealing, Guidance, and Product of Experts
Marta Skreta * 1 2 Tara Akhound-Sadegh * 3 4 Viktor Ohanesian * 5 Roberto Bondesan 5 Alán Aspuru-Guzik 1 2
Arnaud Doucet 6 Rob Brekelmans 2 Alexander Tong † 7 4 8 Kirill Neklyudov † 7 4 9
Abstract
While score-based generative models are the
model of choice across diverse domains, there are
limited tools available for controlling inference-
time behavior in a principled manner, e.g. for
composing multiple pretrained models. Exist-
ing classifier-free guidance methods use a sim-
ple heuristic to mix conditional and uncondi-
tional scores to approximately sample from con-
ditional distributions. However, such methods
do not approximate the intermediate distribu-
tions, necessitating additional ‘corrector’ steps.
In this work, we provide an efficient and prin-
cipled method for sampling from a sequence of
annealed, geometric-averaged, or product distri-
butions derived from pretrained score-based mod-
els. We derive a weighted simulation scheme
which we call FEYNMAN-KAC CORRECTORS
(FKCs) based on the celebrated Feynman-Kac
formula by carefully accounting for terms in the
appropriate partial differential equations (PDEs).
To simulate these PDEs, we propose Sequential
Monte Carlo (SMC) resampling algorithms that
leverage inference-time scaling to improve sam-
pling quality. We empirically demonstrate the util-
ity of our methods by proposing amortized sam-
pling via inference-time temperature annealing,
improving multi-objective molecule generation
using pretrained models, and improving classifier-
free guidance for text-to-image generation. Our
code is available at https://github.com/
martaskrt/fkc-diffusion.
*Equal contribution ,†Equal senior-authorship 1University of
Toronto 2Vector Institute 3McGill University 4Mila - Quebec AI In-
stitute 5Imperial College London 6Google DeepMind 7Université
de Montréal 8Current AT affiliation: Duke University 9Institut
Courtois.
Correspondence to:
AT <a...
Time-o1: Time-Series Forecasting Needs Transformed Label Alignment
NeurIPS 2025

Time-o1: Time-Series Forecasting Needs
Transformed Label Alignment
Hao Wang1∗
Licheng Pan1∗
Zhichao Chen2
Xu Chen3†
Qingyang Dai4
Lei Wang3
Haoxuan Li5†
Zhouchen Lin2,6,7†
1Xiaohongshu Inc.
2State Key Lab of General AI, School of Intelligence Science and Technology, Peking University
3Gaoling School of Artificial Intelligence, Renmin University of China
4Department of Control Science and Engineering, Zhejiang University
5Center for Data Science, Peking University
6Institute for Artificial Intelligence, Peking University
7Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
Abstract
Training time-series forecast models presents unique challenges in designing ef-
fective learning objectives. Existing methods predominantly utilize the temporal
mean squared error, which faces two critical challenges: (1) label autocorrelation,
which leads to bias from the label sequence likelihood; (2) excessive amount of
tasks, which increases with the forecast horizon and complicates optimization.
To address these challenges, we propose Time-o1, a transformation-augmented
learning objective for training time-series forecasting models. The central idea is
to transform the label sequence into decorrelated components with discriminated
significance. Models are then trained to align the most significant components,
thereby effectively mitigating label autocorrelation and reducing task amount.
Extensive experiments demonstrate that Time-o1 achieves state-of-the-art per-
formance and is compatible with various forecast models. Code is available at
https://github.com/Master-PLC/Time-o1.
1
Introduction
Time-series forecasting involves predicting future data from historical observations [60, 23] and
has been applied across diverse domains, such as air quality prediction in meteorology [28], user
behavior analysis in e-commerce [3], and process monitoring in manufacturing [47, 50]. To build
effective forecast models, there are two questions that warrant investigation: (1) How to design a
ne...
Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Space
ICML 2025

Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Space
Max van Spengler 1 Pascal Mettes 1
Abstract
Embedding tree-like data, from hierarchies to on-
tologies and taxonomies, forms a well-studied
problem for representing knowledge across many
domains. Hyperbolic geometry provides a natu-
ral solution for embedding trees, with vastly su-
perior performance over Euclidean embeddings.
Recent literature has shown that hyperbolic tree
embeddings can even be placed on top of neu-
ral networks for hierarchical knowledge integra-
tion in deep learning settings. For all applica-
tions, a faithful embedding of trees is needed,
with combinatorial constructions emerging as the
most effective direction. This paper identifies and
solves two key limitations of existing works. First,
the combinatorial construction hinges on finding
highly separated points on a hypersphere, a no-
toriously difficult problem. Current approaches
achieve poor separation, degrading the quality of
the corresponding hyperbolic embedding. We pro-
pose highly separated Delaunay tree embeddings
(HS-DTE), which integrates angular separation
in a generalized formulation of Delaunay embed-
dings, leading to lower embedding distortion. Sec-
ond, low-distortion requires additional precision.
The current approach for increasing precision is
to use multiple precision arithmetic, which ren-
ders the embeddings useless on GPUs in deep
learning settings. We reformulate the combinato-
rial construction using floating point expansion
arithmetic, leading to superior embedding quality
while retaining utility on accelerated hardware.
1. Introduction
Tree-like structures such as hierarchies are key for knowl-
edge representation, from biological taxonomies (Padial
et al., 2010) and phylogenetics (Kapli et al., 2020) to natu-
1VIS Lab, University of Amsterdam, The Netherlands. Corre-
spondence to: Max van Spengler <m.w.f.vanspengler@uva.nl>.
Proceedings of the 42 nd International Conference on Machine
Learning,...
A geometric framework for momentum-based optimizers for low-rank training
NeurIPS 2025

A geometric framework for momentum-based
optimizers for low-rank training
Steffen Schotthöfer∗Timon Klein† and Jonas Kusch‡
Abstract
Low-rank pre-training and finetuning have recently emerged as promising tech-
niques for reducing the computational and storage costs of large neural networks.
Training low-rank parameterizations typically relies on conventional optimizers
such as heavy ball momentum methods or Adam. In this work, we identify and
analyze potential difficulties that these training methods encounter when used to
train low-rank parameterizations of weights. In particular, we show that classical
momentum methods can struggle to converge to a local optimum due to the ge-
ometry of the underlying optimization landscape. To address this, we introduce
novel training strategies that combine dynamical low-rank approximation with
momentum-based optimization, explicitly accounting for the intrinsic geometry
of the parameter space. We validate our methods through numerical experiments,
demonstrating stronger validation metrics at given parameter budgets.
1
Introduction
Deep learning models have achieved remarkable success across natural language processing and
computer vision tasks, but their deployment remains computationally expensive due to the large
number of trainable parameters. To address this, parameter-efficient strategies have been developed
to reduce memory and compute requirements during training. Common approaches include sparsifi-
cation [9, 22, 12], quantization [37, 6], and layer factorization. The latter has gained considerable
attention for pre-training [36, 16, 28, 29, 41] and especially finetuning [15, 34, 42, 10, 43, 20, 27].
Layer factorization represents weights (or adapters) as low-rank matrices, allowing only the low-rank
factors to be trained. This significantly reduces both memory usage and computational cost.
In the class of low-rank layer factorizations, one of the most popular methods is LoRA [15], which
applied typical optimizers such...
Influence-Guided Diffusion for Dataset Distillation
ICLR 2025

Published as a conference paper at ICLR 2025
INFLUENCE-GUIDED DIFFUSION FOR DATASET DISTIL-
LATION
Mingyang Chen1,2, Jiawei Du3, Bo Huang1,2, Yi Wang4, Xiaobo Zhang5, Wei Wang1,2
1The Hong Kong University of Science and Technology (Guangzhou)
2The Hong Kong University of Science and Technology
3CFAR, A*STAR, Singapore
4Dongguan University of Technology
5Southwest Jiaotong University
mchenbt@connect.ust.hk,weiwcs@ust.hk
ABSTRACT
Dataset distillation aims to streamline the training process by creating a compact
yet effective dataset for a much larger original dataset. However, existing methods
often struggle with distilling large, high-resolution datasets due to prohibitive
resource costs and limited performance, primarily stemming from sample-wise
optimizations in the pixel space. Motivated by the remarkable capabilities of
diffusion generative models in learning target dataset distributions and controllably
sampling high-quality data tailored to user needs, we propose framing dataset
distillation as a controlled diffusion generation task aimed at generating data
specifically tailored for effective training purposes. By establishing a correlation
between the overarching objective of dataset distillation and the trajectory influence
function, we introduce the Influence-Guided Diffusion (IGD) sampling framework
to generate training-effective data without the need to retrain diffusion models. An
efficient guided function is designed by leveraging the trajectory influence function
as an indicator to steer diffusions to produce data with influence promotion and
diversity enhancement. Extensive experiments show that the training performance
of distilled datasets generated by diffusions can be significantly improved by
integrating with our IGD method and achieving state-of-the-art performance in
distilling ImageNet datasets. Particularly, an exceptional result is achieved on
the ImageNet-1K, reaching 60.3% at IPC=50. Our code is available at https:
//github.com/mchen725/DD_...
A Reasoning-Based Approach to Cryptic Crossword Clue Solving
ICML 2025

A Reasoning-Based Approach to Cryptic Crossword Clue Solving
Martin Andrews 1 Sam Witteveen 1
Abstract
Cryptic crossword clues are challenging language
tasks for which new test sets are released daily by
major newspapers on a global basis. Each cryptic
clue contains both the definition of the answer to
be placed in the crossword grid (in common with
regular crosswords), and ‘wordplay’ that proves
that the answer is correct (i.e. a human solver
can be confident that an answer is correct without
needing crossing words as confirmation). This
work describes an LLM-based reasoning system
built from open-licensed components that solves
cryptic clues by (i) hypothesising answers; (ii)
proposing wordplay explanations; and (iii) using
a verifier system that operates on codified reason-
ing steps. Overall, this system establishes a new
state-of-the-art performance on the challenging
Cryptonite dataset of clues from The Times and
The Telegraph newspapers in the UK. Because
each proved solution is expressed in Python, inter-
pretable wordplay reasoning for proven answers
is available for inspection.
1. Introduction
There has been significant work in reasoning in the fields
of mathematics (Jiang et al., 2023; Yang et al., 2023; Trinh
et al., 2024) and code generation (Ni et al., 2023; Ridnik
et al., 2024) which benefit from having strong verifiers to
validate their answers. This work tackles the relatively
under-studied reasoning task of cryptic crossword solving,
which has the following qualities:
• Thousands of people find Cryptic Crosswords a satisfy-
ing intellectual challenge on a daily basis. Solving these
puzzles requires understanding multi-layered language
constructs, blending logic, wordplay, and contextual nu-
ance. This provides a unique challenge for evaluating and
improving LLMs’ capabilities in NLU and reasoning.
1Red Dragon AI, Singapore. Correspondence to: Martin An-
drews <martin@reddragon.ai>.
Proceedings of the 42 nd International Conference on Machine
Learni...
CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking
ICLR 2025

Published as a conference paper at ICLR 2025
CORNSTACK: HIGH-QUALITY CONTRASTIVE DATA
FOR BETTER CODE RETRIEVAL AND RERANKING
Tarun Suresh1,3∗†
Revanth Gangi Reddy1∗
Yifei Xu1,2
Zach Nussbaum3
Andriy Mulyar3
Brandon Duderstadt3
Heng Ji1
1University of Illinois Urbana-Champaign
2Lapis Labs
3Nomic AI
{tsuresh3,revanth3,yifeix5,hengji}@illinois.edu
ABSTRACT
Effective code retrieval plays a crucial role in advancing code generation, bug fix-
ing, and software maintenance, particularly as software systems increase in com-
plexity. While current code embedding models have demonstrated promise in
retrieving code snippets for small-scale, well-defined tasks, they often underper-
form in more demanding real-world applications such as bug localization within
GitHub repositories. We hypothesize that a key issue is their reliance on noisy
and inconsistent datasets for training, which impedes their ability to generalize
to more complex retrieval scenarios. To address these limitations, we introduce
CORNSTACK1, a large-scale, high-quality contrastive training dataset for code
that spans multiple programming languages. This dataset is curated using consis-
tency filtering to eliminate noisy positives and is further enriched with mined hard
negatives, thereby facilitating more effective learning. We demonstrate that con-
trastive training of embedding models using CORNSTACK leads to state-of-the-
art performance across a variety of code retrieval tasks. Furthermore, the dataset
can be leveraged for training code reranking models, a largely underexplored area
compared to text reranking. Our finetuned code reranking model significantly im-
proves the ranking quality over the retrieved results. Finally, by employing our
code retriever and reranker together, we demonstrate significant improvements in
function localization for GitHub issues, an important component of real-world
software development.
1
INTRODUCTION
The rapid advancement of software development has led to an increased rel...