AIS-Bench/AIS-Bench huggingface dataset:
Published as a conference paper at ICLR 2025
COUNTERFACTUAL REALIZABILITY
Arvind Raghavan and Elias Bareinboim
Causal Artificial Intelligence Lab
Columbia University
{ar, eb}@cs.columbia.edu
ABSTRACT
It is commonly believed that, in a real-world environment, samples can only be
drawn from observational and interventional distributions, corresponding to Layers
1 and 2 of the Pearl Causal Hierarchy. Layer 3, representing counterfactual
distributions, is believed to be inaccessible by definition. However, Bareinboim,
Forney, and Pearl (2015) introduced a procedure that allows an agent to sample
directly from a counterfactual distribution, leaving open the question of what other
counterfactual quantities can be estimated directly via physical experimentation.
We resolve this by introducing a formal definition of realizability, the ability to draw
samples from a distribution, and then developing a complete algorithm to determine
whether an arbitrary counterfactual distribution is realizable given fundamental
physical constraints, such as the inability to go back in time and subject the same
unit to a different experimental condition. We illustrate the implications of this
new framework for counterfactual data collection using motivating examples from
causal fairness and causal reinforcement learning. While the baseline approach
in these motivating settings typically follows an interventional or observational
strategy, we show that a counterfactual strategy provably dominates both.
1
INTRODUCTION
The Pearl Causal Hierarchy, or PCH, is an important recent milestone in our understanding of
causality (Pearl & Mackenzie, 2018; Bareinboim et al., 2022). The three layers of the PCH represent
the distinct regimes of seeing, doing, and imagining, with regard to an environment. Consider
an environment involving a decision variable X and an outcome Y . Layer 1 (L1) represents
observational distributions, such as P(Y | x). Layer 2 (L2) represents interventional distributions,
such a...
Feynman-Kac Correctors in Diffusion:
Annealing, Guidance, and Product of Experts
Marta Skreta * 1 2 Tara Akhound-Sadegh * 3 4 Viktor Ohanesian * 5 Roberto Bondesan 5 Alán Aspuru-Guzik 1 2
Arnaud Doucet 6 Rob Brekelmans 2 Alexander Tong † 7 4 8 Kirill Neklyudov † 7 4 9
Abstract
While score-based generative models are the
model of choice across diverse domains, there are
limited tools available for controlling inference-
time behavior in a principled manner, e.g. for
composing multiple pretrained models. Exist-
ing classifier-free guidance methods use a sim-
ple heuristic to mix conditional and uncondi-
tional scores to approximately sample from con-
ditional distributions. However, such methods
do not approximate the intermediate distribu-
tions, necessitating additional ‘corrector’ steps.
In this work, we provide an efficient and prin-
cipled method for sampling from a sequence of
annealed, geometric-averaged, or product distri-
butions derived from pretrained score-based mod-
els. We derive a weighted simulation scheme
which we call FEYNMAN-KAC CORRECTORS
(FKCs) based on the celebrated Feynman-Kac
formula by carefully accounting for terms in the
appropriate partial differential equations (PDEs).
To simulate these PDEs, we propose Sequential
Monte Carlo (SMC) resampling algorithms that
leverage inference-time scaling to improve sam-
pling quality. We empirically demonstrate the util-
ity of our methods by proposing amortized sam-
pling via inference-time temperature annealing,
improving multi-objective molecule generation
using pretrained models, and improving classifier-
free guidance for text-to-image generation. Our
code is available at https://github.com/
martaskrt/fkc-diffusion.
*Equal contribution ,†Equal senior-authorship 1University of
Toronto 2Vector Institute 3McGill University 4Mila - Quebec AI In-
stitute 5Imperial College London 6Google DeepMind 7Université
de Montréal 8Current AT affiliation: Duke University 9Institut
Courtois.
Correspondence to:
AT <a...
Time-o1: Time-Series Forecasting Needs
Transformed Label Alignment
Hao Wang1∗
Licheng Pan1∗
Zhichao Chen2
Xu Chen3†
Qingyang Dai4
Lei Wang3
Haoxuan Li5†
Zhouchen Lin2,6,7†
1Xiaohongshu Inc.
2State Key Lab of General AI, School of Intelligence Science and Technology, Peking University
3Gaoling School of Artificial Intelligence, Renmin University of China
4Department of Control Science and Engineering, Zhejiang University
5Center for Data Science, Peking University
6Institute for Artificial Intelligence, Peking University
7Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
Abstract
Training time-series forecast models presents unique challenges in designing ef-
fective learning objectives. Existing methods predominantly utilize the temporal
mean squared error, which faces two critical challenges: (1) label autocorrelation,
which leads to bias from the label sequence likelihood; (2) excessive amount of
tasks, which increases with the forecast horizon and complicates optimization.
To address these challenges, we propose Time-o1, a transformation-augmented
learning objective for training time-series forecasting models. The central idea is
to transform the label sequence into decorrelated components with discriminated
significance. Models are then trained to align the most significant components,
thereby effectively mitigating label autocorrelation and reducing task amount.
Extensive experiments demonstrate that Time-o1 achieves state-of-the-art per-
formance and is compatible with various forecast models. Code is available at
https://github.com/Master-PLC/Time-o1.
1
Introduction
Time-series forecasting involves predicting future data from historical observations [60, 23] and
has been applied across diverse domains, such as air quality prediction in meteorology [28], user
behavior analysis in e-commerce [3], and process monitoring in manufacturing [47, 50]. To build
effective forecast models, there are two questions that warrant investigation: (1) How to design a
ne...
Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Space
Max van Spengler 1 Pascal Mettes 1
Abstract
Embedding tree-like data, from hierarchies to on-
tologies and taxonomies, forms a well-studied
problem for representing knowledge across many
domains. Hyperbolic geometry provides a natu-
ral solution for embedding trees, with vastly su-
perior performance over Euclidean embeddings.
Recent literature has shown that hyperbolic tree
embeddings can even be placed on top of neu-
ral networks for hierarchical knowledge integra-
tion in deep learning settings. For all applica-
tions, a faithful embedding of trees is needed,
with combinatorial constructions emerging as the
most effective direction. This paper identifies and
solves two key limitations of existing works. First,
the combinatorial construction hinges on finding
highly separated points on a hypersphere, a no-
toriously difficult problem. Current approaches
achieve poor separation, degrading the quality of
the corresponding hyperbolic embedding. We pro-
pose highly separated Delaunay tree embeddings
(HS-DTE), which integrates angular separation
in a generalized formulation of Delaunay embed-
dings, leading to lower embedding distortion. Sec-
ond, low-distortion requires additional precision.
The current approach for increasing precision is
to use multiple precision arithmetic, which ren-
ders the embeddings useless on GPUs in deep
learning settings. We reformulate the combinato-
rial construction using floating point expansion
arithmetic, leading to superior embedding quality
while retaining utility on accelerated hardware.
1. Introduction
Tree-like structures such as hierarchies are key for knowl-
edge representation, from biological taxonomies (Padial
et al., 2010) and phylogenetics (Kapli et al., 2020) to natu-
1VIS Lab, University of Amsterdam, The Netherlands. Corre-
spondence to: Max van Spengler <m.w.f.vanspengler@uva.nl>.
Proceedings of the 42 nd International Conference on Machine
Learning,...
A geometric framework for momentum-based
optimizers for low-rank training
Steffen Schotthöfer∗Timon Klein† and Jonas Kusch‡
Abstract
Low-rank pre-training and finetuning have recently emerged as promising tech-
niques for reducing the computational and storage costs of large neural networks.
Training low-rank parameterizations typically relies on conventional optimizers
such as heavy ball momentum methods or Adam. In this work, we identify and
analyze potential difficulties that these training methods encounter when used to
train low-rank parameterizations of weights. In particular, we show that classical
momentum methods can struggle to converge to a local optimum due to the ge-
ometry of the underlying optimization landscape. To address this, we introduce
novel training strategies that combine dynamical low-rank approximation with
momentum-based optimization, explicitly accounting for the intrinsic geometry
of the parameter space. We validate our methods through numerical experiments,
demonstrating stronger validation metrics at given parameter budgets.
1
Introduction
Deep learning models have achieved remarkable success across natural language processing and
computer vision tasks, but their deployment remains computationally expensive due to the large
number of trainable parameters. To address this, parameter-efficient strategies have been developed
to reduce memory and compute requirements during training. Common approaches include sparsifi-
cation [9, 22, 12], quantization [37, 6], and layer factorization. The latter has gained considerable
attention for pre-training [36, 16, 28, 29, 41] and especially finetuning [15, 34, 42, 10, 43, 20, 27].
Layer factorization represents weights (or adapters) as low-rank matrices, allowing only the low-rank
factors to be trained. This significantly reduces both memory usage and computational cost.
In the class of low-rank layer factorizations, one of the most popular methods is LoRA [15], which
applied typical optimizers such...
Published as a conference paper at ICLR 2025
INFLUENCE-GUIDED DIFFUSION FOR DATASET DISTIL-
LATION
Mingyang Chen1,2, Jiawei Du3, Bo Huang1,2, Yi Wang4, Xiaobo Zhang5, Wei Wang1,2
1The Hong Kong University of Science and Technology (Guangzhou)
2The Hong Kong University of Science and Technology
3CFAR, A*STAR, Singapore
4Dongguan University of Technology
5Southwest Jiaotong University
mchenbt@connect.ust.hk,weiwcs@ust.hk
ABSTRACT
Dataset distillation aims to streamline the training process by creating a compact
yet effective dataset for a much larger original dataset. However, existing methods
often struggle with distilling large, high-resolution datasets due to prohibitive
resource costs and limited performance, primarily stemming from sample-wise
optimizations in the pixel space. Motivated by the remarkable capabilities of
diffusion generative models in learning target dataset distributions and controllably
sampling high-quality data tailored to user needs, we propose framing dataset
distillation as a controlled diffusion generation task aimed at generating data
specifically tailored for effective training purposes. By establishing a correlation
between the overarching objective of dataset distillation and the trajectory influence
function, we introduce the Influence-Guided Diffusion (IGD) sampling framework
to generate training-effective data without the need to retrain diffusion models. An
efficient guided function is designed by leveraging the trajectory influence function
as an indicator to steer diffusions to produce data with influence promotion and
diversity enhancement. Extensive experiments show that the training performance
of distilled datasets generated by diffusions can be significantly improved by
integrating with our IGD method and achieving state-of-the-art performance in
distilling ImageNet datasets. Particularly, an exceptional result is achieved on
the ImageNet-1K, reaching 60.3% at IPC=50. Our code is available at https:
//github.com/mchen725/DD_...
A Reasoning-Based Approach to Cryptic Crossword Clue Solving
Martin Andrews 1 Sam Witteveen 1
Abstract
Cryptic crossword clues are challenging language
tasks for which new test sets are released daily by
major newspapers on a global basis. Each cryptic
clue contains both the definition of the answer to
be placed in the crossword grid (in common with
regular crosswords), and ‘wordplay’ that proves
that the answer is correct (i.e. a human solver
can be confident that an answer is correct without
needing crossing words as confirmation). This
work describes an LLM-based reasoning system
built from open-licensed components that solves
cryptic clues by (i) hypothesising answers; (ii)
proposing wordplay explanations; and (iii) using
a verifier system that operates on codified reason-
ing steps. Overall, this system establishes a new
state-of-the-art performance on the challenging
Cryptonite dataset of clues from The Times and
The Telegraph newspapers in the UK. Because
each proved solution is expressed in Python, inter-
pretable wordplay reasoning for proven answers
is available for inspection.
1. Introduction
There has been significant work in reasoning in the fields
of mathematics (Jiang et al., 2023; Yang et al., 2023; Trinh
et al., 2024) and code generation (Ni et al., 2023; Ridnik
et al., 2024) which benefit from having strong verifiers to
validate their answers. This work tackles the relatively
under-studied reasoning task of cryptic crossword solving,
which has the following qualities:
• Thousands of people find Cryptic Crosswords a satisfy-
ing intellectual challenge on a daily basis. Solving these
puzzles requires understanding multi-layered language
constructs, blending logic, wordplay, and contextual nu-
ance. This provides a unique challenge for evaluating and
improving LLMs’ capabilities in NLU and reasoning.
1Red Dragon AI, Singapore. Correspondence to: Martin An-
drews <martin@reddragon.ai>.
Proceedings of the 42 nd International Conference on Machine
Learni...
Published as a conference paper at ICLR 2025
CORNSTACK: HIGH-QUALITY CONTRASTIVE DATA
FOR BETTER CODE RETRIEVAL AND RERANKING
Tarun Suresh1,3∗†
Revanth Gangi Reddy1∗
Yifei Xu1,2
Zach Nussbaum3
Andriy Mulyar3
Brandon Duderstadt3
Heng Ji1
1University of Illinois Urbana-Champaign
2Lapis Labs
3Nomic AI
{tsuresh3,revanth3,yifeix5,hengji}@illinois.edu
ABSTRACT
Effective code retrieval plays a crucial role in advancing code generation, bug fix-
ing, and software maintenance, particularly as software systems increase in com-
plexity. While current code embedding models have demonstrated promise in
retrieving code snippets for small-scale, well-defined tasks, they often underper-
form in more demanding real-world applications such as bug localization within
GitHub repositories. We hypothesize that a key issue is their reliance on noisy
and inconsistent datasets for training, which impedes their ability to generalize
to more complex retrieval scenarios. To address these limitations, we introduce
CORNSTACK1, a large-scale, high-quality contrastive training dataset for code
that spans multiple programming languages. This dataset is curated using consis-
tency filtering to eliminate noisy positives and is further enriched with mined hard
negatives, thereby facilitating more effective learning. We demonstrate that con-
trastive training of embedding models using CORNSTACK leads to state-of-the-
art performance across a variety of code retrieval tasks. Furthermore, the dataset
can be leveraged for training code reranking models, a largely underexplored area
compared to text reranking. Our finetuned code reranking model significantly im-
proves the ranking quality over the retrieved results. Finally, by employing our
code retriever and reranker together, we demonstrate significant improvements in
function localization for GitHub issues, an important component of real-world
software development.
1
INTRODUCTION
The rapid advancement of software development has led to an increased rel...
DEALing with Image Reconstruction: Deep Attentive Least Squares
Mehrsa Pourya 1 Erich Kobler 2 Michael Unser 1 Sebastian Neumayer 3
Abstract
State-of-the-art image reconstruction often relies
on complex, abundantly parameterized deep ar-
chitectures. We propose an alternative: a data-
driven reconstruction method inspired by the clas-
sic Tikhonov regularization. Our approach itera-
tively refines intermediate reconstructions by solv-
ing a sequence of quadratic problems. These up-
dates have two key components: (i) learned filters
to extract salient image features; and (ii) an atten-
tion mechanism that locally adjusts the penalty of
the filter responses. Our method matches leading
plug-and-play and learned regularizer approaches
in performance while offering interpretability, ro-
bustness, and convergent behavior. In effect, we
bridge traditional regularization and deep learning
with a principled reconstruction approach.
Code: https://github.com/mehrsapo/DEAL.
1. Introduction
Image reconstruction plays a fundamental role in compu-
tational imaging and computer vision (McCann & Unser,
2019; Zeng, 2001). The task is to recover an unknown im-
age of interest x ∈Rd from noisy measurements y ∈RM.
Their relation is modeled as y = Hx, where the forward op-
erator H ∈RM×d encodes the acquisition process. When
H is ill-conditioned, one can resort to the regularized recon-
struction
ˆx ∈arg min
x∈Rd

∥Hx −y∥2
2 + λR(x)

.
(1)
The data fidelity ∥Hx −y∥2
2 controls the consistency of
the reconstruction with the measurements. The regularizer
R: Rd →R≥0 encodes prior information about the solu-
tion and is also intended to make the problem well-posed.
1Biomedical Imaging group, EPFL Lausanne, Switzerland
2Institute for Machine Learning, JKU Linz, Austria 3Faculty
of Mathematics, TU Chemnitz, Germany. Correspondence to:
Mehrsa Pourya <mehrsa.pourya@epfl.ch>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025...
Private Federated Learning using Preference-Optimized Synthetic Data
Charlie Hou * 1 Mei-Yu Wang * 2 Yige Zhu * Daniel Lazar 3 Giulia Fanti 1
Abstract
In practical settings, differentially private feder-
ated learning (DP-FL) is the dominant method
for training models from private, on-device client
data. Recent work has suggested that DP-FL may
be enhanced or outperformed by methods that use
DP synthetic data (Wu et al., 2024; Hou et al.,
2024). The primary algorithms for generating DP
synthetic data for FL applications require careful
prompt engineering based on public information
and/or iterative private client feedback. Our key
insight is that the private client feedback collected
by prior DP synthetic data methods (Hou et al.,
2024; Xie et al., 2024) can be viewed as an RL
reward. Our algorithm, Policy Optimization for
Private Data (POPri) harnesses client feedback
using policy optimization algorithms such as Di-
rect Preference Optimization (DPO) to fine-tune
LLMs to generate high-quality DP synthetic data.
To evaluate POPri, we release LargeFedBench, a
new federated text benchmark for uncontaminated
LLM evaluations on federated client data. POPri
closes the gap in performance between the fully-
private and non-private settings by up to 58%,
compared to 28% for prior synthetic data methods,
and 3% for state-of-the-art DP federated learn-
ing methods. The code and data are available at
https://github.com/meiyuw/POPri.
1. Introduction
Many important machine learning (ML) applications feature
sensitive datasets that are distributed across client devices
(e.g. mobile devices). Such ML models are often hosted
on client devices. These on-device models offer privacy, la-
tency, and storage benefits relative to centrally-hosted mod-
*Equal contribution
1Department of ECE, Carnegie Mellon
University, Pittsburgh, PA 2Pittsburgh Supercomputing Center,
Pittsburgh, USA 3Coldrays, Tucson, AZ. Correspondence to: Char-
lie Hou <hou.charlie2@gmail.com>.
Proceedings of the 42 nd...
Published as a conference paper at ICLR 2025
LOSSY COMPRESSION WITH
PRETRAINED DIFFUSION MODELS
Jeremy Vonderfecht & Feng Liu
Department of Computer Science
Portland State University
{vonder2,fliu}@pdx.edu
ABSTRACT
We apply the DiffC algorithm (Theis et al., 2022) to Stable Diffusion 1.5, 2.1,
XL, and Flux-dev, and demonstrate that these pretrained models are remarkably
capable lossy image compressors. A principled algorithm for lossy compression
using pretrained diffusion models has been understood since at least (Ho et al.,
2020), but challenges in reverse-channel coding have prevented such algorithms
from ever being fully implemented. We introduce simple workarounds that lead
to the first complete implementation of DiffC, which is capable of compressing
and decompressing images using Stable Diffusion in under 10 seconds. Despite
requiring no additional training, our method is competitive with other state-of-the-
art generative compression methods at low ultra-low bitrates.
1
INTRODUCTION
Today’s AI labs are spending millions of dollars training state-of-the-art diffusion models. The
original Stable Diffusion cost around $600,000 to train, while Stable Diffusion 3 is rumored to have
cost around $10 million (Wikipedia, 2024). The sophistication and expense of these models are only
likely to increase as generative modelling expands into the video domain.
While diffusion models are used primarily for prompt-based image generation, they are also highly
powerful and flexible image priors. They provide many other affordances, such as easy multiplica-
tion with other distributions (i.e., posterior sampling), and log-likelihood estimation (Sohl-Dickstein
et al., 2015). Researchers have discovered many remarkable applications of pretrained diffusion
models, including image restoration (Chung et al., 2022), optical illusions (Geng et al., 2024b;a),
and generative 3D modeling (Poole et al., 2022).
Ho et al. (2020) proposed an algorithm by which pretrained diffusion models ca...
Diffusion on Language Model Encodings for Protein Sequence Generation
Viacheslav Meshchaninov * 1 Pavel Strashnov * 2 Andrey Shevtsov * 2 Fedor Nikolaev 2 Nikita Ivanisenko 2
Olga Kardymon 2 Dmitry Vetrov 1
Abstract
Protein sequence design has seen significant ad-
vances through discrete diffusion and autore-
gressive approaches, yet the potential of con-
tinuous diffusion remains underexplored. Here,
we present DiMA, a latent diffusion framework
that operates on protein language model rep-
resentations.
Through systematic exploration
of architectural choices and diffusion compo-
nents, we develop a robust methodology that gen-
eralizes across multiple protein encoders rang-
ing from 8M to 3B parameters.
We demon-
strate that our framework achieves consistently
high performance across sequence-only (ESM-
2, ESMc), dual-decodable (CHEAP), and mul-
timodal (SaProt) representations using the same
architecture and training approach. We exten-
sively evaluate existing methods alongside DiMA
using multiple metrics across two protein modali-
ties, covering quality, diversity, novelty, and dis-
tribution matching of generated proteins. DiMA
consistently produces novel, high-quality and di-
verse protein sequences and achieves strong re-
sults compared to baselines such as autoregres-
sive, discrete diffusion and flow matching lan-
guage models. The model demonstrates versa-
tile functionality, supporting conditional gener-
ation tasks including protein family-generation,
motif scaffolding and infilling, and fold-specific
sequence design. This work provides a univer-
sal continuous diffusion framework for protein
sequence generation, offering both architectural
insights and practical applicability across vari-
ous protein design scenarios. Code is released at
GitHub.
*Equal contribution
1Constructor University, Bremen, Ger-
many 2AIRI, Moscow, Russia.
Correspondence to:
Viach-
eslav Meshchaninov <meshchaninov.viacheslav@gmail.com>,
Pavel
Strashnov
<strashnov@airi.net>,
Andre...