AIS-Bench/AIS-Bench huggingface dataset:
Field Matching: an Electrostatic Paradigm to Generate and Transfer Data
Alexander Kolesov * 1 2 Stepan I. Manukhov * 1 3 Vladimir V. Palyulin 1 Alexander Korotin 1 2
Abstract
We propose Electrostatic Field Matching (EFM),
a novel method that is suitable for both genera-
tive modeling and distribution transfer tasks. Our
approach is inspired by the physics of an electri-
cal capacitor. We place source and target distri-
butions on the capacitor plates and assign them
positive and negative charges, respectively. Then
we learn the electrostatic field of the capacitor
using a neural network approximator. To map the
distributions to each other, we start at one plate
of the capacitor and move the samples along the
learned electrostatic field lines until they reach
the other plate. We theoretically justify that this
approach provably yields the distribution transfer.
In practice, we demonstrate the performance of
our EFM in toy and image data experiments. Our
code is available at https://github.com/
justkolesov/FieldMatching
1. Introduction
The basic task of generative modeling is to learn a transfor-
mation between two distributions accessible by i.i.d. sam-
ples. The typical scenarios considered are noise-to-data
(Goodfellow et al., 2014) and data-to-data (Zhu et al.,
2017). These are usually referred to as the unconditional
data generation and data translation, respectively.
Physics is often at the heart of the principles of generative
modeling. One of the first attempts to link generative mod-
els and physics was made in Energy-Based models (LeCun
& Huang, 2005, EBM). They parameterize data distribu-
tions using the Gibbs-Boltzmann distribution density and
generate data through simulation of Langevin dynamics (Du
& Mordatch, 2019; Song & Kingma, 2021).
*Equal contribution 1Skolkovo Institute of Science and Tech-
nology, Moscow, Russia 2Artificial Intelligence Research Institute,
Moscow, Russia 3Lomonosov Moscow State University, Faculty of
Physics, Moscow, Russia. Corr...
Conformal Tail Risk Control for Large Language Model Alignment
Catherine Chen 1 Jingyan Shen 2 Zhun Deng 3 Lihua Lei 4 5
Abstract
Recent developments in large language models
(LLMs) have led to their widespread usage for
various tasks. The prevalence of LLMs in society
implores the assurance on the reliability of their
performance. In particular, risk-sensitive applica-
tions demand meticulous attention to unexpect-
edly poor outcomes, i.e., tail events, for instance,
toxic answers, humiliating language, and offen-
sive outputs. Due to the costly nature of acquiring
human annotations, general-purpose scoring mod-
els have been created to automate the process of
quantifying these tail events. This phenomenon in-
troduces potential human-machine misalignment
between the respective scoring mechanisms. In
this work, we present a lightweight calibration
framework for blackbox models that ensures the
alignment of humans and machines with provable
guarantees. Our framework provides a rigorous
approach to controlling any distortion risk mea-
sure that is characterized by a weighted average
of quantiles of the loss incurred by the LLM with
high confidence. The theoretical foundation of our
method relies on the connection between confor-
mal risk control and a traditional family of statis-
tics, i.e., L-statistics. To demonstrate the utility of
our framework, we conduct comprehensive exper-
iments that address the issue of human-machine
misalignment.
1. Introduction
Large Language Models (LLMs) have proven to be per-
vasive in society with applications across various settings,
*Equal contribution
1Institute for Computational and Math-
ematical Engineering, Stanford University, CA 2Department of
Industrial Engineering and Operations Research, Columbia Uni-
versity, NY 3Department of Computer Science, UNC Chapel
Hill, NC 4Graduate School of Business, Stanford University, CA
5Department of Statistics (by courtesy), Stanford University, CA.
Correspondence to: Catherine Chen <cyc2...
Published as a conference paper at ICLR 2025
IDENTIFYING LATENT STATE TRANSITIONS IN
NON-LINEAR DYNAMICAL SYSTEMS
C¸ a˘glar Hızlı
Aalto University
caglar.hizli@aalto.fi
C¸ a˘gatay Yıldız
University of T¨ubingen
T¨ubingen AI Center
Matthias Bethge
University of T¨ubingen
T¨ubingen AI Center
ST John
Aalto University
Pekka Marttinen
Aalto University
ABSTRACT
This work aims to recover the underlying states and their time evolution in a
latent dynamical system from high-dimensional sensory measurements. Previous
works on identifiable representation learning in dynamical systems focused on
identifying the latent states, often with linear transition approximations. As such,
they cannot identify nonlinear transition dynamics, and hence fail to reliably
predict complex future behavior. Inspired by the advances in nonlinear ICA, we
propose a state-space modeling framework in which we can identify not just
the latent states but also the unknown transition function that maps the past
states to the present. Our identifiability theory relies on two key assumptions: (i)
sufficient variability in the latent noise, and (ii) the bijectivity of the augmented
transition function. Drawing from this theory, we introduce a practical algorithm
based on variational auto-encoders. We empirically demonstrate that it improves
generalization and interpretability of target dynamical systems by (i) recovering
latent state dynamics with high accuracy, (ii) correspondingly achieving high future
prediction accuracy, and (iii) adapting fast to new environments. Additionally, for
complex real-world dynamics, (iv) it produces state-of-the-art future prediction
results for long horizons, highlighting its usefulness for practical scenarios.
1
INTRODUCTION
We focus on the problem of understanding the underlying states of a dynamical system from its
high-dimensional sensory measurements. This task is prevalent across various fields, including
reinforcement learning (Hafner et al., 2019a) and robotics (Levi...
CurvGAD: Leveraging Curvature for Enhanced Graph Anomaly Detection
Karish Grover 1 Geoffrey J. Gordon 1 Christos Faloutsos 1
Abstract
Does the intrinsic curvature of complex net-
works hold the key to unveiling graph anoma-
lies that conventional approaches overlook?
Reconstruction-based graph anomaly detection
(GAD) methods overlook such geometric outliers,
focusing only on structural and attribute-level
anomalies. To this end, we propose CurvGAD
— a mixed-curvature graph autoencoder that in-
troduces the notion of curvature-based geomet-
ric anomalies. CurvGAD introduces two parallel
pipelines for enhanced anomaly interpretability:
(1) Curvature-equivariant geometry reconstruc-
tion, which focuses exclusively on reconstruct-
ing the edge curvatures using a mixed-curvature,
Riemannian encoder and Gaussian kernel-based
decoder; and (2) Curvature-invariant structure
and attribute reconstruction, which decouples
structural and attribute anomalies from geomet-
ric irregularities by regularizing graph curvature
under discrete Ollivier-Ricci flow, thereby isolat-
ing the non-geometric anomalies. By leveraging
curvature, CurvGAD refines the existing anomaly
classifications and identifies new curvature-driven
anomalies. Extensive experimentation over 10
real-world datasets (both homophilic and het-
erophilic) demonstrates an improvement of up to
6.5% over state-of-the-art GAD methods. The
code is available at:
https://github.com/
karish-grover/curvgad.
1. Introduction
Detecting anomalies in graph-structured data is a pivotal
task across diverse fields, including social networks (Has-
sanzadeh et al., 2012), cybersecurity (Wang & Zhu, 2022),
transportation systems (Hu et al., 2020), and biological net-
works (Singh & Vig, 2017). Traditional methods primarily
1Carnegie Mellon University.
Correspondence to: Karish
Grover <karishg@cs.cmu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(...
Preference Learning for AI Alignment: a Causal Perspective
Katarzyna Kobalczyk 1 Mihaela van der Schaar 1
Abstract
Reward modelling from preference data is a
crucial step in aligning large language models
(LLMs) with human values, requiring robust gen-
eralisation to novel prompt-response pairs. In
this work, we propose to frame this problem in
a causal paradigm, providing the rich toolbox
of causality to identify the persistent challenges,
such as causal misidentification, preference het-
erogeneity, and confounding due to user-specific
factors. Inheriting from the literature of causal
inference, we identify key assumptions neces-
sary for reliable generalisation and contrast them
with common data collection practices. We illus-
trate failure modes of naive reward models and
demonstrate how causally-inspired approaches
can improve model robustness. Finally, we out-
line desiderata for future research and practices,
advocating targeted interventions to address inher-
ent limitations of observational data.
1. Introduction
The remarkable success of LLMs lies partially in their abil-
ity to align with human values, producing responses that
are helpful, harmless, and honest. A central method for
achieving this alignment is reinforcement learning from hu-
man feedback (RLHF), with the LLM’s behaviour shaped
by reward models derived from datasets of human prefer-
ences (Ouyang et al., 2022; Bai et al., 2022). The reward
modelling, aka preference learning stage seeks to address a
seemingly straightforward question: Given two responses
to the same prompt, which one aligns better with human
objectives? However, the challenges of this stage are of-
ten underestimated, assuming that simple regression-based
models fitted to observational datasets can generalise effec-
tively to unseen texts. Meanwhile, evidence suggests this
optimism may be misplaced (Tien et al., 2023; Skalse et al.,
1Department of Applied Mathematics and Theoretical Physics,
University of Cambridge, United Ki...
DCBM: Data-Efficient Visual Concept Bottleneck Models
Katharina Prasse * 1 Patrick Knab * 2 Sascha Marton 2 Christian Bartelt 2 Margret Keuper 1 3
Abstract
Concept Bottleneck Models (CBMs) enhance
the interpretability of neural networks by basing
predictions on human-understandable concepts.
However, current CBMs typically rely on con-
cept sets extracted from large language models
or extensive image corpora, limiting their effec-
tiveness in data-sparse scenarios. We propose
Data-efficient CBMs (DCBMs), which reduce the
need for large sample sizes during concept gener-
ation while preserving interpretability. DCBMs
define concepts as image regions detected by seg-
mentation or detection foundation models, allow-
ing each image to generate multiple concepts
across different granularities. Exclusively con-
taining dataset-specific concepts, DCBMs are
well suited for fine-grained classification and out-
of-distribution tasks. Attribution analysis using
Grad-CAM demonstrates that DCBMs deliver vi-
sual concepts that can be localized in test images.
By leveraging dataset-specific concepts instead
of predefined or general ones, DCBMs enhance
adaptability to new domains. The code is avail-
able at: https://github.com/KathPra/
DCBM.
1. Introduction
Deep neural networks (DNNs) achieve state-of-the-art per-
formance across various domains, yet their opaque nature
limits trust, particularly in safety-critical applications. To ad-
dress this, Explainable Artificial Intelligence (XAI) strives
to make model decisions more transparent and interpretable.
Concept Bottleneck Models (CBMs) (Koh et al., 2020) are
an inherently interpretable framework, as they base predic-
*Equal contribution
1Data and Web Science Group, Univer-
sity of Mannheim, Mannheim, Germany 2Clausthal University of
Technology, Clausthal, Germany 3Max-Planck-Institute for Infor-
matics, Saarland Informatics Campus. Correspondence to: Katha-
rina Prasse <katharina.prasse@uni-mannheim.de>, Patrick Knab
<patrick.kna...
Published as a conference paper at ICLR 2025
MASTERING TASK ARITHMETIC: τJP AS
A KEY INDICATOR FOR WEIGHT DISENTANGLEMENT
Kotaro Yoshida1,∗,†, Yuji Naraki2, Takafumi Horie3,∗, Ryosuke Yamaki 3,4,
Ryotaro Shimizu5,6, Yuki Saito5, Julian McAuley6, Hiroki Naganuma 7,8,†
1 Institute of Science Tokyo, 2 Independent Researcher, 3 Ritsumeikan University, 4 ProPlace,
5 ZOZO Research, 6 University of California San Diego, 7 Universit´e de Montr´eal, 8 Mila
ABSTRACT
Model-editing techniques using task arithmetic have rapidly gained attention.
Through task arithmetic, simply through arithmetic operations on the weights of
pre-trained and fine-tuned models create desired models, such as multi-task mod-
els, models in which specific tasks are unsolvable, or domain-transferred models.
However, task arithmetic faces challenges, such as poor reproducibility and the
high cost associated with adjusting coefficients in the arithmetic operations on
model parameters, which have limited its practical success. In this paper, we
present three key contributions in the context of task addition and task negation
within task arithmetic. First, we propose a new metric called τJp which is based
on the product of the task vector (τ) and the Jacobian of the pre-trained model with
respect to its weights. We show that τJp has a causal relationship with the inter-
ference that occurs from arithmetic operations. Second, we show that introducing
regularization to minimize τJp significantly mitigates interference between task
inference, which leads to the elimination of coefficient tuning and improved accu-
racy on each task. Third, in the context of incremental learning, we demonstrate
that our τJp regularization achieves more robust performance in environments
where access to future tasks is unavailable, thus validating the scalability of the
approach. Finally, we demonstrate that the τJp regularizer further reinforces the
performance of task arithmetic by leveraging publicly available fine-tuned mod-...
GLGENN: A Novel Parameter-Light Equivariant Neural Networks
Architecture Based on Clifford Geometric Algebras
Ekaterina Filimoshina 1 2 Dmitry Shirokov 1 3
Abstract
We propose, implement, and compare with com-
petitors a new architecture of equivariant neural
networks based on geometric (Clifford) algebras:
Generalized Lipschitz Group Equivariant Neu-
ral Networks (GLGENN). These networks are
equivariant to all pseudo-orthogonal transforma-
tions, including rotations and reflections, of a vec-
tor space with any non-degenerate or degenerate
symmetric bilinear form. We propose a weight-
sharing parametrization technique that takes into
account the fundamental structures and operations
of geometric algebras. Due to this technique,
GLGENN architecture is parameter-light and has
less tendency to overfitting than baseline equiv-
ariant models. GLGENN outperforms or matches
competitors on several benchmarking equivariant
tasks, including estimation of an equivariant func-
tion and a convex hull experiment, while using
significantly fewer optimizable parameters.
1. Introduction
Equivariant neural networks are a class of neural networks
that explicitly incorporate symmetries into their architec-
ture, making them well-suited for tasks that inherently re-
quire equivariance to transformations with respect to some
group’s action (e.g., rotations, permutations, translations,
etc.). Equivariant neural networks were introduced in Cohen
& Welling, 2016 and have since been extensively developed
and applied in a wide range of tasks in computer and natural
sciences. These applications include modeling dynamical
systems (Finzi et al., 2021), particle physics (Ruhe et al.,
2023; Finzi et al., 2021), analyzing molecular properties and
1HSE University, Moscow, Russia 2 Skolkovo Institute of
Science and Technology, Moscow, Russia 3 Institute for In-
formation Transmission Problems of the Russian Academy
of Sciences, Moscow, Russia.
Correspondence to:
Ekate-
rina Filimoshina <filimoshinae...
H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease
Detection with Pre-Trained Models
Rushuang Zhou 1 2 Yuan-Ting Zhang 3 4 5 Yining Dong 6 2
Abstract
Fine-tuning large-scale pre-trained models pro-
vides an effective solution to alleviate the la-
bel scarcity problem in cardiovascular diseases
(CVDs) detection using electrocardiogram (ECG).
However, as the pre-trained models scale up, the
computational costs for fine-tuning and inference
become unaffordable on low-level devices de-
ployed for clinical applications.
Additionally,
maintaining the model performance under low
budgets in computational resources remains a sig-
nificant challenge. However, a comprehensive
study that can address them in a joint framework is
still lacking. Here, we propose a holistic method
(H-Tuning) for low-cost and efficient fine-tuning
of pre-trained models on downstream datasets.
Then, the inference costs of the models fine-tuned
by H-Tuning are further reduced significantly us-
ing a knowledge distillation technique. Exper-
iments on four ECG datasets demonstrate that
H-Tuning reduces the GPU memory consumption
during fine-tuning by 6.34 times while achieving
comparable CVDs detection performance to stan-
dard fine-tuning. With the knowledge distillation
technique, the model inference latency and the
memory consumption are reduced by 4.52 times
and 19.83 times. As such, the proposed joint
framework allows for the utilization of pre-trained
models with high computation efficiency and ro-
bust performance, exploring a path toward low-
cost and efficient CVDs detection. Code is avail-
able at https://github.com/KAZABANA/H-Tuning
1Department of Biomedical Engineering, City University
of Hong Kong, Hong Kong, China 2Hong Kong Center for
Cerebro-Cardiovascular Health Engineering, Hong Kong, China
3Department of Electronic Engineering, Chinese University of
Hong Kong, Hong Kong, China 4Hong Kong Institutes of Medical
Engineering, Hong Kong, China 5The AICARE Bay Lab, Gua...
Test-Time Spectrum-Aware Latent Steering for
Zero-Shot Generalization in Vision-Language Models
Konstantinos M. Dafnis∗
Department of Computer Science
Rutgers University
Dimitris N. Metaxas
Department of Computer Science
Rutgers University
Abstract
Vision–Language Models (VLMs) excel at zero-shot inference but often degrade
under test-time domain shifts. For this reason, episodic test-time adaptation strate-
gies have recently emerged as powerful techniques for adapting VLMs to a single
unlabeled image. However, existing adaptation strategies, such as test-time prompt
tuning, typically require backpropagating through large encoder weights or altering
core model components. In this work, we introduce Spectrum-Aware Test-Time
Steering (STS), a lightweight adaptation framework that extracts a spectral sub-
space from the textual embeddings to define principal semantic directions and learns
to steer latent representations in a spectrum-aware manner by adapting a small num-
ber of per-sample shift parameters to minimize entropy across augmented views.
STS operates entirely at inference in the latent space, without backpropagation
through or modification of the frozen encoders. Building on standard evaluation
protocols, our comprehensive experiments demonstrate that STS largely surpasses
or compares favorably against state-of-the-art test-time adaptation methods, while
introducing only a handful of additional parameters and achieving inference speeds
up to 8× faster with a 12× smaller memory footprint than conventional test-time
prompt tuning. The code is available at https://github.com/kdafnis/STS.
1
Introduction
Vision-Language Models (VLMs), such as CLIP [30], have marked a paradigm shift in artificial
intelligence, demonstrating remarkable zero-shot generalization capabilities across a multitude of
downstream visual tasks. By learning rich joint representations from vast quantities of image-text
data, these models can often perform tasks like image classification with...
Benign Overfitting in Token Selection of Attention Mechanism
Keitaro Sakamoto 1 Issei Sato 1
Abstract
Attention mechanism is a fundamental component
of the transformer model and plays a significant
role in its success. However, the theoretical under-
standing of how attention learns to select tokens
is still an emerging area of research. In this work,
we study the training dynamics and generalization
ability of the attention mechanism under classifi-
cation problems with label noise. We show that,
with the characterization of signal-to-noise ratio
(SNR), the token selection of attention mechanism
achieves “benign overfitting”, i.e., maintaining
high generalization performance despite fitting
label noise. Our work also demonstrates an inter-
esting delayed acquisition of generalization after
an initial phase of overfitting. Finally, we provide
experiments to support our theoretical analysis
using both synthetic and real-world datasets.
1. Introduction
The transformer models (Vaswani et al., 2017) have greatly
succeeded across a wide range of fields, including natural
language (Devlin et al., 2018; Brown et al., 2020), vision
(Dosovitskiy et al., 2021; Touvron et al., 2021), and have be-
come a foundational architecture in modern machine learn-
ing. The key component that characterizes the transformer
model is the attention mechanism, which was originally
introduced in recurrent neural network (RNN) and long
short-term memory (LSTM) to capture the long-range struc-
ture of sequence (Bahdanau, 2014; Xu, 2015). The attention
architecture can process variable-length input sequences
and flexibly select important tokens based on the input. The
seminal work on the training dynamics of this token selec-
tion mechanism (Tarzanagh et al., 2023a;b; Vasudeva et al.,
2024; Sheen et al., 2024) studied the implicit bias of gradi-
ent descent to the max-margin token separator. However,
1Department of Computer Science, The University of
Tokyo, Tokyo, Japan. Correspondence to: Keitaro...
Joint Relational Database Generation via
Graph-Conditional Diffusion Models
Mohamed Amine Ketata, David Lüdke, Leo Schwinn, Stephan Günnemann
School of Computation, Information and Technology & Munich Data Science Institute
Technical University of Munich, Germany
Correspondence to: a.ketata@tum.de
Abstract
Building generative models for relational databases (RDBs) is important for many
applications, such as privacy-preserving data release and augmenting real datasets.
However, most prior works either focus on single-table generation or adapt single-
table models to the multi-table setting by relying on autoregressive factorizations
and sequential generation. These approaches limit parallelism, restrict flexibility in
downstream applications, and compound errors due to commonly made conditional
independence assumptions. In this paper, we propose a fundamentally different ap-
proach: jointly modeling all tables in an RDB without imposing any table order. By
using a natural graph representation of RDBs, we propose the Graph-Conditional
Relational Diffusion Model (GRDM), which leverages a graph neural network
to jointly denoise row attributes and capture complex inter-table dependencies.
Extensive experiments on six real-world RDBs demonstrate that our approach sub-
stantially outperforms autoregressive baselines in modeling multi-hop inter-table
correlations and achieves state-of-the-art performance on single-table fidelity met-
rics. Our code is available at https://github.com/ketatam/rdb-diffusion.
1
Introduction
Table A
Table B
Table C
Step 1
Autoregressive
Joint (Ours)
Limiting table order
   
Sequential generation
   
Simplifying assumptions
No table order
     Parallel generation
     Joint modeling of all rows
Step 2
Step 3
Table A
Table B
Table C
Figure 1: Comparison of autoregressive
and joint relational database generation.
Relational databases (RDBs), which organize data into
multiple interlinked tables, are the most widely used
data management system, estim...