AIS-Bench/AIS-Bench huggingface dataset:
One-Step Generalization Ratio Guided Optimization for Domain Generalization
Sumin Cho * 1 Dongwon Kim * 1 Kwangsu Kim 1
Abstract
Domain Generalization (DG) aims to train models
that generalize to unseen target domains but often
overfit to domain-specific features, known as un-
desired correlations. Gradient-based DG methods
typically guide gradients in a dominant direction
but often inadvertently reinforce spurious corre-
lations. Recent work has employed dropout to
regularize overconfident parameters, but has not
explicitly adjusted gradient alignment or ensured
balanced parameter updates. We propose GENIE
(Generalization-ENhancing Iterative Equalizer), a
novel optimizer that leverages the One-Step Gen-
eralization Ratio (OSGR) to quantify each param-
eter’s contribution to loss reduction and assess
gradient alignment. By dynamically equalizing
OSGR via a preconditioning factor, GENIE pre-
vents a small subset of parameters from domi-
nating optimization, thereby promoting domain-
invariant feature learning. Theoretically, GENIE
balances convergence contribution and gradient
alignment among parameters, achieving higher
OSGR while retaining SGD’s convergence rate.
Empirically, it outperforms existing optimizers
and enhances performance when integrated with
various DG and single-DG methods.
1. Introduction
Deep neural networks (DNNs) achieve high accuracy when
training and test data share a similar distribution. How-
ever, in real-world applications, data distributions often shift,
causing performance degradation(Muandet et al., 2013). Do-
main Generalization (DG) addresses this issue by training
models to generalize to out-of-distribution data from un-
seen domains. The main challenge is to prevent overfitting
*Equal contribution
1Department of Computer Science and
Engineering, University of Sungkyunkwan, Suwon, Korea.
Correspondence
to:
Sumin
Cho
<jsm0707@skku.edu>,
Dongwon
Kim
<kdwaha@skku.edu>,
Kwangsu
Kim
<kim.kwangsu@skku.edu>.
Proceedings of the 42 nd Internat...
Sparse Autoencoders, Again?
Yin Lu 1 Xuening Zhu 1 Tong He 2 David Wipf 2
Abstract
Is there really much more to say about sparse au-
toencoders (SAEs)? Autoencoders in general, and
SAEs in particular, represent deep architectures
that are capable of modeling low-dimensional la-
tent structure in data. Such structure could re-
flect, among other things, correlation patterns
in large language model activations, or complex
natural image manifolds. And yet despite the
wide-ranging applicability, there have been rel-
atively few changes to SAEs beyond the origi-
nal recipe from decades ago, namely, standard
deep encoder/decoder layers trained with a classi-
cal/deterministic sparse regularizer applied within
the latent space. One possible exception is the
variational autoencoder (VAE), which adopts a
stochastic encoder module capable of producing
sparse representations when applied to manifold
data. In this work we formalize underappreciated
weaknesses with both canonical SAEs, as well as
analogous VAEs applied to similar tasks, and pro-
pose a hybrid alternative model that circumvents
these prior limitations. In terms of theoretical sup-
port, we prove that global minima of our proposed
model recover certain forms of structured data
spread across a union of manifolds. Meanwhile,
empirical evaluations on synthetic and real-world
datasets substantiate the efficacy of our approach
in accurately estimating underlying manifold di-
mensions and producing sparser latent representa-
tions without compromising reconstruction error.
In general, we are able to exceed the performance
of equivalent-capacity SAEs and VAEs, as well as
recent diffusion models where applicable, within
domains such as images and language model acti-
vation patterns. A link to the code is here.
1School of Data Science, Fudan University 2Amazon Web Services.
Correspondence
to:
Yin
Lu
<yinlu23@m.fudan.edu.cn>,
Xuening
Zhu
<xueningzhu@fudan.edu.cn>,
Tong
He
<htong@amazon.com>, David Wipf <davidwipf@gmail.com...
Function-Space Learning Rates
Edward Milsom 1 Ben Anson 1 Laurence Aitchison 1
Abstract
We consider layerwise function-space learning
rates, which measure the magnitude of the change
in a neural network’s output function in response
to an update to a parameter tensor. This con-
trasts with traditional learning rates, which de-
scribe the magnitude of changes in parameter
space. We develop efficient methods to measure
and set function-space learning rates in arbitrary
neural networks, requiring only minimal compu-
tational overhead through a few additional back-
ward passes that can be performed at the start
of, or periodically during, training. We demon-
strate two key applications: (1) analysing the dy-
namics of standard neural network optimisers in
function space, rather than parameter space, and
(2) introducing FLeRM (Function-space Learn-
ing Rate Matching), a novel approach to hyper-
parameter transfer across model scales. FLeRM
records function-space learning rates while train-
ing a small, cheap base model, then automatically
adjusts parameter-space layerwise learning rates
when training larger models to maintain consistent
function-space updates. FLeRM gives hyperpa-
rameter transfer across model width, depth, initial-
isation scale, and LoRA rank in various architec-
tures including MLPs with residual connections
and transformers with different layer normalisa-
tion schemes.
1. Introduction
The fundamental purpose of neural network training is to
learn a function that maps inputs to desired outputs. How-
ever, we typically understand optimisation methods as act-
ing in parameter space, e.g. traditional learning rates tell us
how much the parameters change during each step rather
than the functional impact of those changes. This raises
1University of Bristol.
Correspondence to: Edward Mil-
som <edward.milsom@bristol.ac.uk>, Laurence Aitchison <lau-
rence.aitchison@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, ...
Clients Collaborate: Flexible Differentially Private Federated Learning with
Guaranteed Improvement of Utility-Privacy Trade-off
Yuecheng Li 1 Lele Fu 1 Tong Wang 2 Jian Lou 1 Bin Chen 3 Lei Yang 1 Jian Shen 4 Zibin Zheng 1
Chuan Chen 1
Abstract
To defend against privacy leakage of user data,
differential privacy is widely used in federated
learning, but it is not free.
The addition of
noise randomly disrupts the semantic integrity
of the model and this disturbance accumulates
with increased communication rounds. In this
paper, we introduce a novel federated learning
framework with rigorous privacy guarantees,
named FedCEO, designed to strike a trade-off
between model utility and user privacy by
letting clients “Collaborate with Each Other”.
Specifically, we perform efficient tensor low-rank
proximal optimization on stacked local model
parameters at the server, demonstrating its
capability to flexibly truncate high-frequency
components in spectral space. This capability
implies that our FedCEO can effectively recover
the disrupted semantic information by smoothing
the global semantic space for different privacy
settings
and
continuous
training
processes.
Moreover, we improve the SOTA utility-privacy
trade-off bound by order of
√
d, where d is the
input dimension. We illustrate our theoretical
results
with
experiments
on
representative
datasets and observe significant performance
improvements and strict privacy guarantees under
different privacy settings. The code is available at
https://github.com/6lyc/FedCEO_
Collaborate-with-Each-Other.
1Sun Yat-sen University, Guangzhou, China 2Texas A&M Uni-
versity, Texas, USA 3Harbin Institute of Technology, Shenzhen,
China 4Zhejiang Sci-Tech University, China. Correspondence to:
Chuan Chen <chenchuan@mail.sysu.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
Federated learning (FL) (McMahan et al., 2017), a priva...
ML2-GCL: Manifold Learning Inspired Lightweight Graph Contrastive
Learning
Jianqing Liang 1 Zhiqiang Li 1 Xinkai Wei 1 Yuan Liu 1 Zhiqiang Wang 1
Abstract
Graph contrastive learning has attracted great
interest as a dominant and promising self-
supervised representation learning approach in
recent years. While existing works follow the ba-
sic principle of pulling positive pairs closer and
pushing negative pairs far away, they still suffer
from several critical problems, such as the under-
lying semantic disturbance brought by augmen-
tation strategies, the failure of GCN in capturing
long-range dependence, rigidness and inefficiency
of node sampling techniques. To address these
issues, we propose Manifold Learning Inspired
Lightweight Graph Contrastive Learning (ML2-
GCL), which inherits the merits of both manifold
learning and GCN. ML2-GCL avoids the poten-
tial risks of semantic disturbance with only one
single view. It achieves global nonlinear struc-
ture recovery from locally linear fits, which can
make up for the defects of GCN. The most amaz-
ing advantage is about the lightweight due to its
closed-form solution of positive pairs weights and
removal of pairwise distances calculation. The-
oretical analysis proves the existence of the op-
timal closed-form solution. Extensive empirical
results on various benchmarks and evaluation pro-
tocols demonstrate effectiveness and lightweight
of ML2-GCL. We release the code at https:
//github.com/a-hou/ML2-GCL.
1. Introduction
Graph representation learning, which aims to learn appro-
priate low-dimensional representations for graph structured
*Equal contribution 1Key Laboratory of Computational Intel-
ligence and Chinese Information Processing of Ministry of Edu-
cation, School of Computer and Information Technology, Shanxi
University, Taiyuan 030006, Shanxi, China. Correspondence to:
Jianqing Liang <liangjq@sxu.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267,...
Stochastic Encodings for Active Feature Acquisition
Alexander Norcliffe 1 Changhee Lee 2 Fergus Imrie 3 Mihaela van der Schaar 4 Pietro Liò 1
Abstract
Active Feature Acquisition is an instance-wise,
sequential decision making problem. The aim is
to dynamically select which feature to measure
based on current observations, independently for
each test instance. Common approaches either
use Reinforcement Learning, which experiences
training difficulties, or greedily maximize the con-
ditional mutual information of the label and unob-
served features, which makes myopic acquisitions.
To address these shortcomings, we introduce a la-
tent variable model, trained in a supervised man-
ner. Acquisitions are made by reasoning about
the features across many possible unobserved re-
alizations in a stochastic latent space. Extensive
evaluation on a large range of synthetic and real
datasets demonstrates that our approach reliably
outperforms a diverse set of baselines.
1. Introduction
The standard supervised learning paradigm is to learn a pre-
dictive model using a training dataset of features and labels,
such that the model can make accurate predictions on unseen
test inputs. A fundamental assumption is that, at test time,
all features are jointly available. However, this assumption
does not always hold. Consider the example of a doctor di-
agnosing a patient (Kachuee et al., 2019b;a). Initially, there
is little to no information available and, while there are many
tests that could be conducted, the doctor will choose which
ones to carry out based on their current understanding of
the specific patient’s condition. For instance, if a patient has
pain in their leg, and the doctor suspects a fracture, a leg X-
1Department of Computer Science and Technology, University
of Cambridge, Cambridge, United Kingdom 2Department of Artifi-
cial Intelligence, Korea University, Seoul, Korea 3Department
of Statistics, University of Oxford, Oxford, United Kingdom
4Department of Applied Mathem...
Diffusion Adaptive Text Embedding for
Text-to-Image Diffusion Models
Byeonghu Na1
Minsang Park1
Gyuwon Sim1
Donghyeok Shin1
HeeSun Bae1
Mina Kang1
Se Jung Kwon2
Wanmo Kang1
Il-Chul Moon1,3
1KAIST, 2NAVER Cloud, 3summary.ai
{byeonghu.na,pagemu,gkwlaks4886,tlsehdgur0,cat2507,kasong13}@kaist.ac.kr,
sejung.kwon@navercorp.com , {wanmo.kang,icmoon}@kaist.ac.kr
Abstract
Text-to-image diffusion models rely on text embeddings from a pre-trained text
encoder, but these embeddings remain fixed across all diffusion timesteps, limiting
their adaptability to the generative process. We propose Diffusion Adaptive Text
Embedding (DATE), which dynamically updates text embeddings at each diffusion
timestep based on intermediate perturbed data. We formulate an optimization
problem and derive an update rule that refines the text embeddings at each sampling
step to improve alignment and preference between the mean predicted image and
the text. This allows DATE to dynamically adapts the text conditions to the
reverse-diffused images throughout diffusion sampling without requiring additional
model training. Through theoretical analysis and empirical results, we show that
DATE maintains the generative capability of the model while providing superior
text-image alignment over fixed text embeddings across various tasks, including
multi-concept generation and text-guided image editing. Our code is available at
https://github.com/aailab-kaist/DATE.
1
Introduction
Text-to-image generation has recently received significant attention due to its capability to generate
realistic and semantically accurate images from textual prompts. This progress has been largely driven
by diffusion models [18, 53], particularly with large-scale models such as DALL·E [45] and Stable
Diffusion [47]. These models use pre-trained text encoders like CLIP [42] and T5 [43] to encode
prompts into embeddings, providing crucial semantic information to diffusion models. Notably, the
quality and semantic alignment of the gener...
Training-Free Safe Text Embedding Guidance for
Text-to-Image Diffusion Models
Byeonghu Na1
Mina Kang1
Jiseok Kwak1
Minsang Park1
Jiwoo Shin1
SeJoon Jun1
Gayoung Lee2
Jin-Hwa Kim2,3
Il-Chul Moon1,4
1KAIST, 2NAVER AI Lab, 3SNU AIIS, 4summary.ai
{byeonghu.na,kasong13,jskwak,pagemu,natu33,sjmathy,icmoon}@kaist.ac.kr,
{gayoung.lee,j1nhwa.kim}@navercorp.com
Abstract
Text-to-image models have recently made significant advances in generating realis-
tic and semantically coherent images, driven by advanced diffusion models and
large-scale web-crawled datasets. However, these datasets often contain inappropri-
ate or biased content, raising concerns about the generation of harmful outputs when
provided with malicious text prompts. We propose Safe Text embedding Guidance
(STG), a training-free approach to improve the safety of diffusion models by guid-
ing the text embeddings during sampling. STG adjusts the text embeddings based
on a safety function evaluated on the expected final denoised image, allowing the
model to generate safer outputs without additional training. Theoretically, we show
that STG aligns the underlying model distribution with safety constraints, thereby
achieving safer outputs while minimally affecting generation quality. Experiments
on various safety scenarios, including nudity, violence, and artist-style removal,
show that STG consistently outperforms both training-based and training-free base-
lines in removing unsafe content while preserving the core semantic intent of input
prompts. Our code is available at https://github.com/aailab-kaist/STG.
Warning: This paper contains model-generated content that may be disturbing.
1
Introduction
Recent advances in text-to-image models have received considerable attention for their ability to
generate realistic images that semantically align with given text prompts [30, 32, 34, 44]. These
advances have largely been driven by the development of diffusion models [9, 33] and the avail-
ability of large-scale datasets...
Published as a conference paper at ICLR 2025
LOCALITY ALIGNMENT IMPROVES VISION-LANGUAGE
MODELS
Ian Covert, Tony Sun, James Zou∗, Tatsunori Hashimoto∗
Stanford University
{icovert, suntony, jamesz, thashim}@stanford.edu
ABSTRACT
Vision language models (VLMs) have seen growing adoption in recent years,
but many still struggle with basic spatial reasoning errors. We hypothesize that
this is due to VLMs adopting pre-trained vision backbones, specifically vision
transformers (ViTs) trained with image-level supervision and minimal inductive
biases. Such models may fail to encode the class contents at each position in
the image, and our goal is to resolve this with a vision backbone that effectively
captures both local and global image semantics. Our main insight is that we do
not require new supervision to learn this capability – pre-trained models contain
significant knowledge of local semantics that we can extract and use for scalable
self-supervision. We propose a new efficient post-training stage for ViTs called
locality alignment and a novel fine-tuning procedure called MaskEmbed that uses
a masked reconstruction loss to learn semantic contributions for each image patch.
We first evaluate locality alignment with a vision-only benchmark, finding that it
improves a model’s performance at patch-level semantic segmentation, especially
for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We
then train a series of VLMs with and without locality alignment, and show that
locality-aligned backbones improve performance across a range of benchmarks,
particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref,
TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local
semantic extraction via a locality alignment stage, and that this procedure benefits
VLM training recipes that use off-the-shelf vision backbones.
1
INTRODUCTION
Auto-regressive VLMs are an exciting new type of model that emerged in the last couple ...
Published as a conference paper at ICLR 2025
ROBUST SIMULATION-BASED INFERENCE
UNDER MISSING DATA VIA NEURAL PROCESSES
Yogesh Verma, Ayush Bharti
Department of Computer Science, Aalto University
{yogesh.verma, ayush.bharti}@aalto.fi
Vikas Garg
YaiYai Ltd and Aalto University
vgarg@csail.mit.edu
ABSTRACT
Simulation-based inference (SBI) methods typically require fully observed data to
infer parameters of models with intractable likelihood functions. However, datasets
often contain missing values due to incomplete observations, data corruptions
(common in astrophysics), or instrument limitations (e.g., in high-energy physics
applications). In such scenarios, missing data must be imputed before applying any
SBI method. We formalize the problem of missing data in SBI and demonstrate that
naive imputation methods can introduce bias in the estimation of SBI posterior. We
also introduce a novel amortized method that addresses this issue by jointly learning
the imputation model and the inference network within a neural posterior estimation
(NPE) framework. Extensive empirical results on SBI benchmarks show that our
approach provides robust inference outcomes compared to standard baselines
for varying levels of missing data. Moreover, we demonstrate the merits of our
imputation model on two real-world bioactivity datasets (Adrenergic and Kinase
assays). Code is available at https://github.com/Aalto-QuML/RISE.
1
INTRODUCTION
Mechanistic models for studying complex physical or biological phenomena have become indispens-
able tools in research fields as diverse as genetics (Riesselman et al., 2018), epidemiology (Kypraios
et al., 2017), gravitational wave astronomy (Dax et al., 2021), and radio propagation (Bharti et al.,
2022a). However, fitting such models to observational data can be challenging due to the intractabil-
ity of their likelihood functions, which renders standard Bayesian inference methods inapplicable.
Simulation-based inference (SBI) methods (Cranmer et al., 2...
On topological descriptors for graph products
Mattie Ji
University of Pennsylvania
mji13@sas.upenn.edu
Amauri H. Souza
Federal Institute of Ceará
amauriholanda@ifce.edu.br
Vikas Garg
Aalto University
YaiYai Ltd
vgarg@csail.mit.edu
Abstract
Topological descriptors have been increasingly utilized for capturing multiscale
structural information in relational data. In this work, we consider various filtrations
on the (box) product of graphs and the effect on their outputs on the topological de-
scriptors - the Euler characteristic (EC) and persistent homology (PH). In particular,
we establish a complete characterization of the expressive power of EC on general
color-based filtrations. We also show that the PH descriptors of (virtual) graph prod-
ucts contain strictly more information than the computation on individual graphs,
whereas EC does not. Additionally, we provide algorithms to compute the PH dia-
grams of the product of vertex- and edge-level filtrations on the graph product. We
also substantiate our theoretical analysis with empirical investigations on runtime
analysis, expressivity, and graph classification performance. Overall, this work
paves way for powerful graph persistent descriptors via product filtrations. Code is
available at https://github.com/Aalto-QuML/tda_graph_product.
1
Introduction
Message-passing GNNs are prominent models for graph representation learning [19]. However, they
are bounded in expressivity by the WL hierarchy [33–35, 50] and cannot compute fundamental graph
properties such as cycles or connected components [10, 18]. Topological descriptors (TDs) such
as those based on persistent homology (PH) and the Euler characteristics (EC) can provide such
information and thus are being increasingly employed to augment GNNs, boosting their empirical
performance [7–9, 24, 48, 51].
In this work, we study two prominent types of topological descriptors based on PH [1, 13] and EC
[40, 42, 46] respectively. Both descriptors are defined with respect ...
Private Evolution Converges
Tomas Gonzalez
Carnegie Mellon University
tcgonzal@andrew.cmu.edu
Giulia Fanti
Carnegie Mellon University
gfanti@andrew.cmu.edu
Aaditya Ramdas
Carnegie Mellon University
aramdas@cs.cmu.edu
Abstract
Private Evolution (PE) is a promising training-free method for differentially private
(DP) synthetic data generation. While it achieves strong performance in some
domains (e.g., images and text), its behavior in others (e.g., tabular data) is less
consistent. To date, the only theoretical analysis of the convergence of PE depends
on unrealistic assumptions about both the algorithm’s behavior and the structure
of the sensitive dataset. In this work, we develop a new theoretical framework
to understand PE’s practical behavior and identify sufficient conditions for its
convergence. For d-dimensional sensitive datasets with n data points from a
convex and compact domain, we prove that under the right hyperparameter settings
and given access to the Gaussian variation API proposed in [33], PE produces an
(ε, δ)-DP synthetic dataset with expected 1-Wasserstein distance ˜O(d(nε)−1/d)
from the original; this establishes worst-case convergence of the algorithm as
n →∞. Our analysis extends to general Banach spaces as well. We also connect
PE to the Private Signed Measure Mechanism, a method for DP synthetic data
generation that has thus far not seen much practical adoption. We demonstrate the
practical relevance of our theoretical findings in experiments.
1
Introduction
Many modern machine learning applications rely on user data, making data privacy protection a
central concern. In this regard, differential privacy (DP) [17, 19] has become a de facto standard for
safeguarding sensitive information of individuals. Given a private dataset, many problems—including
regression [40, 12], deep learning [1], and stochastic optimization [6, 2]—can be performed in a DP
manner. While it is possible to design a different DP algorithm for each specific task, a reason...