GenAI4ELab/papercli-papers huggingface dataset:
FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch
Authors: Virginia Aglietti; Ira Ktena; Jessica Schrouff; Eleni Sgouritsa; Francisco Ruiz; Alan Malek; Alexis Bellot; Silvia Chiappa
Venue: ICML 2025 (ICML 2025 poster)

Abstract: The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AFs can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a number of evaluations for a limited set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well both in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.
AffinityFlow: Guided Flows for Antibody Affinity Maturation
Authors: Can Chen; Karla-Luise Herpoldt; Chenchao Zhao; Zichen Wang; Marcus D. Collins; Shang Shang; Ron Benson
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Antibodies are widely used as therapeutics, but their development requires costly affinity maturation, involving iterative mutations to enhance binding affinity. This paper explores a sequence-only scenario for affinity maturation, using solely antibody and antigen sequences. Recently AlphaFlow wraps AlphaFold within flow matching to generate diverse protein structures, enabling a sequence-conditioned generative model of structure. Building on this, we propose an \textit{alternating optimization} framework that (1) fixes the sequence to guide structure generation toward high binding affinity using a structure-based predictor, then (2) applies inverse folding to create sequence mutations, refined by a sequence-based predictor. A key challenge is the lack of labeled data for training both predictors. To address this, we develop a \textit{co-teaching} module that incorporates valuable information from noisy biophysical energies into predictor refinement. The sequence-based predictor selects consensus samples to teach the structure-based predictor, and vice versa. Our method, \textit{AffinityFlow}, achieves state-of-the-art performance in proof-of-concept affinity maturation experiments.
Federated Learning for Feature Generalization with Convex Constraints
Authors: Dongwon Kim; Donghee Kim; Sung Kuk Shyn; Kwangsu Kim
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Federated learning (FL) often struggles with generalization due to heterogeneous client data. Local models are prone to overfitting their local data distributions, and even transferable features can be distorted during aggregation. To address these challenges, we propose FedCONST, an approach that adaptively modulates update magnitudes based on the global model’s parameter strength. This prevents over-emphasizing well-learned parameters while reinforcing underdeveloped ones. Specifically, FedCONST employs linear convex constraints to ensure training stability and preserve locally learned generalization capabilities during aggregation. A Gradient Signal-to-Noise Ratio (GSNR) analysis further validates FedCONST's effectiveness in enhancing feature transferability and robustness. As a result, FedCONST effectively aligns local and global objectives, mitigating overfitting and promoting stronger generalization across diverse FL environments, achieving state-of-the-art performance.
Multi-Objective Causal Bayesian Optimization
Authors: Shriya Bhatija; Paul-David Zuercher; Jakob Thumm; Thomas Bohné
Venue: ICML 2025 (ICML 2025 poster)

Abstract: In decision-making problems, the outcome of an intervention often depends on the causal relationships between system components and is highly costly to evaluate. In such settings, causal Bayesian optimization (CBO) exploits the causal relationships between the system variables and sequentially performs interventions to approach the optimum with minimal data. Extending CBO to the multi-outcome setting, we propose *multi-objective Causal Bayesian optimization* (MO-CBO), a paradigm for identifying Pareto-optimal interventions within a known multi-target causal graph. Our methodology first reduces the search space by discarding sub-optimal interventions based on the structure of the given causal graph. We further show that any MO-CBO problem can be decomposed into several traditional multi-objective optimization tasks. Our proposed MO-CBO algorithm is designed to identify Pareto-optimal interventions by iteratively exploring these underlying tasks, guided by relative hypervolume improvement. Experiments on synthetic and real-world causal graphs demonstrate the superiority of our approach over non-causal multi-objective Bayesian optimization in settings where causal information is available.
Continuous Semi-Implicit Models
Authors: Longlin Yu; Jiajun Zha; Tong Yang; Tianyu Xie; Xiangyu Zhang; S.-H. Chan; Cheng Zhang
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Semi-implicit distributions have shown great promise in variational inference and generative modeling.
    Hierarchical semi-implicit models, which stack multiple semi-implicit layers, enhance the expressiveness of semi-implicit distributions and can be used to accelerate diffusion models given pretrained score networks. 
    However, their sequential training often suffers from slow convergence.
    In this paper, we introduce CoSIM, a continuous semi-implicit model that extends hierarchical semi-implicit models into a continuous framework.
    By incorporating a continuous transition kernel, CoSIM enables efficient, simulation-free training.
    Furthermore, we show that CoSIM achieves consistency with a carefully designed transition kernel, offering a novel approach for multistep distillation of generative models at the distributional level.
    Extensive experiments on image generation demonstrate that CoSIM performs on par or better than existing diffusion model acceleration methods, achieving superior performance on FD-DINOv2.
A Unified Comparative Study with Generalized Conformity Scores for Multi-Output Conformal Regression
Authors: Victor Dheur; Matteo Fontana; Yorick Estievenart; Naomi Desobry; Souhaib Ben Taieb
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Conformal prediction provides a powerful framework for constructing distribution-free prediction regions with finite-sample coverage guarantees. While extensively studied in univariate settings, its extension to multi-output problems presents additional challenges, including complex output dependencies and high computational costs, and remains relatively underexplored. In this work, we present a unified comparative study of nine conformal methods with different multivariate base models for constructing multivariate prediction regions within the same framework. This study highlights their key properties while also exploring the connections between them. Additionally, we introduce two novel classes of conformity scores for multi-output regression that generalize their univariate counterparts. These scores ensure asymptotic conditional coverage while maintaining exact finite-sample marginal coverage. One class is compatible with any generative model, offering broad applicability, while the other is computationally efficient, leveraging the properties of invertible generative models. Finally, we conduct a comprehensive empirical evaluation across 13 tabular datasets, comparing all the multi-output conformal methods explored in this work. To ensure a fair and consistent comparison, all methods are implemented within a unified code base.
Learning the Electronic Hamiltonian of Large Atomic Structures
Authors: Chen Hao Xia; Manasa Kaniselvan; Alexandros Nikolaos Ziogas; Marko Mladenović; Rayen Mahjoub; Alexander Maeder; Mathieu Luisier
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Graph neural networks (GNNs) have shown promise in learning the ground-state electronic properties of materials, subverting *ab initio* density functional theory (DFT) calculations when the underlying lattices can be represented as small and/or repeatable unit cells (i.e., molecules and periodic crystals). Realistic systems are, however, non-ideal and generally characterized by higher structural complexity. As such, they require large (10+ Å) unit cells and thousands of atoms to be accurately described. At these scales, DFT becomes computationally prohibitive, making GNNs especially attractive. In this work, we present a strictly local equivariant GNN capable of learning the electronic Hamiltonian (**H**) of realistically extended materials. It incorporates an *augmented partitioning* approach that enables training on arbitrarily large structures while preserving local atomic environments beyond boundaries. We demonstrate its capabilities by predicting the electronic Hamiltonian of various systems with up to 3,000 nodes (atoms), 500,000+ edges, ~ 28 million orbital interactions (nonzero entries of **H**), and $\leq$0.53\% error in the eigenvalue spectra. Our work expands the applicability of current electronic property prediction methods to some of the most challenging cases encountered in computational materials science, namely systems with disorder, interfaces, and defects.
Topology-Aware Dynamic Reweighting for Distribution Shifts on Graph
Authors: Weihuang Zheng; Jiashuo Liu; Jiaxing Li; Jiayun Wu; Peng Cui; Youyong Kong
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Graph Neural Networks (GNNs) are widely used for node classification tasks but often fail to generalize when training and test nodes come from different distributions, limiting their practicality. To address this challenge, recent approaches have adopted invariant learning and sample reweighting techniques from the out-of-distribution (OOD) generalization field. However, invariant learning-based methods face difficulties when applied to graph data, as they rely on the impractical assumption of obtaining real environment labels and strict invariance, which may not hold in real-world graph structures. Moreover, current sample reweighting methods tend to overlook topological information, potentially leading to suboptimal results. In this work, we introduce the Topology-Aware Dynamic Reweighting (TAR) framework to address distribution shifts by leveraging the inherent graph structure. TAR dynamically adjusts sample weights through gradient flow on the graph edges during training. Instead of relying on strict invariance assumptions, we theoretically prove that our method is able to provide distributional robustness, thereby enhancing the out-of-distribution generalization performance on graph data. Our framework's superiority is demonstrated through standard testing on extensive node classification OOD datasets, exhibiting marked improvements over existing methods.
Benign Overfitting in Token Selection of Attention Mechanism
Authors: Keitaro Sakamoto; Issei Sato
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Attention mechanism is a fundamental component of the transformer model and plays a significant role in its success.
However, the theoretical understanding of how attention learns to select tokens is still an emerging area of research.
In this work, we study the training dynamics and generalization ability of the attention mechanism, under classification problems with label noise.
We show that, with the characterization of signal-to-noise ratio (SNR), the token selection of attention mechanism achieves ``benign overfitting'', i.e., maintaining high generalization performance despite fitting label noise.
Our work also demonstrates an interesting delayed acquisition of generalization after an initial phase of overfitting.
Finally, we provide experiments to support our theoretical analysis using both synthetic and real-world datasets.
When to retrain a machine learning model
Authors: Florence Regol; Leo Schwinn; Kyle Sprague; Mark Coates; Thomas Markovich
Venue: ICML 2025 (ICML 2025 poster)

Abstract: A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments, addressing classification tasks, show that the method consistently outperforms existing baselines on 7 datasets. We thoroughly assess its robustness to varying cost trade-off values and mis-specified cost trade-offs.
The Elicitation Game: Evaluating Capability Elicitation Techniques
Authors: Felix Hofstätter; Teun van der Weij; Jayden Teoh; Rada Djoneva; Henning Bartsch; Francis Rhys Ward
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Capability evaluations are required to understand and regulate AI systems that may
be deployed or further developed. Therefore, it is important that evaluations provide
an accurate estimation of an AI system’s capabilities. However, in numerous cases,
previously latent capabilities have been elicited from models, sometimes long
after initial release. Accordingly, substantial efforts have been made to develop
methods for eliciting latent capabilities from models. In this paper, we evaluate the
effectiveness of capability elicitation techniques by intentionally training model
organisms – language models with hidden capabilities that are revealed by a
password. We introduce a novel method for training model organisms, based
on circuit-breaking, which is more robust to elicitation techniques than standard
password-locked models. We focus on elicitation techniques based on prompting
and activation steering, and compare these to fine-tuning methods. Prompting
techniques can elicit the actual capability of both password-locked and circuit-
broken model organisms in an MCQA setting, while steering fails to do so. For
a code-generation task, only fine-tuning can elicit the hidden capabilities of our
novel model organism. Additionally, our results suggest that combining techniques
improves elicitation. Still, if possible, fine-tuning should be the method of choice
to improve the trustworthiness of capability evaluations.
WILTing Trees: Interpreting the Distance Between MPNN Embeddings
Authors: Masahiro Negishi; Thomas Gärtner; Pascal Welke
Venue: ICML 2025 (ICML 2025 poster)

Abstract: We investigate the distance function learned by message passing neural networks (MPNNs) in specific tasks, aiming to capture the _functional_ distance between prediction targets that MPNNs implicitly learn. 
This contrasts with previous work, which links MPNN distances on arbitrary tasks to _structural_ distances on graphs that ignore task-specific information. 
To address this gap, we distill the distance between MPNN embeddings into an interpretable graph distance. 
Our method uses optimal transport on the Weisfeiler Leman Labeling Tree (WILT), where the edge weights reveal subgraphs that strongly influence the distance between embeddings. 
This approach generalizes two well-known graph kernels and can be computed in linear time. 
Through extensive experiments, we demonstrate that MPNNs define the relative position of embeddings by focusing on a small set of subgraphs that are known to be functionally important in the domain.
Models of Heavy-Tailed Mechanistic Universality
Authors: Liam Hodgkinson; Zhichao Wang; Michael W. Mahoney
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Recent theoretical and empirical successes in deep learning, including the celebrated neural scaling laws, are punctuated by the observation that many objects of interest tend to exhibit some form of heavy-tailed or power law behavior. In particular, the prevalence of heavy-tailed spectral densities in Jacobians, Hessians, and weight matrices has led to the introduction of the concept of *heavy-tailed mechanistic universality* (HT-MU). Multiple lines of empirical evidence suggest a robust correlation between heavy-tailed metrics and model performance, indicating that HT-MU may be a fundamental aspect of deep learning efficacy. Here, we propose a general family of random matrix models---the *high-temperature Marchenko-Pastur (HTMP) ensemble*---to explore attributes that give rise to heavy-tailed behavior in trained neural networks. Under this model, spectral densities with power laws on (upper and lower) tails arise through a combination of three independent factors (complex correlation structures in the data; reduced temperatures during training; and reduced eigenvector entropy), appearing as an implicit bias in the model structure, and they can be controlled with an "eigenvalue repulsion'' parameter. Implications of our model on other appearances of heavy tails, including neural scaling laws, optimizer trajectories, and the five-plus-one phases of neural network training, are discussed.
Explaining the role of Intrinsic Dimensionality in Adversarial Training
Authors: Enes Altinisik; Safa Messaoud; Husrev Taha Sencar; Hassan Sajjad; Sanjay Chawla
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Adversarial Training (AT) impacts different architectures in distinct ways: vision models gain robustness but face reduced generalization, encoder-based models exhibit limited robustness improvements with minimal generalization loss, and recent work in latent-space adversarial training demonstrates that decoder-based models achieve improved robustness by applying AT across multiple layers.
We provide the first explanation for these trends by leveraging the manifold conjecture: off-manifold adversarial examples (AEs) enhance robustness, while on-manifold AEs improve generalization.
We show that vision and decoder-based models exhibit low intrinsic dimensionality in earlier layers (favoring off-manifold AEs), whereas encoder-based models do so in later layers (favoring on-manifold AEs). 
Exploiting this property, we introduce SMAAT, which improves the scalability of AT for encoder-based models by perturbing the layer with the lowest intrinsic dimensionality. This reduces the projected gradient descent (PGD) chain length required for AE generation, cutting GPU time by 25–33% while significantly boosting robustness. We validate SMAAT across multiple tasks, including text generation, sentiment classification, safety filtering, and retrieval augmented generation setups, demonstrating superior robustness with comparable generalization to standard training.
Fast Estimation of Partial Dependence Functions using Trees
Authors: Jinyang Liu; Tessa Steensgaard; Marvin N. Wright; Niklas Pfister; Munir Hiabu
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Many existing interpretation methods are based on Partial Dependence (PD) functions that, for a pre-trained machine learning model, capture how a subset of the features affects the predictions by averaging over the remaining features. Notable methods include Shapley additive explanations (SHAP) which computes feature contributions based on a game theoretical interpretation and PD plots (i.e., 1-dim PD functions) that capture average marginal main effects. Recent work has connected these approaches using a functional decomposition and argues that SHAP values can be misleading since they merge main and interaction effects into a single local effect. However, a major advantage of SHAP compared to other PD-based interpretations has been the availability of fast estimation techniques, such as `TreeSHAP`. In this paper, we propose a new tree-based estimator, `FastPD`, which efficiently estimates arbitrary PD functions. We show that `FastPD` consistently estimates the desired population quantity -- in contrast to path-dependent `TreeSHAP` which is inconsistent when features are correlated. For moderately deep trees, `FastPD` improves the complexity of existing methods from quadratic to linear in the number of observations. By estimating PD functions for arbitrary feature subsets, `FastPD` can be used to extract PD-based interpretations such as SHAP, PD plots and higher-order interaction effects.
Regret-Free Reinforcement Learning for Temporal Logic Specifications
Authors: R Majumdar; Mahmoud Salamati; Sadegh Soudjani
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Learning to control an unknown dynamical system with respect to high-level temporal specifications is an important problem in control theory. 
We present the first regret-free online algorithm for learning a controller for linear temporal logic (LTL) specifications for systems with unknown dynamics.
We assume that the underlying (unknown) dynamics is modeled by a finite-state and action Markov decision process (MDPs).
Our core technical result is a regret-free learning algorithm for infinite-horizon reach-avoid problems on MDPs.
For general LTL specifications, we show that the synthesis problem can be reduced to a reach-avoid problem once the graph structure is known.
Additionally, we provide an algorithm for learning the graph structure, assuming knowledge of a minimum transition probability, which operates independently of the main regret-free algorithm. Our LTL controller synthesis algorithm provides sharp bounds on how close we are to achieving optimal behavior after a finite number of learning episodes.
In contrast, previous algorithms for LTL synthesis only provide asymptotic guarantees, which give no insight into the transient performance during the learning phase.
DeFoG: Discrete Flow Matching for Graph Generation
Authors: Yiming QIN; Manuel Madeira; Dorina Thanou; Pascal Frossard
Venue: ICML 2025 (ICML 2025 oral)

Abstract: Graph generative models are essential across diverse scientific domains by capturing complex distributions over relational data. Among them, graph diffusion models achieve superior performance but face inefficient sampling and limited flexibility due to the tight coupling between training and sampling stages. We introduce DeFoG, a novel graph generative framework that disentangles sampling from training, enabling a broader design space for more effective and efficient model optimization. DeFoG employs a discrete flow-matching formulation that respects the inherent symmetries of graphs. We theoretically ground this disentangled formulation by explicitly relating the training loss to the sampling algorithm and showing that DeFoG faithfully replicates the ground truth graph distribution. Building on these foundations, we thoroughly investigate DeFoG's design space and propose novel sampling methods that significantly enhance performance and reduce the required number of refinement steps. Extensive experiments demonstrate state-of-the-art performance across synthetic, molecular, and digital pathology datasets, covering both unconditional and conditional generation settings. It also outperforms most diffusion-based models with just 5–10\% of their sampling steps.