GenAI4ELab/papercli-papers huggingface dataset:
Private Lossless Multiple Release
Authors: Joel Daniel Andersson; Lukas Retschmeier; Boel Nelson; Rasmus Pagh
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Koufogiannis et al. (2016) showed a $\textit{gradual release}$ result for Laplace noise-based differentially private mechanisms: given an $\varepsilon$-DP release, a new release with privacy parameter $\varepsilon' > \varepsilon$ can be computed such that the combined privacy loss of both releases is at most $\varepsilon'$ and the distribution of the latter is the same as a single release with parameter $\varepsilon'$.
They also showed gradual release techniques for Gaussian noise, later also explored by Whitehouse et al. (2022).

In this paper, we consider a more general $\textit{multiple release}$ setting in which analysts hold private releases with different privacy parameters corresponding to different access/trust levels.
These releases are determined one by one, with privacy parameters in arbitrary order.
A multiple release is $\textit{lossless}$ if having access to a subset $S$ of the releases has the same privacy guarantee as the least private release in $S$, and each release has the same distribution as a single release with the same privacy parameter.
Our main result is that lossless multiple release is possible for a large class of additive noise mechanisms.
For the Gaussian mechanism we give a simple method for lossless multiple release with a short, self-contained analysis that does not require knowledge of the mathematics of Brownian motion.
We also present lossless multiple release for the Laplace and Poisson mechanisms.
Finally, we consider how to efficiently do gradual release of sparse histograms, and present a mechanism with running time independent of the number of dimensions.
NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric
Authors: Jingtan Wang; Xiaoqiang Lin; Rui Qiao; Pang Wei Koh; Chuan-Sheng Foo; Bryan Kian Hsiang Low
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Curating data for instruction tuning is crucial for enhancing the performance of large language models (LLMs). This work aims to select training data for instruction tuning to improve the LLM performance on specific tasks. Existing methods often rely on next-token prediction (NTP) loss as a proxy for target task performance due to the non-differentiable nature of performance evaluation metrics. They select training data points that are most helpful in reducing validation loss. However, there is a discrepancy between minimizing NTP loss and maximizing performance (e.g., code pass rate in code generation). To remedy this, we introduce a novel Non-differentiable evaluation metric-based InfluenCe Estimation (NICE), which leverages the policy gradient to select the training data that improves the performance. Moreover, NICE can perform data selection in the absence of labels (ground-truth responses) when the evaluation metrics do not require labels (e.g., a reward model can output reward scores without supervision from labels). Experimental results show that our approach outperforms existing data selection baselines that use NTP loss in diverse and realistic scenarios. Notably, subsets selected by NICE often produce models that outperform those trained on the full dataset. Our code is available at https://github.com/JTWang2000/NICE.
Aligning LLMs by Predicting Preferences from User Writing Samples
Authors: Stéphane Aroca-Ouellette; Natalie Mackraz; Barry-John Theobald; Katherine Metcalf
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting as writing agents to infer a description of user preferences. Agent alignment then comes from conditioning on the inferred preference description. However, existing methods often produce generic preference descriptions that fail to capture the unique and individualized nature of human preferences. This paper introduces PROSE, a method designed to enhance the precision of preference descriptions inferred from user writing samples. PROSE incorporates two key elements: (1) iterative refinement of inferred preferences, and (2) verification of inferred preferences across multiple user writing samples. We evaluate PROSE with several LLMs (i.e., Qwen2.5 7B and 72B Instruct, GPT-mini, and GPT-4o) on a summarization and an email writing task. We find that PROSE more accurately infers nuanced human preferences, improving the quality of the writing agent's generations over CIPHER (a state-of-the-art method for inferring preferences) by 33\%. Lastly, we demonstrate that ICL and PROSE are complementary methods, and combining them provides up to a 9\% improvement over ICL alone. Code: https://github.com/apple/ml-predict
Learning Representations of Instruments for Partial Identification of Treatment Effects
Authors: Jonas Schweisthal; Dennis Frauen; Maresa Schröder; Konstantin Hess; Niki Kilbertus; Stefan Feuerriegel
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Reliable estimation of treatment effects from observational data is important in many disciplines such as medicine. However, estimation is challenging when unconfoundedness as a standard assumption in the causal inference literature is violated. In this work, we leverage arbitrary (potentially high-dimensional) instruments to estimate bounds on the conditional average treatment effect (CATE). Our contributions are three-fold: (1) We propose a novel approach for partial identification through a mapping of instruments to a discrete representation space so that we yield valid bounds on the CATE. This is crucial for reliable decision-making in real-world applications. (2) We derive a two-step procedure that learns tight bounds using a tailored neural partitioning of the latent instrument space. As a result, we avoid instability issues due to numerical approximations or adversarial training. Furthermore, our procedure aims to reduce the estimation variance in finite-sample settings to yield more reliable estimates. (3) We show theoretically that our procedure obtains valid bounds while reducing estimation variance. We further perform extensive experiments to demonstrate the effectiveness across various settings. Overall, our procedure offers a novel path for practitioners to make use of potentially high-dimensional instruments (e.g., as in Mendelian randomization).
Exact Upper and Lower Bounds for the Output Distribution of Neural Networks with Random Inputs
Authors: Andrey Kofnov; Daniel Kapla; Ezio Bartocci; Efstathia Bura
Venue: ICML 2025 (ICML 2025 poster)

Abstract: We derive exact upper and lower bounds for the cumulative distribution function (cdf)  of the output of a neural network (NN) over its entire support subject to noisy (stochastic) inputs. The upper and lower bounds converge to the true cdf over its domain as the resolution increases. Our method applies to any feedforward NN using continuous monotonic piecewise twice continuously differentiable activation functions (e.g.,  ReLU, tanh and softmax) and convolutional NNs, which were beyond the scope of competing approaches. The novelty and instrumental tool of our approach is to bound general NNs with ReLU NNs. The ReLU NN-based bounds are then used to derive the upper and lower bounds of the cdf of the NN output. Experiments demonstrate that our method delivers guaranteed bounds of the predictive output distribution over its support, thus providing exact error guarantees, in contrast to competing approaches.
Strong and Weak Identifiability of Optimization-based Causal Discovery in Non-linear Additive Noise Models
Authors: Mingjia Li; Hong Qian; Tian-Zuo Wang; ShujunLi; Min Zhang; Aimin Zhou
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Causal discovery aims to identify causal relationships from observational data. Recently, optimization-based causal discovery methods have attracted extensive attention in the literature due to their efficiency in handling high-dimensional problems. However, we observe that optimization-based methods often perform well on certain problems but struggle with others. This paper identifies a specific characteristic of causal structural equations that determines the difficulty of identification in causal discovery and, in turn, the performance of optimization-based methods. We conduct an in-depth study of the additive noise model (ANM) and propose to further divide identifiable problems into strongly and weakly identifiable types based on the difficulty of identification. We also provide a sufficient condition to distinguish the two categories. Inspired by these findings, this paper further proposes GENE, a generic method for addressing strongly and weakly identifiable problems in a unified way under the ANM assumption. GENE adopts an order-based search framework that incorporates conditional independence tests into order fitness evaluation, ensuring effectiveness on weakly identifiable problems. In addition, GENE restricts the dimensionality of the effect variables to ensure \emph{scale invariance}, a property crucial for practical applications. Experiments demonstrate that GENE is uniquely effective in addressing weakly identifiable problems while also remaining competitive with state-of-the-art causal discovery algorithms for strongly identifiable problems.
Revisiting Non-Acyclic GFlowNets in Discrete Environments
Authors: Nikita Morozov; Ian Maksimov; Daniil Tiapkin; Sergey Samsonov
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects from a given probability distribution, potentially known up to a normalizing constant. Instead of working in the object space, GFlowNets proceed by sampling trajectories in an appropriately constructed directed acyclic graph environment, greatly relying on the acyclicity of the graph. In our paper, we revisit the theory that relaxes the acyclicity assumption and present a simpler theoretical framework for non-acyclic GFlowNets in discrete environments. Moreover, we provide various novel theoretical insights related to training with fixed backward policies, the nature of flow functions, and connections between entropy-regularized RL and non-acyclic GFlowNets, which naturally generalize the respective concepts and theoretical results from the acyclic setting. In addition, we experimentally re-examine the concept of loss stability in non-acyclic GFlowNet training, as well as validate our own theoretical findings.
Fully Dynamic Euclidean Bi-Chromatic Matching in Sublinear Update Time
Authors: Gramoz Goranci; Peter Kiss; Neel Patel; Martin P. Seybold; Eva Szilagyi; Da Wei Zheng
Venue: ICML 2025 (ICML 2025 oral)

Abstract: We consider the Euclidean bi-chromatic matching problem in the dynamic setting, where the goal is to efficiently process point insertions and deletions while maintaining a high-quality solution. Computing the minimum cost bi-chromatic matching is one of the core problems in geometric optimization that has found many applications, most notably in estimating Wasserstein distance between two distributions. In this work, we present the first fully dynamic algorithm for Euclidean bi-chromatic matching with sublinear update time. For any fixed $\varepsilon > 0$, our algorithm achieves $O(1/\varepsilon)$-approximation and handles updates in $O(n^{\varepsilon})$ time.  Our experiments show that our algorithm enables effective monitoring of the distributional drift in the Wasserstein distance on real and synthetic data sets, while outperforming the runtime of baseline approximations by orders of magnitudes.
Quantifying Treatment Effects: Estimating Risk Ratios via Observational Studies
Authors: Ahmed BOUGHDIRI; Julie Josse; Erwan Scornet
Venue: ICML 2025 (ICML 2025 poster)

Abstract: The Risk Difference (RD), an absolute measure of effect, is widely used and well-studied in both randomized controlled trials (RCTs) and observational studies. Complementary to the RD, the Risk Ratio (RR), as a relative measure, is critical for a comprehensive understanding of intervention effects: RD can downplay small absolute changes, while RR can highlight them. Despite its significance, the theoretical study of RR has received less attention, particularly in observational settings. This paper addresses this gap by tackling the estimation of RR in observational data. We propose several RR estimators and establish their theoretical properties, including asymptotic normality and confidence intervals. Through analyses on simulated and real-world datasets, we evaluate the performance of these estimators in terms of bias, efficiency, and robustness to generative data models. We also examine the coverage and length of the associated confidence intervals. Due to the non-linear nature of RR, influence function theory yields two distinct efficient estimators with different convergence assumptions. Based on theoretical and empirical insights, we recommend, among all estimators, one of the two doubly-robust estimators, which, intriguingly, challenges conventional expectations.
AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling
Authors: Alexander Capstick; Rahul Krishnan; Payam Barnaghi
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Large language models (LLMs) acquire a breadth of information across various domains. However, their computational complexity, cost, and lack of transparency often hinder their direct application for predictive tasks where privacy and interpretability are paramount. In fields such as healthcare, biology, and finance, specialised and interpretable linear models still hold considerable value. In such domains, labelled data may be scarce or expensive to obtain. Well-specified prior distributions over model parameters can reduce the sample complexity of learning through Bayesian inference; however, eliciting expert priors can be time-consuming. We therefore introduce AutoElicit to extract knowledge from LLMs and construct priors for predictive models. We show these priors are informative and can be refined using natural language. We perform a careful study contrasting AutoElicit with in-context learning and demonstrate how to perform model selection between the two methods. We find that AutoElicit yields priors that can substantially reduce error over uninformative priors, using fewer labels, and consistently outperform in-context learning. We show that AutoElicit saves over 6 months of labelling effort when building a new predictive model for urinary tract infections from sensor recordings of people living with dementia.
Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based 	Stochastic Controlled Weight Averaging
Authors: Junkang Liu; Yuanyuan Liu; Fanhua Shang; Hongying Liu; Jin Liu; Wei Feng
Venue: ICML 2025 (ICML 2025 poster)

Abstract: For federated learning (FL) algorithms such as FedSAM, their generalization capability is crucial for real-word applications. In this paper, we revisit the generalization problem in FL and investigate the impact of data heterogeneity on FL generalization. We find that FedSAM usually performs worse than FedAvg in the case of highly heterogeneous data, and thus propose a novel and effective federated learning algorithm with Stochastic Weight Averaging (called \texttt{FedSWA}), which aims to find flatter minima in the setting of highly heterogeneous data. Moreover, we introduce a new momentum-based stochastic controlled weight averaging FL algorithm (\texttt{FedMoSWA}), which is designed to better align local and global models. 
 Theoretically, we provide both convergence analysis and generalization bounds for \texttt{FedSWA} and \texttt{FedMoSWA}. We also prove that the optimization and generalization errors of \texttt{FedMoSWA} are smaller than those of their counterparts, including FedSAM and its variants. Empirically, experimental results on CIFAR10/100 and Tiny ImageNet demonstrate the superiority of the proposed algorithms compared to their counterparts.
Point-Level Topological Representation Learning on Point Clouds
Authors: Vincent Peter Grande; Michael T Schaub
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Topological Data Analysis (TDA) allows us to extract powerful topological and higher-order information on the global shape of a data set or point cloud.
Tools like Persistent Homology give a single complex description of the global structure of the point cloud.
However, common machine learning applications like classification require point-level information and features.
In this paper, we bridge this gap and propose a novel method to extract node-level topological features from complex point clouds using discrete variants of concepts from algebraic topology and differential geometry.
We verify the effectiveness of these topological point features (TOPF) on both synthetic and real-world data and study their robustness under noise and heterogeneous sampling.
A Hitchhiker's Guide to Scaling Law Estimation
Authors: Leshem Choshen; Yang Zhang; Jacob Andreas
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Scaling laws predict the loss of a target machine learning model by extrapolating
from easier-to-train models with fewer parameters or smaller training sets. This
provides an efficient way for practitioners and researchers alike to compare pre-
training decisions involving optimizers, datasets, and model architectures. Despite
the widespread use of scaling laws to model the dynamics of language model
training, there has been little work on understanding how to best estimate and
interpret them. We collect (and release) a large-scale dataset containing losses and
downstream evaluations for 485 previously published pretrained models. We use
these to estimate more than 1000 scaling laws, then derive a set of best practices
for estimating scaling laws in new model families. We find that fitting scaling laws
to intermediate checkpoints of training runs (and not just their final losses) substan-
tially improves accuracy, and that—all else equal—estimates of performance are
generally most accurate when derived from other models of similar sizes. However,
because there is a significant degree of variability across model seeds, training
multiple small models is sometimes more useful than training a single large one.
Moreover, while different model families differ in scaling behavior, they are often
similar enough that a target model’s behavior can be predicted from a single model
with the same architecture, along with scaling parameter estimates derived from
other model families.
iN2V: Bringing Transductive Node Embeddings to Inductive Graphs
Authors: Nicolas Lell; Ansgar Scherp
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Shallow node embeddings like node2vec (N2V) can be used for nodes without features or to supplement existing features with structure-based information.
Embedding methods like N2V are limited in their application on new nodes, which restricts them to the transductive setting where the entire graph, including the test nodes, is available during training.
We propose inductive node2vec (iN2V), which combines a post-hoc procedure to compute embeddings for nodes unseen during training and modifications to the original N2V training procedure to prepare the embeddings for this post-hoc procedure.
We conduct experiments on several benchmark datasets and demonstrate that iN2V is an effective approach to bringing transductive embeddings to an inductive setting.
Using iN2V embeddings improves node classification by 1 point on average, with up to 6 points of improvement depending on the dataset and the number of unseen nodes.
Our iN2V is a plug-in approach to create new or enrich existing embeddings. 
It can also be combined with other embedding methods, making it a versatile approach for inductive node representation learning. 
Code to reproduce the results is available at https://github.com/Foisunt/iN2V.
FLAM: Frame-Wise Language-Audio Modeling
Authors: Yusong Wu; Christos Tsirigotis; Ke Chen; Cheng-Zhi Anna Huang; Aaron Courville; Oriol Nieto; Prem Seetharaman; Justin Salamon
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Recent multi-modal audio-language models (ALMs) excel at text-audio retrieval but struggle with frame-wise audio understanding. Prior works use temporal-aware labels or unsupervised training to improve frame-wise capabilities, but they still lack fine-grained labeling capability to pinpoint when an event occurs. While traditional sound event detection models can precisely localize events, they are limited to pre-defined categories, making them ineffective for real-world scenarios with out-of-distribution events. In this work, we introduce FLAM, an open-vocabulary contrastive audio-language model capable of localizing specific sound events. FLAM employs a memory-efficient and calibrated frame-wise objective with logit adjustment to address spurious correlations, such as event dependencies and label imbalances during training. To enable frame-wise supervision, we leverage a large-scale dataset with diverse audio events, LLM-generated captions and simulation. Experimental results and case studies demonstrate that FLAM significantly improves the open-vocabulary localization capability while maintaining strong performance in global retrieval and downstream tasks.
Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning
Authors: Fengqiang Wan; Yang Yang
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Incremental learning (IL) aims to sequentially learn new tasks while mitigating catastrophic forgetting. Among various IL strategies, parameter-isolation methods stand out by using mask techniques to allocate distinct parameters to each task, explicitly addressing forgetting. However, existing approaches often disregard parameter dependencies, resulting in an over-reliance on newly allocated parameters. To address this issue, we propose Probabilistic Group Mask selection (PGM), a group-wise approach that captures parameter dependencies by exploring candidate masks within each group. Specifically, PGM partitions parameters into groups with multiple candidate masks, assigning probabilities to these masks and leveraging Gumbel-Softmax for differentiable sampling, enabling efficient optimization of the discrete mask selection process. Our theoretical analysis demonstrates that incorporating parameter dependencies enhances sub-network selection. Experiments conducted on standard benchmarks confirm its superior effectiveness compared to existing IL approaches. The source code is available at: \url{https://github.com/njustkmg/ICML25-PGM}.
The Value of Prediction in Identifying the Worst-Off
Authors: Unai Fischer-Abaigar; Christoph Kern; Juan Carlos Perdomo
Venue: ICML 2025 (ICML 2025 oral)

Abstract: Machine learning is increasingly used in government programs to identify and support the most vulnerable individuals, prioritizing assistance for those at greatest risk over optimizing aggregate outcomes. This paper examines the welfare impacts of prediction in equity-driven contexts, and how they compare to other policy levers, such as expanding bureaucratic capacity. Through mathematical models and a real-world case study on long-term unemployment amongst German residents, we develop a comprehensive understanding of the relative effectiveness of prediction in surfacing the worst-off. Our findings provide clear analytical frameworks and practical, data-driven tools that empower policymakers to make principled decisions when designing these systems.