GenAI4ELab/papercli-papers huggingface dataset:
Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation
Authors: Michal Lukasik; Lin Chen; Harikrishna Narasimhan; Aditya Krishna Menon; Wittawat Jitkrittum; Felix X. Yu; Sashank J. Reddi; Gang Fu; Mohammadhossein Bateni; Sanjiv Kumar
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem—loss aggregation and label aggregation—by characterizing their Bayes-optimal solutions. We show that while both approaches can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.
KernelBench: Can LLMs Write Efficient GPU Kernels?
Authors: Anne Ouyang; Simon Guo; Simran Arora; Alex L Zhang; William Hu; Christopher Re; Azalia Mirhoseini
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Efficient GPU kernels are crucial for building performant machine learning architectures, but writing them is a time-consuming challenge that requires significant expertise; therefore, we explore using language models (LMs) to automate kernel generation. We introduce **KernelBench**, an open-source framework for evaluating LMs' ability to write fast and correct kernels on a suite of 250 carefully selected PyTorch ML workloads. KernelBench represents a real-world engineering environment and making progress on the introduced benchmark directly translates to faster practical kernels. We introduce a new evaluation metric $\text{fast}_p$, which measures the percentage of generated kernels that are functionally correct and offer a speedup greater than an adjustable threshold $p$ over baseline. Our experiments across various state-of-the-art models and test-time methods show that frontier reasoning models perform the best out of the box but still fall short overall, matching the PyTorch baseline in less than 20\% of the cases. While we show that results can improve by leveraging execution and profiling feedback during iterative refinement, KernelBench remains a challenging benchmark, with its difficulty increasing as we raise speedup threshold $p$.
LLM-Augmented Chemical Synthesis and Design Decision Programs
Authors: Haorui Wang; Jeff Guo; Lingkai Kong; Rampi Ramprasad; Philippe Schwaller; Yuanqi Du; Chao Zhang
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Retrosynthesis, the process of breaking down a target molecule into simpler precursors through a series of valid reactions, stands at the core of organic chemistry and drug development. Although recent machine learning (ML) research has advanced single-step retrosynthetic modeling and subsequent route searches, these solutions remain restricted by the extensive combinatorial space of possible pathways. Concurrently, large language models (LLMs) have exhibited remarkable chemical knowledge, hinting at their potential to tackle complex decision-making tasks in chemistry. In this work, we explore whether LLMs can successfully navigate the highly constrained, multi-step retrosynthesis planning problem. We introduce an efficient scheme for encoding reaction pathways and present a new route-level search strategy, moving beyond the conventional step-by-step reactant prediction. Through comprehensive evaluations, we show that our LLM-augmented approach excels at retrosynthesis planning and extends naturally to the broader challenge of synthesizable molecular design.
Efficient Bisection Projection to Ensure Neural-Network Solution Feasibility for Optimization over General Set
Authors: Enming Liang; Minghua Chen
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Neural networks (NNs) have emerged as promising tools for solving constrained optimization problems in real-time. However, ensuring constraint satisfaction for NN-generated solutions remains challenging due to prediction errors. Existing methods to ensure NN feasibility either suffer from high computational complexity or are limited to specific constraint types.
We present Bisection Projection, an efficient approach to ensure NN solution feasibility for optimization over general compact sets with non-empty interiors.
Our method comprises two key components:
(i) a dedicated NN (called IPNN) that predicts interior points (IPs) with low eccentricity, which naturally accounts for approximation errors;
(ii) a bisection algorithm that leverages these IPs to recover solution feasibility when initial NN solutions violate constraints.
We establish theoretical guarantees by providing sufficient conditions for IPNN feasibility and proving bounded optimality loss of the bisection operation under IP predictions. 
Extensive evaluations on real-world non-convex problems demonstrate that Bisection Projection achieves superior feasibility and computational efficiency compared to existing methods, while maintaining comparable optimality gaps.
From RAG to Memory: Non-Parametric Continual Learning for Large Language Models
Authors: Bernal Jiménez Gutiérrez; Yiheng Shu; Weijian Qi; Sizhe Zhou; Yu Su
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Our ability to continuously acquire, organize, and leverage knowledge is a key feature of human intelligence that AI systems must approximate to unlock their full potential. Given the challenges in continual learning with large language models (LLMs), retrieval-augmented generation (RAG) has become the dominant way to introduce new information. However, its reliance on vector retrieval hinders its ability to mimic the dynamic and interconnected nature of human long-term memory. Recent RAG approaches augment vector
embeddings with various structures like knowledge graphs to address some of these gaps, namely sense-making and associativity. However, their performance on more basic factual memory tasks drops considerably below standard RAG. We address this unintended deterioration and propose HippoRAG 2, a framework that outperforms standard RAG comprehensively on factual, sense-making, and associative memory tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in HippoRAG and enhances it with deeper passage integration and more effective online use of an LLM. This combination pushes this RAG system closer to the effectiveness of human long-term memory, achieving a 7% improvement in associative memory tasks over the state-of-the-art embedding model while also exhibiting superior factual knowledge and sense-making memory capabilities. This work paves the way for non-parametric continual learning for LLMs. Code and data are available at https://github.com/OSU-NLP-Group/HippoRAG.
Pretraining Generative Flow Networks with Inexpensive Rewards for Molecular Graph Generation
Authors: Mohit Pandey; Gopeshh Subbaraj; Artem Cherkasov; Martin Ester; Emmanuel Bengio
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Generative Flow Networks (GFlowNets) have recently emerged as a suitable framework for generating diverse and high-quality molecular structures by learning from rewards treated as unnormalized distributions. Previous works in this framework often restrict exploration by using predefined molecular fragments as building blocks, limiting the chemical space that can be accessed. In this work, we introduce Atomic GFlowNets (A-GFNs), a foundational generative model leveraging individual atoms as building blocks to explore drug-like chemical space more comprehensively. We propose an unsupervised pre-training approach using drug-like molecule datasets, which teaches A-GFNs about inexpensive yet informative molecular descriptors such as drug-likeliness, topological polar surface area, and synthetic accessibility scores. These properties serve as proxy rewards, guiding A-GFNs towards regions of chemical space that exhibit desirable pharmacological properties. We further implement a goal-conditioned finetuning process, which adapts A-GFNs to optimize for specific target properties. In this work, we pretrain A-GFN on a subset of ZINC dataset, and by employing robust evaluation metrics we show the effectiveness of our approach when compared to other relevant baseline methods for a wide range of drug design tasks.  The code is accessible at https://github.com/diamondspark/AGFN.
Training High Performance Spiking Neural Network  by Temporal Model Calibration
Authors: Jiaqi Yan; Changping Wang; De Ma; Huajin Tang; Qian Zheng; Gang Pan
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Spiking Neural Networks (SNNs) are considered promising energy-efficient models due to their dynamic capability to process spatial-temporal spike information. Existing work has demonstrated that SNNs exhibit temporal heterogeneity, which leads to diverse outputs of SNNs at different time steps and has the potential to enhance their performance. Although SNNs obtained by direct training methods achieve state-of-the-art performance, current methods introduce limited temporal heterogeneity through the dynamics of spiking neurons or network structures. They lack the improvement of temporal heterogeneity through the lens of the gradient. In this paper, we first conclude that the diversity of the temporal logit gradients in current methods is limited. This leads to insufficient temporal heterogeneity and results in temporally miscalibrated SNNs with degraded performance. Based on the above analysis, we propose a Temporal Model Calibration (TMC) method, which can be seen as a logit gradient rescaling mechanism across time steps. Experimental results show that our method can improve the temporal logit gradient diversity and generate temporally calibrated SNNs with enhanced performance. In particular, our method achieves state-of-the-art accuracy on ImageNet, DVSCIFAR10, and N-Caltech101. Codes are available at https://github.com/zju-bmi-lab/TMC.
Unbiased Recommender Learning from Implicit Feedback via Weakly Supervised Learning
Authors: Hao Wang; Zhichao Chen; Haotian Wang; Yanchao Tan; Licheng Pan; Tianqiao Liu; Xu Chen; Haoxuan Li; Zhouchen Lin
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Implicit feedback recommendation is challenged by the missing negative feedback essential for effective model training. Existing methods often resort to negative sampling, a technique that assumes unlabeled interactions as negative samples. This assumption risks misclassifying potential positive samples within the unlabeled data, thereby undermining model performance. To address this issue, we introduce PURL, a model-agnostic framework that reframes implicit feedback recommendation as a weakly supervised learning task, eliminating the need for negative samples. However, its unbiasedness hinges on the accurate estimation of the class prior. To address this challenge, we propose Progressive Proximal Transport (PPT), which estimates the class prior by minimizing the proximal transport cost between positive and unlabeled samples. Experiments on three real-world datasets validate the efficacy of PURL in terms of improved recommendation quality.  Code is available at https://github.com/HowardZJU/weakrec.
Scalable First-order Method for Certifying Optimal k-Sparse GLMs
Authors: Jiachang Liu; Soroosh Shafiee; Andrea Lodi
Venue: ICML 2025 (ICML 2025 poster)

Abstract: This paper investigates the problem of certifying optimality for sparse generalized linear models (GLMs), where sparsity is enforced through an $\ell_0$ cardinality constraint. While branch-and-bound (BnB) frameworks can certify optimality by pruning nodes using dual bounds, existing methods for computing these bounds are either computationally intensive or exhibit slow convergence, limiting their scalability to large-scale problems. To address this challenge, we propose a first-order proximal gradient algorithm designed to solve the perspective relaxation of the problem within a BnB framework. Specifically, we formulate the relaxed problem as a composite optimization problem and demonstrate that the proximal operator of the non-smooth component can be computed exactly in log-linear time complexity, eliminating the need to solve a computationally expensive second-order cone program. Furthermore, we introduce a simple restart strategy that enhances convergence speed while maintaining low per-iteration complexity. Extensive experiments on synthetic and real-world datasets show that our approach significantly accelerates dual bound computations and is highly effective in providing optimality certificates for large-scale problems.
DataDecide: How to Predict Best Pretraining Data with Small Experiments
Authors: Ian Magnusson; Nguyen Tai; Ben Bogin; David Heineman; Jena D. Hwang; Luca Soldaini; Akshita Bhagia; Jiacheng Liu; Dirk Groeneveld; Oyvind Tafjord; Noah A. Smith; Pang Wei Koh; Jesse Dodge
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and methods of making decisions from observed performance at small scale most accurately predict the datasets that yield the best large models? To empower open exploration of this question, we release models, data, and evaluations in DataDecide—the most extensive open suite of models over differences in data and scale. We conduct controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds. We find that the ranking of models at a single, small size (e.g., 150M parameters) is a strong baseline for predicting best models at our larger target scale (1B) (∼ 80% of comparisons correct). No scaling law methods among 8 baselines exceed the compute-decision frontier of single-scale predictions, but DataDecide can measure improvement in future scaling laws. We also identify that using continuous likelihood metrics as proxies in small experiments makes benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval > 80% predictable at the target 1B scale with just 0.01% of the compute.
On the Query Complexity of Verifier-Assisted Language Generation
Authors: Edoardo Botta; Yuchen Li; Aashay Mehta; Jordan T. Ash; Cyril Zhang; Andrej Risteski
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Recently, a plethora of works have proposed
inference-time algorithms (e.g. best-of-n), which
incorporate verifiers to assist the generation process. Their quality-efficiency trade-offs have been
empirically benchmarked on a variety of constrained generation tasks, but the algorithmic design landscape is still largely poorly understood.
In this paper, we develop a mathematical framework for reasoning about constrained generation
using a pre-trained language model generator oracle and a process verifier—which can decide
whether a prefix can be extended to a string which
satisfies the constraints of choice. We show that
even in very simple settings, access to a verifier
can render an intractable problem (information-theoretically or computationally) to a tractable
one. In fact, we show even simple algorithms,
like tokenwise rejection sampling, can enjoy significant benefits from access to a verifier. Empirically, we show that a natural modification of
tokenwise rejection sampling, in which the sampler is allowed to "backtrack" (i.e., erase the final
few generated tokens) has robust and substantive
benefits over natural baselines (e.g. (blockwise)
rejection sampling, nucleus sampling)—both in
terms of computational efficiency, accuracy and
diversity.
Dimension-Free Adaptive Subgradient Methods with Frequent Directions
Authors: Sifan Yang; Yuanyu Wan; Peijia Li; Yibo Wang; Xiao Zhang; Zhewei Wei; Lijun Zhang
Venue: ICML 2025 (ICML 2025 poster)

Abstract: In this paper, we investigate the acceleration of adaptive subgradient methods through frequent directions (FD), a widely-used matrix sketching technique. The state-of-the-art regret bound exhibits a _linear_ dependence on the dimensionality $d$, leading to unsatisfactory guarantees for high-dimensional problems.  Additionally, it  suffers from an $O(\tau^2 d)$ time complexity per round, which scales quadratically with the sketching size $\tau$.  To overcome these issues, we first propose an algorithm named FTSL, achieving a tighter regret bound that is independent of the dimensionality. The key idea is to integrate FD with adaptive subgradient methods under _the primal-dual framework_ and add the cumulative discarded information of FD back.   To reduce its time complexity, we further utilize fast FD to expedite FTSL, yielding a better complexity of $O(\tau d)$ while maintaining the same regret bound. Moreover, to mitigate the computational cost for optimization problems involving matrix variables (e.g., training neural networks), we adapt FD to Shampoo, a popular optimization algorithm that accounts for the structure of decision, and give a novel analysis under  _the primal-dual framework_. Our proposed method obtains an improved dimension-free regret bound.  Experimental results have verified the efficiency and effectiveness of our approaches.
SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecasting
Authors: Yitian Zhang; Liheng Ma; Antonios Valkanas; Boris N. Oreshkin; Mark Coates
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Koopman operator theory provides a framework for nonlinear dynamical system analysis and time-series forecasting by mapping dynamics to a space of real-valued measurement functions, enabling a linear operator representation. Despite the advantage of linearity, the operator is generally infinite-dimensional. Therefore, the objective is to learn measurement functions that yield a tractable finite-dimensional Koopman operator approximation. In this work, we establish a connection between Koopman operator approximation and linear Recurrent Neural Networks (RNNs), which have recently demonstrated remarkable success in sequence modeling. We show that by considering an extended state consisting of lagged observations, we can establish an equivalence between a structured Koopman operator and linear RNN updates. Building on this connection, we present SKOLR, which integrates a learnable spectral decomposition of the input signal with a multilayer perceptron (MLP) as the measurement functions and implements a structured Koopman operator via a highly parallel linear RNN stack. Numerical experiments on various forecasting benchmarks and dynamical systems show that this streamlined, Koopman-theory-based design delivers exceptional performance. Our code is available at: https://github.com/networkslab/SKOLR.
Stochastic Deep Restoration Priors for Imaging Inverse Problems
Authors: Yuyang Hu; Albert Peng; Weijie Gan; Peyman Milanfar; Mauricio Delbracio; Ulugbek S. Kamilov
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Deep neural networks trained as image denoisers are widely used as priors for solving imaging inverse problems. We introduce Stochastic deep Restoration Priors (ShaRP), a novel framework that stochastically leverages an ensemble of deep restoration models beyond denoisers to regularize inverse problems. By using generalized restoration models trained on a broad range of degradations beyond simple Gaussian noise, ShaRP effectively addresses structured artifacts and enables self-supervised training without fully sampled data. We prove that ShaRP minimizes an objective function involving a regularizer derived from the score functions of minimum mean square error (MMSE) restoration operators. We also provide theoretical guarantees for learning restoration operators from incomplete measurements. ShaRP achieves state-of-the-art performance on tasks such as magnetic resonance imaging reconstruction and single-image super-resolution,  surpassing both denoiser- and diffusion-model-based methods without requiring retraining.
Multi-Armed Bandits with Interference: Bridging Causal Inference and Adversarial Bandits
Authors: Su Jia; Peter I. Frazier; Nathan Kallus
Venue: ICML 2025 (ICML 2025 poster)

Abstract: Experimentation with interference poses a significant challenge in contemporary online platforms. Prior research on experimentation with interference has concentrated on the final output of a policy. Cumulative performance, while equally important, is less well understood. To address this gap, we introduce the problem of Multi-armed Bandits with Interference (MABI), where the learner assigns an arm to each of $N$ experimental units over $T$ rounds. The reward of each unit in each round depends on the treatments of all units, where the interference between two units decays in their distance. The reward functions are chosen by an adversary and may vary arbitrarily over time and across different units. We first show that the optimal expected regret (against the best fixed-arm policy) is $\tilde O(\sqrt T)$, and can be achieved by a switchback policy. However, the regret (as a random variable) for any switchback policy suffers a high variance, since it does not account for $N$. We propose a policy based on a novel clustered randomization scheme, whose regret (i) is optimal in  expectation and (ii) admits a high probability bound that vanishes in $N$.
Leveraging Per-Instance Privacy for Machine Unlearning
Authors: Nazanin Mohammadi Sepahvand; Anvith Thudi; Berivan Isik; Ashmita Bhattacharyya; Nicolas Papernot; Eleni Triantafillou; Daniel M. Roy; Gintare Karolina Dziugaite
Venue: ICML 2025 (ICML 2025 poster)

Abstract: We present a principled, per-instance approach to quantifying the difficulty of unlearning via fine-tuning. We begin by sharpening an analysis of noisy gradient descent for unlearning (Chien et al., 2024), obtaining a better utility–unlearning trade-off by replacing worst-case privacy loss bounds with per-instance privacy losses (Thudi et al., 2024), each of which bounds the (R ´enyi) divergence to retraining without an individual datapoint. To demonstrate the practical applicability of our theory, we present empirical results showing that our theoretical predictions are born out both for Stochastic Gradient Langevin Dynamics (SGLD) as well as for standard fine-tuning without explicit noise. We further demonstrate that per-instance privacy losses correlate well with several existing data difficulty metrics, while also
identifying harder groups of data points, and introduce novel evaluation methods based on loss barriers. All together, our findings provide a foundation for more efficient and adaptive unlearning strategies tailored to the unique properties of individual data points.