GenAI4ELab/papercli-papers huggingface dataset:
GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
Authors: Simin Fan; Maria Ios Glarou; Martin Jaggi
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora.
Existing domain reweighting algorithms primarily optimize data mixtures for a single target task, thereby resulting in models that overfit to specialized objectives while exhibiting substantial performance degradation on other benchmarks.
This paper introduces $\textbf{G}$roup $\textbf{R}$obust Multi-target $\textbf{A}$daptive $\textbf{P}$r$\textbf{E}$training (GRAPE), a novel multi-source-multi-target domain reweighting framework designed to calibrate pretraining data mixtures for robust performance across multiple target tasks simultaneously.
GRAPE dynamically adjusts sampling weights across source domains ($\textit{domain weights}$) while concurrently modulating $\textit{task weights}$ that quantify the relative importance of each individual target task.
This adaptive process prioritizes tasks based on their learning difficulty throughout training. 
We formulate this interleaved reweighting mechanism as a minimax optimization problem: 
The inner maximization adjusts task weights leveraging group distributed-robust-optimization (DRO), where those tasks demonstrating the least improvement under the current data mixture are prioritized with higher weights; 
The outer minimization then optimizes domain weights to maximize loss reduction on the prioritized tasks.
Experiments on $\texttt{ClimbLab}$ and $\texttt{SlimPajama}$ datasets demonstrate that GRAPE consistently outperforms baseline methods in terms of reasoning accuracies across 6 benchmarks. 
Furthermore, when applied to multilingual targets, GRAPE effectively identifies optimal training mixtures from mainstream languages, achieving superior language modeling capabilities across 8 low-resource target languages.
Two-Steps Diffusion Policy for Robotic Manipulation via Genetic Denoising
Authors: Mateo Clémente; Leo Maxime Brunswic; Rui Heng Yang; Xuan Zhao; Yasser H. Khalil; Haoyu LEI; Amir Rasouli; Yinchuan Li
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Diffusion models, such as diffusion policy, have achieved state-of-the-art results in robotic manipulation by imitating expert demonstrations. While diffusion models were originally developed for vision tasks like image and video generation, many of their inference strategies have been directly transferred to control domains without adaptation. In this work, we show that by tailoring the denoising process to the specific characteristics of embodied AI tasks—particularly the structured, low-dimensional nature of action distributions---diffusion policies can operate effectively with as few as 5 neural function evaluations (NFE).
Building on this insight, we propose a population-based sampling strategy, genetic denoising, which enhances both performance and stability by selecting denoising trajectories with low out-of-distribution risk. Our method solves challenging tasks with only 2 NFE while improving or matching performance. We evaluate our approach across 14 robotic manipulation tasks from D4RL and Robomimic, spanning multiple action horizons and inference budgets. In over 2 million evaluations, our method consistently outperforms standard diffusion-based policies, achieving up to 20\% performance gains with significantly fewer inference steps.
Spectral Analysis of Diffusion Models with Application to Schedule Design
Authors: Roi Benita; Michael Elad; Joseph Keshet
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Diffusion models (DMs) have emerged as powerful  tools for modeling complex data distributions and generating realistic new samples. Over the years, advanced architectures and sampling methods have been developed to make these models practically usable. However, certain synthesis process decisions still rely on heuristics without a solid theoretical foundation. 

In our work, we offer a novel analysis of the DM's inference process, introducing a comprehensive frequency response perspective. Specifically, by relying on Gaussianity assumption, we present the inference process as a closed-form spectral transfer function, capturing how the generated signal evolves in response to the initial noise. We demonstrate how the proposed analysis can be leveraged to design a noise schedule that aligns effectively with the characteristics of the data. The spectral perspective also provides insights into the underlying dynamics and sheds light on the relationship between spectral properties and noise schedule structure. Our results lead to scheduling curves that are dependent on the spectral content of the data, offering a theoretical justification for some of the heuristics taken by practitioners.
Efficient semantic uncertainty quantification in language models via diversity-steered sampling
Authors: Ji Won Park; Kyunghyun Cho
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Accurately estimating *semantic* aleatoric and epistemic uncertainties in large language models (LLMs) is particularly challenging in free-form question answering (QA), where obtaining stable estimates often requires many expensive generations. We introduce a **diversity-steered sampler** that discourages semantically redundant outputs during decoding, covers both autoregressive and masked diffusion paradigms, and yields substantial sample-efficiency gains. The key idea is to inject a continuous semantic-similarity penalty into the model’s proposal distribution using a natural language inference (NLI) model lightly finetuned on partial prefixes or intermediate diffusion states. We debias downstream uncertainty estimates with importance reweighting and shrink their variance with control variates. Across four QA benchmarks, our method matches or surpasses baselines while covering more semantic clusters with the same number of samples. Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation in risk-sensitive model deployments.
Generating Physically Sound Designs from Text and a Set of Physical Constraints
Authors: Gregory Barber; Todd Henry; Mulugeta A Haile
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We present TIDES, a text informed design approach for generating physically sound designs based on a textual description and a set of physical constraints. TIDES jointly optimizes structural (topology) and visual properties. A pre-trained text-image model is used to measure the design's visual alignment with a text prompt and a differentiable physics simulator is used to measure its physical performance. We evaluate TIDES on a series of structural optimization problems operating under different load and support conditions, at different resolutions, and experimentally in the lab by performing the 3-point bending test on 2D beam designs that are extruded and 3D printed. We find that it can jointly optimize the two objectives and return designs that satisfy engineering design requirements (compliance and density) while utilizing features specified by text.
Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
Authors: Ali Taghibakhshi; Sharath Turuvekere Sreenivas; Saurav Muralidharan; Marcin Chochowski; Yashaswi Karnati; Raviraj Bhuminand Joshi; Ameya Sunil Mahabaleshwarkar; ZIJIA CHEN; Yoshi Suhara; Oluwatobi Olabiyi; Daniel Korzekwa; Mostofa Patwary; Mohammad Shoeybi; Jan Kautz; Bryan Catanzaro; Ashwath Aithal; Nima Tajbakhsh; Pavlo Molchanov
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Hybrid language models that combine Attention and State Space Models (SSMs) have been shown to achieve state-of-the-art accuracy and runtime performance. Recent work has also demonstrated that applying pruning and distillation to Attention-only models yields smaller, more accurate models at a fraction of the training cost. In this work, we explore the effectiveness of compressing Hybrid architectures. To this end, we introduce a novel group-aware pruning method for Mamba layers that preserves the structural integrity of SSM blocks and their sequence modeling capabilities. We combine this method with FFN, embedding dimension, and layer pruning, along with knowledge distillation-based retraining to obtain a unified compression recipe for hybrid models. Using this recipe, we compress the Nemotron-H 8B Hybrid model down to 4B parameters with up to $40\times$ fewer training tokens compared to similarly-sized models. The resulting model surpasses the accuracy of similarly-sized models while achieving $\sim2\times$ faster inference throughput, significantly advancing the Pareto frontier.
Differentiable extensions with rounding guarantees for combinatorial optimization over permutations
Authors: Robert R Nerem; Zhishang Luo; Akbar Rafiey; Yusu Wang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Continuously extending combinatorial optimization objectives is a powerful technique commonly applied to the optimization of set functions. However, few such methods exist for extending functions on permutations, despite the fact that many combinatorial optimization problems, such as the quadratic assignment problem (QAP) and the traveling salesperson problem (TSP), are inherently optimization over permutations. We present Birkhoff Extension (BE), an almost-everywhere-differentiable continuous polytime-computable extension of any real-valued function on permutations to doubly stochastic matrices. Key to this construction is our introduction of a continuous variant of the well-known Birkhoff decomposition. 
Our extension has several nice properties making it appealing for optimization problems. 
First, BE provides a rounding guarantee, namely any solution to the extension can be efficiently rounded to a permutation without increasing the function value. Furthermore, an approximate solution in the relaxed case will give rise to an approximate solution in the space of permutations. 
Second, using BE, any real-valued optimization objective on permutations can be extended to an almost-everywhere-differentiable objective function over the space of doubly stochastic matrices. 
This makes our BE amenable to not only gradient-descent based optimization, but also unsupervised neural combinatorial optimization where training often requires a differentiable loss. 
Third, based on the above properties, we present a simple optimization procedure which can be readily combined with existing optimization approaches to offer local improvements (i.e., the quality of the final solution is no worse than the initial solution). 
Finally, we also adapt our extension to optimization problems over a class of trees, such as Steiner tree and optimization-based hierarchical clustering.
We present experimental results to verify our theoretical results on several combinatorial optimization problems related to permutations.
Rescaled Influence Functions: Accurate Data Attribution in High Dimension
Authors: Ittai Rubinstein; Samuel B. Hopkins
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: How does the training data affect a model's behavior?
This is the question we seek to answer with *data attribution*.
The leading practical approaches to data attribution are based on *influence functions* (IF).
IFs utilize a first-order Taylor approximation to efficiently predict the effect of removing a set of samples from the training set without retraining the model, and are used in a wide variety of machine learning applications.
However, especially in the high-dimensional regime (# params $\geq \Omega($# samples$)$), they are often imprecise and tend to underestimate the effect of sample removals, even for simple models such as logistic regression.
We present *rescaled influence functions* (RIF) -- a tool for data attribution which can be used as a drop-in replacement for influence functions, with little computational overhead but significant improvement in accuracy.
We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice, and present a theoretical analysis explaining this improvement.
Finally, we present a simple class of data poisoning attacks that would fool IF-based detections but would be detected by RIF.
Estimating Hitting Times Locally at Scale
Authors: Themistoklis Haris; Fabian Christian Spaeh; Spyros Dragazis; Charalampos Tsourakakis
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Hitting times provide a fundamental measure of distance in random processes, quantifying the expected number of steps for a random walk starting at node $u$ to reach node $v$. They have broad applications across domains such as network centrality analysis, ranking and recommendation systems, and epidemiology.  

In this work, we develop local algorithms for estimating hitting times between a pair of vertices $u,v$ without accessing the full graph, overcoming scalability issues of prior global methods. Our first algorithm uses the key insight  that hitting time computations can be truncated at the meeting time of two independent random walks from $u$ and $v$. This leads to an efficient estimator analyzed via the Kronecker product graph and Markov Chain Chernoff bounds. We also present an algorithm extending the work of Peng et al. [2021] that introduces a novel adaptation of the spectral cutoff technique to account for the asymmetry of hitting times. This adaptation captures the directionality of the underlying random walk and requires non-trivial modifications to ensure accuracy and efficiency. In addition to the algorithmic upper bounds, we also provide tight asymptotic lower bounds. 

Finally, we reveal a connection between hitting time estimation and distribution testing, and we validate our algorithms using experiments on both real and synthetic data.
Activated LoRA: Fine-tuned LLMs for Intrinsics
Authors: Kristjan Greenewald; Luis A. Lastras; Thomas Parnell; Vraj Shah; Lucian Popa; Giulio Zizzo; Chulaka Gunasekara; Ambrish Rawat; David Daniel Cox
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Low-Rank Adaptation (LoRA) has emerged as a highly efficient framework for finetuning the weights of large foundation models, and has become the go-to method for data-driven customization of LLMs. Despite the promise of highly customized behaviors and capabilities, switching between relevant LoRAs in a multiturn setting is inefficient, as the key-value (KV) cache of the entire turn history must be recomputed with the LoRA weights before generation can begin. To address this problem, we propose Activated LoRA (aLoRA), an adapter architecture which modifies the LoRA framework to only adapt weights for the tokens in the sequence after the aLoRA is invoked. This change crucially allows aLoRA to accept the base model's KV cache of the input string, meaning that aLoRA can be instantly activated whenever needed in a chain without recomputing the prior keys and values. This enables building what we call intrinsics, i.e. specialized models invoked to perform well-defined operations on portions of an input chain or conversation that otherwise uses the base model by default. We train a set of aLoRA-based intrinsics models, demonstrating competitive accuracy with standard LoRA while significantly improving inference efficiency. We contributed our Activated LoRA implementation to the Huggingface PEFT library.
Constrained Posterior Sampling: Time Series Generation with Hard Constraints
Authors: Sai Shankar Narasimhan; Shubhankar Agarwal; Litu Rout; Sanjay Shakkottai; Sandeep P. Chinchali
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Generating realistic time series samples is crucial for stress-testing models and protecting user privacy by using synthetic data. In engineering and safety-critical applications, these samples must meet certain hard constraints that are domain-specific or naturally imposed by physics or nature. Consider, for example, generating electricity demand patterns with constraints on peak demand times. This can be used to stress-test the functioning of power grids during adverse weather conditions. Existing approaches for generating constrained time series are either not scalable or degrade sample quality. To address these challenges, we introduce Constrained Posterior Sampling (CPS), a diffusion-based sampling algorithm that aims to project the posterior mean estimate into the constraint set after each denoising update. Notably, CPS scales to a large number of constraints ($\sim100$) without requiring additional training. We provide theoretical justifications highlighting the impact of our projection step on sampling. Empirically, CPS outperforms state-of-the-art methods in sample quality and similarity to real time series by around 70\% and 22\%, respectively, on real-world stocks, traffic, and air quality datasets.
What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models
Authors: Keyon Vafa; Sarah Bentley; Jon Kleinberg; Sendhil Mullainathan
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: How should we evaluate the quality of generative models? Many existing metrics focus on a model's producibility, i.e. the quality and breadth of outputs it can generate. However, the actual value from using a generative model stems not just from what it can produce but whether a user with a specific goal can produce an output that satisfies that goal. We refer to this property as steerability. In this paper, we first introduce a mathematical decomposition for quantifying steerability independently from producibility. Steerability is more challenging to evaluate than producibility because it requires knowing a user's goals. We address this issue by creating a benchmark task that relies on one key idea: sample an output from a generative model and ask users to reproduce it. We implement this benchmark in user studies of text-to-image and large language models. Despite the ability of these models to produce high-quality outputs, they all perform poorly on steerability. These results suggest that we need to focus on improving the steerability of generative models.  We show such improvements are indeed possible: simple image-based steering mechanisms achieve more than 2x improvement on this benchmark.
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Authors: Yonggan Fu; Xin Dong; Shizhe Diao; Matthijs Van keirsbilck; Hanrong Ye; Wonmin Byeon; Yashaswi Karnati; Lucas Liebenwein; Maksim Khadkevich; Alexander Keller; Jan Kautz; Yingyan Celine Lin; Pavlo Molchanov
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints.While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. 
This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. 
Specifically, we identify two central architectural factors: depth–width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth–width ratios, with the key finding that although deep–thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy–latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy–latency frontier.
In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. This technique can serve as a generalizable component for future SLMs.
Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy–efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5\% average accuracy, 1.3$\times$/1.9$\times$ lower latency, and 18.7$\times$/45.6$\times$ higher throughput compared to Qwen3-1.7B/0.6B, respectively.
Depth-Bounds for Neural Networks via the Braid Arrangement
Authors: Moritz Leo Grillo; Christoph Hertrich; Georg Loho
Venue: NeurIPS 2025 (NeurIPS 2025 oral)

Abstract: We contribute towards resolving the open question of how many hidden layers are required in ReLU networks for exactly representing all continuous and piecewise linear functions on $\mathbb{R}^d$. 
While the question has been resolved in special cases, the best known lower bound in general is still 2. 
We focus on neural networks that are compatible with certain polyhedral complexes, more precisely with the braid fan.  
For such neural networks, we prove a non-constant lower bound of $\Omega(\log\log d)$ hidden layers required to exactly represent the maximum of $d$ numbers. Additionally, we provide a combinatorial proof that neural networks satisfying this assumption require three hidden layers to compute the maximum of 5 numbers; this had only been verified with an excessive computation so far.
Finally, we show that a natural generalization of the best known upper bound to maxout networks is not tight, by demonstrating that a rank-3 maxout layer followed by a rank-2 maxout layer is sufficient to represent the maximum of 7 numbers.
Private Training Large-scale Models with Efficient DP-SGD
Authors: Liangyu Wang; Junxiao Wang; Jie Ren; Zihang Xiang; David E. Keyes; Di Wang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: As large language models (LLMs) increasingly underpin technological advancements, the privacy of their training data emerges as a critical concern. Differential Privacy (DP) serves as a rigorous mechanism to protect this data, yet its integration via Differentially Private Stochastic Gradient Descent (DP-SGD) introduces substantial challenges, primarily due to the complexities of per-sample gradient clipping. Current explicit methods, such as Opacus, necessitate extensive storage for per-sample gradients, significantly inflating memory requirements. Conversely, implicit methods like GhostClip reduce storage needs by recalculating gradients multiple times, which leads to inefficiencies due to redundant computations. This paper introduces FlashDP, an innovative cache-friendly per-layer DP-SGD that consolidates necessary operations into a single task, calculating gradients only once in a fused manner. This approach not only diminishes memory movement by up to 50\% but also cuts down redundant computations by 20\%, compared to previous methods. Consequently, FlashDP does not increase memory demands and achieves a 90\% throughput compared to the Non-DP method on a four-A100 system during the pre-training of the Llama-13B model, while maintaining parity with standard per-layer clipped DP-SGD in terms of accuracy. These advancements establish FlashDP as a pivotal development for efficient and privacy-preserving training of LLMs. FlashDP's code has been open-sourced in https://github.com/kaustpradalab/flashdp.