GenAI4ELab/papercli-papers huggingface dataset:
Greed is Good: A Unifying Perspective on Guided Generation
Authors: Zander W. Blasingame; Chen Liu
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Training-free guided generation is a widely used and powerful technique that allows the end user to exert further control over the generative process of flow/diffusion models.
Generally speaking, two families of techniques have emerged for solving this problem for *gradient-based guidance*: namely, *posterior guidance* (*i.e.*, guidance via projecting the current sample to the target distribution via the target prediction model) and *end-to-end guidance* (*i.e.*, guidance by performing backpropagation throughout the entire ODE solve).
    In this work, we show that these two seemingly separate families can actually be *unified* by looking at posterior guidance as a *greedy strategy* of *end-to-end guidance*.
    We explore the theoretical connections between these two families and provide an in-depth theoretical of these two techniques relative to the *continuous ideal gradients*.
    Motivated by this analysis we then show a method for *interpolating* between these two families enabling a trade-off between compute and accuracy of the guidance gradients.
We then validate this work on several inverse image problems and property-guided molecular generation.
Transferring Linear Features Across Language Models With Model Stitching
Authors: Alan Chen; Jack Merullo; Alessandro Stolfo; Ellie Pavlick
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply this technique to transfer the \textit{weights} of Sparse Autoencoders (SAEs) between models of different sizes to compare their representations. We find that small and large models learn highly similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings. For example, using a small-to-large transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs on larger models. Next, we show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate a method for improving the training efficiency of SAEs.
TreeGen: A Bayesian Generative Model for Hierarchies
Authors: Marcel Kollovieh; Nils Fleischmann; Filippo Guerranti; Bertrand Charpentier; Stephan Günnemann
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: In this work, we introduce TreeGen, a novel generative framework modeling distributions over hierarchies. We extend Bayesian Flow Networks (BFNs) to enable transitions between probabilistic and discrete hierarchies parametrized via categorical distributions. Our proposed scheduler provides smooth and consistent entropy decay across varying numbers of categories. We empirically evaluate TreeGen on the jet-clustering task in high-energy physics, demonstrating that it consistently generates valid trees that adhere to physical constraints and closely align with ground-truth log-likelihoods. Finally, by comparing TreeGen’s samples to the exact posterior distribution and performing likelihood maximization via rejection sampling, we demonstrate that TreeGen outperforms various baselines.
Streaming Stochastic Submodular Maximization with On-Demand User Requests
Authors: Honglian Wang; Sijing Tu; Lutz Oettershagen; Aristides Gionis
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We explore a novel problem in streaming submodular maximization, 
inspired by the dynamics of news-recommendation platforms.
We consider a setting where users can visit a news web\-site at any time, and upon each visit, 
the web\-site must display up to $k$ news items. 
User interactions are inherently stochastic: each news item presented to the user 
is consumed with a certain acceptance probability by the user, 
and each news item covers certain topics. 
Our goal is to design a streaming algorithm that maximizes the expected total topic coverage.
To address this problem, we establish a connection to submodular maximization subject to a matroid constraint.
We show that we can effectively adapt previous methods to address our problem when the number of user visits is known in advance 
or linear-size memory in the stream length is available.
However, in more realistic scenarios where only an upper bound on the visits and sublinear memory is available, the algorithms fail to guarantee any bounded performance.
To overcome these limitations, we introduce a new online streaming algorithm that 
achieves a competitive ratio of $1/8\delta$,  where $\delta$ controls the approximation quality. Moreover, it requires only a single pass over the stream, and uses memory independent of the stream length. 
Empirically, our algorithms consistently outperform the baselines.
Causal Differentiating Concepts:  Interpreting LM Behavior via Causal Representation Learning
Authors: Navita Goyal; Hal Daumé III; Alexandre Drouin; Dhanya Sridhar
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Language model activations entangle concepts that mediate their behavior, making it difficult to interpret these factors, which has implications for generalizability and robustness. We introduce an approach for disentangling these concepts without supervision. Existing methods for concept discovery often rely on external labels, contrastive prompts, or known causal structures, which limits their scalability and biases them toward predefined, easily annotatable features. In contrast, we propose a new unsupervised algorithm that identifies causal differentiating concepts—interpretable latent directions in LM activations that must be changed to elicit a different model behavior. These concepts are discovered using a constrained contrastive learning objective, guided by the insight that eliciting a target behavior requires only sparse changes to the underlying concepts. We formalize this notion and show that, under a particular assumption about the sparsity of these causal differentiating concepts, our method learns disentangled representations that align with human-interpretable factors influencing LM decisions. We empirically show the ability of our method to recover ground-truth causal factors in synthetic and semi-synthetic settings. Additionally, we illustrate the utility of our method through a case study on refusal behavior in language models. Our approach offers a scalable and interpretable lens into the internal workings of LMs, providing a principled foundation for interpreting language model behavior.
Fourier Token Merging: Understanding and Capitalizing Frequency Domain for Efficient Image Generation
Authors: Jiesong Liu; Xipeng Shen
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Image generation requires intensive computations and faces challenges due to long latency.
    Exploiting redundancy in the input images and intermediate representations throughout the neural network pipeline is an effective way to accelerate image generation.
    Token merging (ToMe) exploits similarities among input tokens by clustering them and merges similar tokens into one, thus significantly reducing the number of tokens that are fed into the transformer block.
    This work introduces Fourier Token Merging, a new method for understanding and capitalizing frequency domain for efficient image generation.
    By introducing frequency token merging, we find that transforming the token into the frequency domain representation for clustering can better exert the ability of clustering based on the underlying redundancy after de-correlation.
    Through analytical and empirical studies, we demonstrate the benefits of using Fourier clustering over the original time domain clustering.
    We experimented fourier token merging on the stable diffusion model, and the results show up to 25\% reduction in latency without impairing image quality. 
The code is available at https://github.com/Fred1031/Fourier-Token-Merging.
Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows
Authors: Adriel Sosa Marco; John D. Kirwan; Alexia Toumpa; Simos Gerasimou
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Quantifying uncertainty in deep regression models is important both for understanding the confidence of the model and for safe decision-making in high-risk domains. Existing approaches that yield prediction intervals overlook distributional information, neglecting the effect of multimodal or asymmetric distributions on decision-making. Similarly, full or approximated Bayesian methods, while yielding the predictive posterior density, demand major modifications to the model architecture and retraining. We introduce MCNF, a novel post hoc uncertainty quantification method that produces both prediction intervals and the full conditioned predictive distribution. MCNF operates on top of the underlying trained predictive model; thus, no predictive model retraining is needed. We provide experimental evidence that the MCNF-based uncertainty estimate is well calibrated, is competitive with state-of-the-art uncertainty quantification methods, and provides richer information for downstream decision-making tasks
Ambient Proteins - Training Diffusion Models on Noisy Structures
Authors: Giannis Daras; Jeffrey Ouyang-Zhang; Krithika Ravishankar; Constantinos Costis Daskalakis; Adam Klivans; Daniel Jesus Diaz
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: We present Ambient Protein Diffusion, a framework for training protein diffusion models that generates structures with unprecedented diversity and quality. State-of-the-art generative models are trained on computationally derived structures from AlphaFold2 (AF), as experimentally determined structures are relatively scarce. The resulting models are therefore limited by the quality of synthetic datasets.  Since the accuracy of AF predictions degrades with increasing protein length and complexity, de novo generation of long, complex proteins remains challenging. Ambient Protein Diffusion overcomes this problem by treating low-confidence AF structures as corrupted data.  Rather than simply filtering out low-quality AF structures, our method adjusts the diffusion objective for each structure based on its corruption level, allowing the model to learn from both high and low quality structures. Empirically, ambient protein diffusion yields major improvements: on proteins with 700 residues, diversity increases from 45% to 85% from the previous state-of-the-art, and designability improves from 70% to 88%.
On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
Authors: Riccardo Alberghi; Elizaveta Demyanenko; Luca Biggio; Luca Saglietti
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Recent advances in natural language processing highlight two key factors for improving reasoning in large language models (LLMs): (i) allocating more test-time compute tends to help on harder problems but often introduces redundancy in the reasoning trace, and (ii) compute is most effective when reasoning is systematic and incremental, forming structured chains of thought (CoTs) akin to human problem-solving. To study these factors in isolation, we introduce a controlled setting based on shortest-path tasks in layered graphs. We train decoder-only transformers on question–trace–answer triples using a custom tokenizer, comparing models trained on optimal bottom-up dynamic programming traces with those trained on longer, valid traces involving backtracking. Surprisingly, under the same training-token budget, the latter models generalize better to unseen graphs. This benefit is not due to length alone—injecting arbitrary redundancy into reasoning traces fails to help and can even hurt performance. Instead, we find that generalization correlates with the model's confidence in next-token prediction, suggesting that long, coherent, and locally incremental traces make the training signal easier to optimize.
A Unified Framework for Provably Efficient Algorithms to Estimate Shapley Values
Authors: Tyler Chen; Akshay Seshadri; Mattia Jacopo Villani; Pradeep Niroula; Shouvanik Chakrabarti; Archan Ray; Pranav Deshpande; Romina Yalovetzky; Marco Pistoia; Niraj Kumar
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Shapley values have emerged as a critical tool for explaining which features impact the decisions made by machine learning models.
However, computing exact Shapley values is difficult, generally requiring an exponential (in the feature dimension) number of model evaluations. To address this, many model-agnostic randomized estimators have been developed, the most influential and widely used being the KernelSHAP method (Lundberg & Lee, 2017). While related estimators such as unbiased KernelSHAP (Covert & Lee, 2021) and LeverageSHAP (Musco & Witter, 2025) are known to satisfy theoretical guarantees, bounds for KernelSHAP have remained elusive. We describe a broad and unified framework that encompasses KernelSHAP and related estimators constructed using both with and without replacement sampling strategies. We then prove strong non-asymptotic theoretical guarantees that apply to all estimators from our framework. This provides, to the best of our knowledge, the first theoretical guarantees for KernelSHAP and sheds further light on tradeoffs between existing estimators. Through comprehensive benchmarking on small and medium dimensional datasets for Decision-Tree models, we validate our approach against exact Shapley values, consistently achieving low mean squared error with modest sample sizes. Furthermore, we make specific implementation improvements to enable scalability of our methods to high-dimensional datasets. 
Our methods, tested on datasets such MNIST and CIFAR10, provide consistently better results compared to the KernelSHAP library.
On Universality Classes of Equivariant Networks
Authors: Marco Pacini; Gabriele Santin; Bruno Lepri; Shubhendu Trivedi
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Equivariant neural networks provide a principled framework for incorporating symmetry into learning architectures and have been extensively analyzed through the lens of their *separation power*, that is, the ability to distinguish inputs modulo symmetry. This notion plays a central role in settings such as graph learning, where it is often formalized via the Weisfeilern&ndash;Leman hierarchy. In contrast, the *universality* of equivariant models&mdash;their capacity to approximate target functions&mdash;remains comparatively underexplored. 
In this work, we investigate the approximation power of equivariant neural networks beyond separation constraints.
We show that separation power does not fully capture expressivity: models with identical separation power may differ in their approximation ability.
To demonstrate this, we characterize the universality classes of shallow invariant networks, providing a general framework for understanding which functions these architectures can approximate. Since equivariant models reduce to invariant ones under projection, this analysis yields sufficient conditions under which shallow equivariant networks fail to be universal.
Conversely, we identify settings where shallow models do achieve separation-constrained universality.
These positive results, however, depend critically on structural properties of the symmetry group, such as the existence of adequate normal subgroups, which may not hold in important cases like permutation symmetry.
Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations
Authors: Yu Eric Qian; Wilson S. Geisler; Xue-Xin Wei
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Previous studies have compared neural activities in the visual cortex to representations in deep neural networks trained on image classification. Interestingly, while some suggest that their representations are highly similar, others argued the opposite. Here, we propose a new approach to characterize the similarity of the decision strategies of two observers (models or brains) using decision variable correlation (DVC). DVC quantifies the image-by-image correlation between the decoded decisions based on the internal neural representations in a classification task. Thus, it can capture task-relevant information rather than general representational alignment. We evaluate DVC using monkey V4/IT recordings and network models trained on image classification tasks.
We find that model–model similarity is comparable to monkey-monkey similarity, whereas model–monkey similarity is consistently lower. Strikingly, DVC decreases with increasing network performance on ImageNet-1k. Adversarial training does not improve model–monkey similarity in task-relevant dimensions assessed using DVC, although it markedly increases the model–model similarity.
Similarly, pre-training on larger datasets does not improve model–monkey similarity. These results suggest a divergence between the task-relevant representations in monkey V4/IT and those learned by models trained on image classification tasks.
AdvPrefix: An Objective for Nuanced LLM Jailbreaks
Authors: Sicheng Zhu; Brandon Amos; Yuandong Tian; Chuan Guo; Ivan Evtimov
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Many jailbreak attacks on large language models (LLMs) rely on a common objective: making the model respond with the prefix ``Sure, here is (harmful request)''. While straightforward, this objective has two limitations: limited control over model behaviors, yielding incomplete or unrealistic jailbroken responses, and a rigid format that hinders optimization. We introduce AdvPrefix, a plug-and-play prefix-forcing objective that selects one or more model-dependent prefixes by combining two criteria: high prefilling attack success rates and low negative log-likelihood. AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free. For example, replacing GCG's default prefixes on Llama-3 improves nuanced attack success rates from 14\% to 80\%, revealing that current safety alignment fails to generalize to new prefixes. Code and selected prefixes are released.
Guarantees for Alternating Least Squares in Overparameterized Tensor Decompositions
Authors: Dionysis Arvanitakis; Vaidehi Srinivas; Aravindan Vijayaraghavan
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Tensor decomposition is a canonical non-convex optimization problem that is computationally challenging, and yet important due to applications in factor analysis and parameter estimation of latent variable models. In practice, scalable iterative methods, particularly Alternating Least Squares (ALS), remain the workhorse for tensor decomposition despite the lack of global convergence guarantees. A popular approach to tackle challenging non-convex optimization problems is overparameterization--- on input an $n \times n \times n$ tensor of rank $r$, the algorithm can output a decomposition of potentially rank $k$ (potentially larger than $r$).  On the theoretical side, overparameterization for iterative methods is challenging to reason about and requires new techniques.  The work of Wang et al., (NeurIPS 2020) makes progress by showing that a variant of gradient descent globally converges when overparameterized to $k=O(r^{7.5} \log n)$.  Our main result shows that overparameterization provably enables global convergence of ALS: on input a third order $n \times n \times n$ tensor with a decomposition of rank $r \ll n$, ALS overparameterized with rank $k=O(r^2)$ achieves global convergence with high probability under random initialization. Moreover our analysis also gives guarantees for the more general low-rank approximation problem. The analysis introduces new techniques for understanding iterative methods in the overparameterized regime based on new matrix anticoncentration arguments.
KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products
Authors: Zixuan Xia; Aram Davtyan; Paolo Favaro
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We propose KOALA++, a scalable Kalman-based optimization algorithm that explicitly models structured gradient uncertainty in neural network training. Unlike second-order methods, which rely on expensive second order gradient calculation, our method directly estimates the parameter covariance matrix by recursively updating compact gradient covariance products. This design improves upon the original KOALA framework that assumed diagonal covariance by implicitly capturing richer uncertainty structure without storing the full covariance matrix and avoiding large matrix inversions. Across diverse tasks, including image classification and language modeling, KOALA++ achieves accuracy on par or better than state-of-the-art second-order optimizers while maintaining the efficiency of first-order methods.
Preserving Task-Relevant Information Under Linear Concept Removal
Authors: Floris Holstege; Shauli Ravfogel; Bram Wouters
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Modern neural networks often encode unwanted concepts alongside task-relevant information, leading to fairness and interpretability concerns. Existing post-hoc approaches can remove undesired concepts but often degrade useful signals. We introduce SPLINCE—Simultaneous Projection for LINear concept removal and Covariance prEservation—which eliminates sensitive concepts from representations while exactly preserving their covariance with a target label. SPLINCE achieves this via an oblique projection that ``splices out'' the unwanted direction yet protects important label correlations. Theoretically, it is the unique solution that removes linear concept predictability and maintains target covariance with minimal embedding distortion. Empirically, SPLINCE outperforms baselines on benchmarks such as Bias in Bios and Winobias, removing protected attributes while minimally damaging main-task information.
Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful
Authors: Martin Marek; Sanae Lotfi; Aditya Somasundaram; Andrew Gordon Wilson; Micah Goldblum
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Conventional wisdom dictates that small batch sizes make language model pretraining and fine-tuning unstable, motivating gradient accumulation, which trades off the number of optimizer steps for a proportional increase in batch size. While it is common to decrease the learning rate for smaller batch sizes, other hyperparameters are often held fixed. In this work, we revisit small batch sizes all the way down to batch size one, and we propose a rule for scaling Adam hyperparameters to small batch sizes.  In particular, rather than holding the decay rate of the second moment fixed across batch sizes, we propose to hold its half-life fixed in terms of tokens. We find that small batch sizes (1) train stably, (2) are consistently more robust to hyperparameter choices, (3) achieve equal or better per-FLOP performance than larger batch sizes, and (4) notably enable stable language model training with vanilla SGD, even without momentum, despite storing no optimizer state. Building on these results, we provide practical recommendations for selecting a batch size and setting optimizer hyperparameters. We further recommend against gradient accumulation unless training on multiple devices with multiple model replicas. Finally, we show that a small batch size combined with an optimizer with a small state size can provide the performance benefits of full fine-tuning while maintaining a similar memory footprint to LoRA.