GenAI4ELab/papercli-papers huggingface dataset:
ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Authors: Zedong Liu; Shenggan Cheng; Guangming Tan; Yang You; Dingwen Tao
Venue: NeurIPS 2025 (NeurIPS 2025 oral)

Abstract: Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. However, these additional components—combined with complex inference pipelines and heterogeneous workloads—introduce significant inference overhead. Therefore, efficiently serving MLLMs remains a major challenge. Current tightly coupled serving architectures struggle to distinguish between mixed request types or adapt parallelism strategies to different inference stages, leading to increased time-to-first-token (TTFT) and poor resource utilization. To address this, we introduce Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity across request types and inference stages. Building upon EMP, we develop ElasticMM, an MLLM serving system that (1) separates requests into independent modality groups with dynamic resource allocation via a modality-aware load balancer; (2) decouples inference stages and enables parallelism adjustment and adaptive scaling via elastic partition scheduling; and (3) improves inference efficiency through unified multimodal prefix caching and non-blocking encoding. Experiments on diverse real-world datasets show that ElasticMM outperforms state-of-the-art (SOTA) serving systems, reducing TTFT by up to 4.2$\times$ and achieving 3.2–4.5$\times$ higher throughput while meeting service-level objectives (SLOs).
Better Training Data Attribution via Better Inverse Hessian-Vector Products
Authors: Andrew Wang; Elisa Nguyen; Runshi Yang; Juhan Bae; Sheila A. McIlraith; Roger Baker Grosse
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.
Exploring the Noise Robustness of Online Conformal Prediction
Authors: HuaJun Xi; Kangdao Liu; Hao Zeng; Wenguang Sun; Hongxin Wei
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. 
Recent work develops online conformal prediction methods that adaptively construct prediction sets to accommodate distribution shifts. 
However, existing algorithms typically assume *perfect label accuracy* which rarely holds in practice. 
In this work, we investigate the robustness of online conformal prediction under uniform label noise with a known noise rate. 
We show that label noise causes a persistent gap between the actual mis-coverage rate and the desired rate $\alpha$, leading to either overestimated or underestimated coverage guarantees. 
To address this issue, we propose a novel loss function *robust pinball loss*, which provides an unbiased estimate of clean pinball loss without requiring ground-truth labels.
Theoretically, we demonstrate that robust pinball loss enables online conformal prediction to eliminate the coverage gap under uniform label noise, achieving a convergence rate of $\mathcal{O}(T^{-1/2})$ for both empirical and expected coverage errors (i.e., absolute deviation of the empirical and expected mis-coverage rate from the target level $\alpha$).
This loss offers a general solution to the uniform label noise, and is complementary to existing online conformal prediction methods.
Extensive experiments demonstrate that the proposed loss enhances the noise robustness of various online conformal prediction methods by achieving a precise coverage guarantee.
Automated Detection of Visual Attribute Reliance with a Self-Reflective Agent
Authors: Christy Li; Josep Lopez Camuñas; Jake Thomas Touchet; Jacob Andreas; Agata Lapedriza; Antonio Torralba; Tamar Rott Shaham
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: When a vision model performs image recognition, which visual attributes drive its predictions? Detecting unintended reliance on specific visual features is critical for ensuring model robustness, preventing overfitting, and avoiding spurious correlations. We introduce an automated framework for detecting such dependencies in trained vision models. At the core of our method is a self-reflective agent that systematically generates and tests hypotheses about visual attributes that a model may rely on. This process is iterative: the agent refines its hypotheses based on experimental outcomes and uses a self-evaluation protocol to assess whether its findings accurately explain model behavior. When inconsistencies arise, the agent self-reflects over its findings and triggers a new cycle of experimentation. We evaluate our approach on a novel benchmark of 130 models designed to exhibit diverse visual attribute dependencies across 18 categories. Our results show that the agent's performance consistently improves with self-reflection, with a significant performance increase over non-reflective baselines. We further demonstrate that the agent identifies real-world visual attribute dependencies in state-of-the-art models, including CLIP's vision encoder and the YOLOv8 object detector.
Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
Authors: Yi-Lun Wu; Bo-Kai Ruan; Chiang Tseng; Hong-Han Shuai
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Direct preference optimization (DPO) methods have shown strong potential in aligning text-to-image diffusion models with human preferences by training on paired comparisons. These methods improve training stability by avoiding the REINFORCE algorithm but still struggle with challenges such as accurately estimating image probabilities due to the non-linear nature of the sigmoid function and the limited diversity of offline datasets. In this paper, we introduce Diffusion Denoising Ranking Optimization (Diffusion-DRO), a new preference learning framework grounded in inverse reinforcement learning. Diffusion-DRO removes the dependency on a reward model by casting preference learning as a ranking problem, thereby simplifying the training objective into a denoising formulation and overcoming the non-linear estimation issues found in prior methods. Moreover, Diffusion-DRO uniquely integrates offline expert demonstrations with online policy-generated negative samples, enabling it to effectively capture human preferences while addressing the limitations of offline data. Comprehensive experiments show that Diffusion-DRO delivers improved generation quality across a range of challenging and unseen prompts, outperforming state-of-the-art baselines in both both quantitative metrics and user studies. Our source code and pre-trained models are available at https://github.com/basiclab/DiffusionDRO.
Flow Equivariant Recurrent Neural Networks
Authors: T. Anderson Keller
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.
AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking
Authors: Xiangqi Wang; Yue Huang; Yanbo Wang; Xiaonan Luo; Kehan Guo; Yujun Zhou; Xiangliang Zhang
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: LLMs often need effective configurations, like temperature and reasoning steps, to handle tasks requiring sophisticated reasoning and problem-solving, ranging from joke generation to mathematical reasoning. Existing prompting approaches usually adopt general-purpose, fixed configurations that work “well enough” across tasks but seldom achieve task-specific optimality.
To address this gap, we introduce AdaReasoner, an LLM-agnostic plugin designed for any LLM to automate adaptive reasoning configurations for tasks requiring different types of thinking. AdaReasoner is trained using a reinforcement learning (RL) framework, combining a factorized action space with a targeted exploration strategy, along with  a pretrained reward model to optimize the policy model for reasoning configurations with only a few-shot guide.
AdaReasoner is backed by theoretical guarantees and experiments of fast convergence and a sublinear policy gap. Across six different LLMs and a variety of reasoning tasks, it consistently outperforms standard baselines, preserves out-of-distribution robustness, 
and yield gains on knowledge-intensive tasks through tailored prompts.
Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
Authors: Vlad Sobal; Wancong Zhang; Kyunghyun Cho; Randall Balestriero; Tim G. J. Rudner; Yann LeCun
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: A long-standing goal in AI is to develop agents capable of solving diverse tasks across a range of environments, including those never seen during training. Two dominant paradigms address this challenge: (i) reinforcement learning (RL), which learns policies via trial and error, and (ii) optimal control, which plans actions using a known or learned dynamics model. However, their comparative strengths in the offline setting—where agents must learn from reward-free trajectories—remain underexplored. In this work, we systematically evaluate RL and control-based methods on a suite of navigation tasks, using offline datasets of varying quality. On the RL side, we consider goal-conditioned and zero-shot methods. On the control side, we train a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning. We investigate how factors such as data diversity, trajectory quality, and environment variability influence the performance of these approaches. Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts and is more data-efficient, while achieving trajectory stitching performance comparable to leading model-free methods. Notably, planning with a latent dynamics model proves to be a strong approach for handling suboptimal offline data and adapting to diverse environments.
LABridge: Text–Image Latent Alignment Framework via Mean-Conditioned OU Process
Authors: Huiyang Shao; Xin Xia; Yuxi Ren; XING WANG; Xuefeng Xiao
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Diffusion models have emerged as state‑of‑the‑art in image synthesis.However, it often suffer from semantic instability and slow iterative denoising. We introduce Latent Alignment Framework (LABridge), a novel Text–Image Latent Alignment Framework via an Ornstein–Uhlenbeck (OU) Process, which explicitly preserves and aligns textual and visual semantics in an aligned latent space. LABridge employs a Text-Image Alignment Encoder (TIAE) to encode text prompts into structured priors that are directly aligned with image latents. Instead of a homogeneous Gaussian, Mean-Conditioned OU process smoothly interpolates between these text‑conditioned priors and image latents, improving stability and reducing the number of denoising steps. Extensive experiments on standard text-to-image benchmarks show that LABridge achieves better text–image alignment metric and competitive FID scores compared to leading diffusion baselines. By unifying text and image representations through principled latent alignment, LABridge paves the way for more efficient, semantically consistent, and high‑fidelity text to image generation.
REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing
Authors: Weihan Xu; Yimeng Ma; Jingyue Huang; Yang Li; Wenye Ma; Taylor Berg-Kirkpatrick; Julian McAuley; Paul Pu Liang; Hao-Wen Dong
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Short videos are an effective tool for promoting contents and improving knowledge accessibility. While existing extractive video summarization methods struggle to produce a coherent narrative, existing abstractive methods cannot `quote' from the input videos, i.e., inserting short video clips in their outputs. In this work, we explore novel video editing models for generating shorts that feature a coherent narrative with embedded video insertions extracted from a long input video. We propose a novel retrieval-embedded generation framework that allows a large language model to quote multimodal resources while maintaining a coherent narrative. Our proposed REGen system first generates the output story script with quote placeholders using a finetuned large language model, and then uses a novel retrieval model to replace the quote placeholders by selecting a video clip that best supports the narrative from a pool of candidate quotable video clips. We examine the proposed method on the task of documentary teaser generation, where short interview insertions are commonly used to support the narrative of a documentary. Our objective evaluations show that the proposed method can effectively insert short video clips while maintaining a coherent narrative. In a subjective survey, we show that our proposed method outperforms existing abstractive and extractive approaches in terms of coherence, alignment, and realism in teaser generation.
Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity
Authors: Mouin Ben Ammar; David Brellmann; Arturo Mendoza; Antoine Manzanera; Gianni Franchi
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: **Out-of-distribution (OOD) detection** is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on **post-hoc OOD detection**, which enables identifying OOD samples without altering the model's training procedure or objective. Our primary goal is to investigate the relationship between **model capacity** and its OOD detection performance. Specifically, we aim to answer the following question:
*Does the Double Descent phenomenon manifest in post-hoc OOD detection?* This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection.
Despite the growing interest in these topics by the classic supervised machine learning community, this intersection remains unexplored for OOD detection.
We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, we provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and we propose a method to identify the optimal regime for OOD detection based on our observations.
Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions
Authors: Ofir Gaash; Kfir Yehuda Levy; Yair Carmon
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We study stochastic gradient descent (SGD) with gradient clipping on convex functions under a generalized smoothness assumption called $(L_0,L_1)$-smoothness. Using gradient clipping, we establish a high probability convergence rate that matches the SGD rate in the $L$ smooth case up to polylogarithmic factors and additive terms. We also propose a variation of adaptive SGD with gradient clipping, which achieves the same guarantee. We perform empirical experiments to examine our theory and algorithmic choices.
Smooth Regularization for Efficient Video Recognition
Authors: Gil Goldman; Raja Giryes; Mahadev Satyanarayanan
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We propose a smooth regularization technique that instills a strong temporal inductive 
bias in video recognition models, particularly benefiting lightweight architectures. 
Our method encourages smoothness in the intermediate-layer embeddings
of consecutive frames by modeling their changes as a Gaussian Random Walk
(GRW). This penalizes abrupt representational shifts, thereby promoting low-
acceleration solutions that better align with the natural temporal coherence inherent
in videos. By leveraging this enforced smoothness, lightweight models can more
effectively capture complex temporal dynamics. Applied to such models, our 
technique yields a 3.8%–6.4% accuracy improvement on Kinetics-600. Notably, the
MoViNets model family trained with our smooth regularization improves the 
current state-of-the-art by 3.8%–6.1% within their respective FLOP constraints, while
MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9%–6.4% over
prior state-of-the-art models with comparable memory footprints. Our code and
models are available at https://github.com/cmusatyalab/grw-smoothing.
LLM Safety Alignment is Divergence Estimation in Disguise
Authors: Rajdeep Haldar; Ziyi Wang; Guang Lin; Yue Xing; Qifan Song
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We present a theoretical framework showing that popular LLM alignment methods—including RLHF and its variants—can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less-preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance–refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.
Learning-Augmented Algorithms for $k$-median via Online Learning
Authors: Anish Hebbar; Rong Ge; Amit Kumar; Debmalya Panigrahi
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: The field of learning-augmented algorithms seeks to use ML techniques on past instances of a problem to inform an algorithm designed for a future instance. In this paper, we introduce a novel model for learning-augmented algorithms inspired by online learning. In this model, we are given a sequence of instances of a problem and the goal of the learning-augmented algorithm is to use prior instances to propose a solution to a future instance of the problem. The performance of the algorithm is measured by its average performance across all the instances, where the performance on a single instance is the ratio between the cost of the algorithm's solution and that of an optimal solution for that instance. We apply this framework to the classic $k$-median clustering problem, and give an efficient learning algorithm that can approximately match the average performance of the best fixed $k$-median solution in hindsight across all the instances. We also experimentally evaluate our algorithm and show that its empirical performance is close to optimal, and also that it automatically adapts the solution to a dynamically changing sequence.
Alias-Free ViT: Fractional Shift Invariance via Linear Attention
Authors: Hagay Michaeli; Daniel Soudry
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Transformers have emerged as a competitive alternative to convnets in vision tasks, yet they lack the architectural inductive bias of convnets, which may hinder their potential performance.  
Specifically, Vision Transformers (ViTs) are not translation‑invariant and are more sensitive to minor image translations than standard convnets.
Previous studies have shown, however, that convnets are also not perfectly shift‑invariant, due to aliasing in downsampling and nonlinear layers. 
Consequently, anti‑aliasing approaches have been proposed to certify convnets translation robustness. 
Building on this line of work, we propose an Alias‑Free ViT, which combines two main components. 
First, it uses alias-free downsampling and nonlinearities. 
Second, it uses linear cross‑covariance attention that is shift‑equivariant to both integer and fractional translations, enabling a shift-invariant global representation.
Our model maintains competitive performance in image classification and outperforms similar‑sized models in terms of robustness to adversarial translations.
TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
Authors: Andreas Auer; Patrick Podest; Daniel Klotz; Sebastian Böck; Günter Klambauer; Sepp Hochreiter
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: In-context learning, the ability of large language models to perform tasks using only examples provided in the prompt, has recently been adapted for time series forecasting. 
This paradigm enables zero-shot prediction, where past values serve as context for forecasting future values, 
making powerful forecasting tools accessible to non-experts and increasing the performance when training data are scarce.
Most existing zero-shot forecasting approaches rely on transformer architectures, which, despite their success in language, often fall short of expectations in time series forecasting, where recurrent models like LSTMs frequently have the edge. Conversely, while LSTMs are well-suited for time series modeling due to their state-tracking capabilities, they lack strong in-context learning abilities.
We introduce *TiRex* that closes this gap by leveraging xLSTM, an enhanced LSTM with competitive in-context learning skills. Unlike transformers, state-space models, or parallelizable RNNs such as RWKV, TiRex retains state tracking, a critical property for long-horizon forecasting. To further facilitate its state-tracking ability, we propose a training-time masking strategy called CPM.
TiRex sets a new state of the art in zero-shot time series forecasting on the Hugging Face benchmarks *GiftEval* and *Chronos-ZS*, outperforming significantly larger models including *TabPFN-TS* (Prior Labs), *Chronos Bolt* (Amazon), *TimesFM* (Google), and *Moirai* (Salesforce) across both short- and long-term forecasts.