GenAI4ELab/papercli-papers huggingface dataset:
Elastic ViTs from Pretrained Models without Retraining
Authors: Walter Simoncini; Michael Dorkenwald; Tijmen Blankevoort; Cees G. M. Snoek; Yuki M Asano
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/
S-Crescendo: A Nested Transformer Weaving Framework for Scalable Nonlinear System in S-Domain Representation
Authors: Junlang Huang; Chen Hao; Li Luo; Yong Cai; Lexin Zhang; Tianhao Ma; Yitian Zhang; Zhong Guan
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Simulation of high-order nonlinear system requires extensive computational resources, especially in modern VLSI backend design where bifurcation-induced instability and chaos-like transient behaviors pose challenges. We present S-Crescendo - a nested transformer weaving framework that synergizes S-domain with neural operators for scalable time-domain prediction in high-order nonlinear networks, alleviating the computational bottlenecks of conventional solvers via Newton-Raphson method. By leveraging the partial-fraction decomposition of an n-th order transfer function into first-order modal terms with repeated poles and residues, our method bypasses the conventional Jacobian matrix-based iterations and efficiently reduces computational complexity from cubic $O(n^3)$ to linear $O(n)$.The proposed architecture seamlessly integrates an S-domain encoder with an attention-based correction operator to simultaneously isolate dominant response and adaptively capture higher-order non-linearities. Validated on order-1 to order-10 networks, our method achieves up to 0.99 test-set \(R^2\) accuracy against HSPICE golden waveforms and accelerates simulation by up to 18\(\times\), providing a scalable, physics-aware framework for high-dimensional nonlinear modeling.
Fixed-Point RNNs: Interpolating from Diagonal to Dense
Authors: Sajad Movahedi; Felix Sarnthein; Nicola Muca Cirone; Antonio Orvieto
Venue: NeurIPS 2025 (NeurIPS 2025 spotlight)

Abstract: Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer architectures. Current models, however, do not exhibit the full state-tracking expressivity of RNNs because they rely on channel-wise (i.e. diagonal) sequence mixing. 
In this paper, we investigate parameterizations of a large class of dense linear RNNs as fixed-points of parallelizable diagonal linear RNNs.
The resulting models can naturally trade expressivity for efficiency at a fixed number of parameters
and achieve state-of-the-art results on the state-tracking benchmarks $A_5$ and $S_5$, while matching performance on copying and other tasks.
Adaptive Time Encoding for Irregular Multivariate Time-Series Classification
Authors: Sangho Lee; Kyeongseo Min; Youngdoo Son; Hyungrok Do
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Time series are often irregularly sampled with uneven time intervals. In multivariate cases, such irregularities may lead to misaligned observations across variables and varying observation counts, making it difficult to extract intrinsic patterns and degrading the classification performance of deep learning models. In this study, we propose an adaptive time encoding approach to address the challenge of irregular sampling in multivariate time-series classification. Our approach generates latent representations at learnable reference points that capture missingness patterns in irregular sequences, enhancing classification performance. We also introduce consistency regularization techniques to incorporate intricate temporal and intervariable information into the learned representations. Extensive experiments demonstrate that our method achieves state-of-the-art performance with high computational efficiency in irregular multivariate time-series classification tasks.
Prior Forgetting and In-Context Overfitting
Authors: Sungyoon Lee
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: In-context learning (ICL) is one of the key capabilities contributing to the great success of LLMs. At test time, ICL is known to operate in the two modes: task recognition and task learning. In this paper, we investigate the emergence and dynamics of the two modes of ICL during pretraining. To provide an analytical understanding of the learning dynamics of the ICL abilities, we investigate the in-context random linear regression problem with a simple linear-attention-based transformer, and define and disentangle the strengths of the task recognition and task learning abilities stored in the transformer model’s parameters. We show that, during the pretraining phase, the model first learns the task learning and the task recognition abilities together in the beginning, but it (a) gradually forgets the task recognition ability to recall the priorly learned tasks and (b) relies more on the given context in the later phase, which we call (a) \textit{prior forgetting} and (b) \textit{in-context overfitting}, respectively.
Boosting Adversarial Transferability with Spatial Adversarial Alignment
Authors: Zhaoyu Chen; HaiJing Guo; Kaixun Jiang; Jiyuan Fu; Xinyu Zhou; Dingkang Yang; Hao Tang; Bo Li; Wenqiang Zhang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Deep neural networks are vulnerable to adversarial examples that exhibit transferability across various models. Numerous approaches are proposed to enhance the transferability of adversarial examples, including advanced optimization, data augmentation, and model modifications. However, these methods still show limited transferability, partiovovocularly in cross-architecture scenarios, such as from CNN to ViT. To achieve high transferability, we propose a technique termed Spatial Adversarial Alignment (SAA), which employs an alignment loss and leverages a witness model to fine-tune the surrogate model. Specifically, SAA consists of two key parts: spatial-aware alignment and adversarial-aware alignment. First, we minimize the divergences of features between the two models in both global and local regions, facilitating spatial alignment. Second, we introduce a self-adversarial strategy that leverages adversarial examples to impose further constraints, aligning features from an adversarial perspective. Through this alignment, the surrogate model is trained to concentrate on the common features extracted by the witness model. This facilitates adversarial attacks on these shared features, thereby yielding perturbations that exhibit enhanced transferability. Extensive experiments on various architectures on ImageNet show that aligned surrogate models based on SAA can provide higher transferable adversarial examples, especially in cross-architecture attacks.
Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Authors: Shuang Wu; Youtian Lin; Feihu Zhang; Yifei Zeng; Yikang Yang; yajie bao; Jiachen Qian; Siyu Zhu; Xun Cao; Philip Torr; Yao Yao
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Generating high-resolution 3D shapes using volumetric representations such as Signed Distance Functions (SDFs) presents substantial computational and memory challenges. We introduce Direct3D-S2, a scalable 3D generation framework based on sparse volumes that achieves superior output quality with dramatically reduced training costs.
Our key innovation is the Spatial Sparse Attention (SSA) mechanism, which greatly enhances the efficiency of Diffusion Transformer (DiT) computations on sparse volumetric data. SSA allows the model to effectively process large token sets within sparse volumes, significantly reducing computational overhead and achieving a 3.9$\times$ speedup in the forward pass and a 9.6$\times$ speedup in the backward pass.
Our framework also includes a variational autoencoder (VAE) that maintains a consistent sparse volumetric format across input, latent, and output stages. Compared to previous methods with heterogeneous representations in 3D VAE, this unified design significantly improves training efficiency and stability.
Our model is trained on public datasets, and experiments demonstrate that Direct3D-S2 not only surpasses state-of-the-art methods in generation quality and efficiency, but also enables training at 1024³ resolution using only 8 GPUs—a task typically requiring at least 32 GPUs for volumetric representations at $256^3$ resolution, thus making gigascale 3D generation both practical and accessible. Project page: https://www.neural4d.com/research-page/direct3d-s2.
Addressing Mark Imbalance in Integration-free Marked Temporal Point Processes
Authors: Sishun Liu; KE DENG; Yongli Ren; Yan Wang; Xiuzhen Zhang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Marked Temporal Point Process (MTPP) has been well studied to model the event distribution in marked event streams, which can be used to predict the mark and arrival time of the next event. However, existing studies overlook that the distribution of event marks is highly imbalanced in many real-world applications, with some marks being frequent but others rare. The imbalance poses a significant challenge to the performance of the next event prediction, especially for events of rare marks. To address this issue, we propose a thresholding method, which learns thresholds to tune the mark probability normalized by the mark's prior probability to optimize mark prediction, rather than predicting the mark directly based on the mark probability as in existing studies. In conjunction with this method, we predict the mark first and then the time.  In particular, we develop a novel neural Marked Temporal Point Process (MTPP) model to support effective time sampling and estimation of mark probability without computationally expensive numerical improper integration. Extensive experiments on real-world datasets demonstrate the superior performance of our solution against various baselines for the next event mark and time prediction. The code is available at https://github.com/undes1red/IFNMTPP.
More Than Just Functional: LLM-as-a-Critique for Efficient Code Generation
Authors: Derui Zhu; Dingfan Chen; jinfu chen; Jens Grossklags; Walter Pretschner; Weiyi Shang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Large language models (LLMs) have demonstrated remarkable progress in generating functional code, leading to numerous AI-based coding program tools. However, their reliance on the perplexity objective during both training and inference primarily emphasizes functionality, often at the expense of efficiency—an essential consideration for real-world coding tasks. Perhaps interestingly, we observed that well-trained LLMs inherently possess knowledge about code efficiency, but this potential remains underutilized with standard decoding approaches. To address this, we design strategic prompts to activate the model’s embedded efficiency understanding, effectively using LLMs as \textit{efficiency critiques} to guide code generation toward higher efficiency without sacrificing—and sometimes even improving—functionality, all without the need for costly real code execution. Extensive experiments on benchmark datasets (EffiBench, HumanEval+) across multiple representative code models demonstrate up to a 70.6\% reduction in average execution time and a 13.6\% decrease in maximum memory usage, highlighting the computational efficiency and practicality of our approach compared to existing alternatives.
A Circular Argument: Does RoPE need to be Equivariant for Vision?
Authors: Chase van de Geijn; Timo Lüddecke; Polina Turishcheva; Alexander S. Ecker
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards generalizing RoPE to higher-dimensional data such as images and videos. The success of RoPE has been thought to be due to its positional equivariance, i.e. its status as a \textit{relative} positional encoding. In this paper, we mathematically show RoPE to be one of the most general solutions for equivariant positional embedding in one-dimensional data. Moreover, we show Mixed RoPE to be the analogously general solution for $M$-dimensional data, if we require commutative generators -- a property necessary for RoPE's equivariance. However, we question the necessity of equivariance. We propose Spherical RoPE, a method analogous to Mixed RoPE, but with the assumption of anti-commutative generators -- relaxing the equivariant condition. Empirically, we find Spherical RoPE to have the equivalent learning behavior as its equivariant analogues. This strongly suggests that relative positional embeddings are not as important as is commonly believed. We expect this discovery to facilitate future work in positional encodings for vision that are faster and generalize better by removing the preconception that they must be relative.
UnCLe: Towards Scalable Dynamic Causal Discovery in Non-linear Temporal Systems
Authors: Tingzhu Bi; Yicheng Pan; Xinrui Jiang; Huize Sun; Meng Ma; Ping Wang
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Uncovering cause-effect relationships from observational time series is fundamental to understanding complex systems. While many methods infer static causal graphs, real-world systems often exhibit *dynamic causality*—where relationships evolve over time. Accurately capturing these temporal dynamics requires time-resolved causal graphs. We propose UnCLe, a novel deep learning method for scalable dynamic causal discovery. UnCLe employs a pair of Uncoupler and Recoupler networks to disentangle input time series into semantic representations and learns inter-variable dependencies via auto-regressive Dependency Matrices. It estimates dynamic causal influences by analyzing datapoint-wise prediction errors induced by temporal perturbations. Extensive experiments demonstrate that UnCLe not only outperforms state-of-the-art baselines on static causal discovery benchmarks but, more importantly, exhibits a unique capability to accurately capture and represent evolving temporal causality in both synthetic and real-world dynamic systems (e.g., human motion). UnCLe offers a promising approach for revealing the underlying, time-varying mechanisms of complex phenomena.
Beyond Least Squares: Uniform Approximation and the Hidden Cost of Misspecification
Authors: Davide Maran; Csaba Szepesvari
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We study the problem of controlling worst-case errors in misspecified linear regression under the random design setting, where the regression function is estimated via (penalized) least-squares. This setting arises naturally in value function approximation for bandit algorithms and reinforcement learning (RL).
Our first main contribution is the observation that the amplification of the misspecification error when using least-squares is governed by the \emph{Lebesgue constant}, a classical quantity from approximation theory that depends on the choice of the feature subspace and the covariate distribution.
We also show that this dependence on the misspecification error is tight for least-squares regression: in general, no method minimizing the empirical squared loss, including regularized least-squares, can improve it substantially.
We argue  this explains the empirical observation that some feature-maps (e.g., those derived from the Fourier bases) ``work better in RL'' than others (e.g., polynomials): 
given some covariate distribution, the Lebesgue constant is known to be highly sensitive to choice of the feature-map.
As a second contribution, we propose a method that augments the original feature set with auxiliary features designed to reduce the error amplification. 
We then prove that the method successfully competes with an "oracle'' that knows the best way of using
the auxiliary features to reduce this amplification.
For example, when the domain is a real interval and the features are monomials, 
our method reduces the amplification factor to $O(1)$ as $d\to\infty$, while 
without our method, least-squares with the monomials (and in fact polynomials) will suffer a worst-case error amplification of order $\Omega(d)$. It follows that there are functions and feature maps for which our method is consistent, while least-squares is inconsistent.
Preference-driven Knowledge Distillation for Few-shot Node Classification
Authors: Xing Wei; Chunchun Chen; Rui Fan; Xiaofeng Cao; Sourav Medya; Wei Ye
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Graph neural networks (GNNs) can efficiently process text-attributed graphs (TAGs) due to their message-passing mechanisms, but their training heavily relies on the human-annotated labels. Moreover, the complex and diverse local topologies of nodes of real-world TAGs make it challenging for a single mechanism to handle. Large language models (LLMs) perform well in zero-/few-shot learning on TAGs but suffer from a scalability challenge. Therefore, we propose a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of LLMs and various GNNs for few-shot node classification. Specifically, we develop a GNN-preference-driven node selector that effectively promotes prediction distillation from LLMs to teacher GNNs. To further tackle nodes' intricate local topologies, we develop a node-preference-driven GNN selector that identifies the most suitable teacher GNN for each node, thereby facilitating tailored knowledge distillation from teacher GNNs to the student GNN. Extensive experiments validate the efficacy of our proposed framework in few-shot node classification on real-world TAGs.
Our code can be available at <https://github.com/GEEX-Weixing/PKD>.
AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound
Authors: Gijs Wijngaard; Elia Formisano; Michele Esposito; Michel Dumontier
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AudSemThinker, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AudSem, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AudSem addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AudSemThinker outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AudSemThinker and the AudSem dataset are released publicly.
On Evaluating Policies for Robust POMDPs
Authors: Merlijn Krale; Eline M. Bovy; Maris F. L. Galesloot; Thiago D. Simão; Nils Jansen
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Robust partially observable Markov decision processes (RPOMDPs) model sequential decision-making problems under partial observability, where an agent must be robust against a range of dynamics. RPOMDPs can be viewed as a two-player game between an agent, who selects actions, and nature, who adversarially selects the dynamics. Evaluating an agent policy requires finding an adversarial nature
policy, which is computationally challenging. In this paper, we advance the evaluation of agent policies for RPOMDPs in three ways. First, we discuss suitable benchmarks. We observe that for some RPOMDPs, an optimal agent policy can be found by considering only subsets of nature policies, making them easier to solve. We formalize this concept of solvability and construct three benchmarks that are only solvable for expressive sets of nature policies. Second, we describe a new method to evaluate agent policies for RPOMDPs by solving an equivalent MDP. Third, we lift two well-known upper bounds from POMDPs to RPOMDPs, which can be used to efficiently approximate the optimality gap of a policy and serve as baselines. Our experimental evaluation shows that (1) our proposed benchmarks cannot be solved by assuming naive nature policies, (2) our method of evaluating policies is accurate, and (3) the upper bounds provide solid baselines for evaluation.
Neural Rule Lists: Learning Discretizations, Rules, and Order in One Go
Authors: Sascha Xu; Nils Philipp Walter; Jilles Vreeken
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: Interpretable machine learning is essential in high-stakes domains like healthcare. Rule lists are a
popular choice due to their transparency and accuracy, but learning them effectively remains a challenge.
Existing methods require feature pre-discretization, constrain rule complexity or ordering, or struggle
to scale. We present NeuRules, a novel end-to-end framework that overcomes these limitations. At its
core, NeuRules transforms the inherently combinatorial task of rule list learning into a differentiable
optimization problem, enabling gradient-based learning. It simultaneously discovers feature conditions,
assembles them into conjunctive rules, and determines their order—without pre-processing or manual
constraints. A key contribution here is a gradient shaping technique that steers learning toward sparse
rules with strong predictive performance.  To produce ordered lists, we introduce a differentiable relaxation
that, through simulated annealing, converges to a strict rule list. Extensive experiments show
that NeuRules consistently outperforms combinatorial and neural baselines on binary as well as
multi-class classification tasks across a wide range of datasets.
Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design
Authors: Andreas Schlaginhaufen; Reda Ouhamma; Maryam Kamgarpour
Venue: NeurIPS 2025 (NeurIPS 2025 poster)

Abstract: We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.