GenAI4ELab/papercli-papers huggingface dataset:
LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation
Authors: Hao Huang; Jingtao Ding; Mengqi Liao; Xin Wang; Jinyang Ban; Yuan Yuan; Huaiyu Wan; Yong Li
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: The widespread adoption of Low-Rank Adaptation (LoRA) for efficient fine-tuning of large language models has created demand for scalable parameter generation methods that can synthesize adaptation weights directly from task descriptions, avoiding costly task-specific training. We present LoRAGen, a structure-aware method for generating LoRA parameters from natural language descriptions. Through empirical analysis of LoRA libraries, we identify two key structural properties of LoRA parameter spaces: non-uniqueness of low-rank decomposition and heterogeneous weight distributions across network modules. These properties necessitate specialized parameter generation methods rather than general weight space learning approaches. LoRAGen employs a latent diffusion model with two innovations: weight-space supervision on full adaptation matrices to handle decomposition non-uniqueness, and a module-aware Mix-of-Experts decoder that adapts to module-specific weight distributions. Experiments show LoRAGen achieves 96.0\% performance relative to task-specific LoRAs on FLAN-T5-large and 72.7\% on Gemma-2-2B-Instruct for in-distribution tasks, while obtaining 40.2\% on zero-shot generation across unseen tasks—surpassing baselines by nearly 5\%. Our work establishes the first structure-aware approach to LoRA generation with insights into adaptation weight space geometry.
WaterDrum: Watermark-based Data-centric Unlearning Metric
Authors: Xinyang Lu; Xinyuan Niu; Gregory Kang Ruey Lau; Nhung Bui; Rachael Hwee Ling Sim; John Russell Himawan; Fanyu Wen; Chuan-Sheng Foo; See-Kiong Ng; Bryan Kian Hsiang Low
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Large language model (LLM) unlearning is critical in real-world applications where it is necessary to efficiently remove the influence of private, copyrighted, or harmful data from some users. Existing utility-centric unlearning metrics (based on model utility) may fail to accurately evaluate the extent of unlearning in realistic settings such as when the forget and retain sets have semantically similar content and/or retraining the model from scratch on the retain set is impractical. This paper presents the first data-centric unlearning metric for LLMs called WaterDrum that exploits robust text watermarking to overcome these limitations. We introduce new benchmark datasets (with different levels of data similarity) for LLM unlearning that can be used to rigorously evaluate unlearning algorithms via WaterDrum. Our code is available on [Github](https://github.com/lululu008/WaterDrum) and our new benchmark datasets are released on [HuggingFace](https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax).
DeNOTS: Stable Deep Neural ODEs for Time Series
Authors: Ilya Kuleshov; Evgenia Romanenkova; Vladislav Andreevich Zhuzhel; Galina Boeva; Evgeni Vorsin; Alexey Zaytsev
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Neural Controlled Differential Equations (Neural CDEs) provide a principled framework for modelling irregular time series in continuous time. 
Their number of function evaluations (NFEs) acts as a natural analogue of depth in discrete neural networks and is typically controlled indirectly via solver tolerances. 
However, tightening tolerances increases numerical precision without necessarily improving expressiveness.
We propose a simple alternative: scaling the integration time horizon to increase NFEs and thereby "deepen" the model. 
Since enlarging the interval can cause uncontrolled growth in standard vector fields, we introduce a Negative Feedback (NF) mechanism that ensures provable stability without limiting flexibility. 
We further establish general risk bounds for Neural CDEs and quantify discretization error using Gaussian process theory, improving robustness to integration and interpolation error.
On four public benchmarks, our method, **DeNOTS**, outperforms existing approaches—including Neural RDEs and state space models—by up to $20$%. 
DeNOTS combines expressiveness, stability, and robustness for reliable continuous-time modelling.
Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction
Authors: Tianyi Alex Qiu; Micah Carroll; Cameron Allen
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: The evaluation and post-training of large language models (LLMs) rely on supervision, but strong supervision for difficult tasks is often unavailable, especially when evaluating strong models. In such cases, models have been demonstrated to exploit evaluation schemes built on such imperfect supervision, leading to deceptive results. 

However, underutilized in LLM research, a wealth of mechanism design research focuses on game-theoretic *incentive compatibility* - eliciting honest and informative answers with weak supervision. 
Drawing from this literature, we introduce the peer prediction method for model evaluation and post-training. It rewards honest and informative answers over deceptive and uninformative ones, using a metric based on mutual predictability and without requiring ground truth labels. 

We demonstrate the method's effectiveness and resistance to deception, with both theoretical guarantees and empirical validation on models with up to 405B parameters. We show that training an 8B model with peer prediction-based reward recovers most of the drop in truthfulness due to prior malicious finetuning, even when the reward is produced by a 0.135B language model with no finetuning.

On the evaluation front, in contrast to LLM-as-a-Judge which requires strong and trusted judges, we discover an inverse scaling property in peer prediction, where, surprisingly, resistance to deception is *strengthened* as the capability gap between the experts and participants *widens*, enabling reliable evaluation of strong models with weak supervision. In particular, LLM-as-a-Judge become worse than random guess when facing deceptive models 5-20$\times$ the judge's size, while peer prediction thrives when such gaps are large, including in cases with over 100$\times$ size difference.
Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
Authors: Yujin Jo; Taesup Kim
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range of real-world tasks without additional task-specific training.
However, in real deployment scenarios with evolving environments or emerging classes, these models inevitably face distributional shifts and novel tasks.
In such contexts, static zero-shot capabilities are insufficient, and there is a growing need for continual learning methods that allow models to adapt over time while avoiding catastrophic forgetting.
We introduce NuSA-CL (Null Space Adaptation for Continual Learning), a lightweight memory-free continual learning framework designed to address this challenge.
NuSA-CL employs low-rank adaptation and constrains task-specific weight updates to lie within an approximate null space of the model's current parameters.
This strategy minimizes interference with previously acquired knowledge, effectively preserving the zero-shot capabilities of the original model.
Unlike methods relying on replay buffers or costly distillation, NuSA-CL imposes minimal computational and memory overhead, making it practical for deployment in resource-constrained, real-world continual learning environments.
Experiments show that our framework not only effectively preserves zero-shot transfer capabilities but also achieves highly competitive performance on continual learning benchmarks. 
These results position NuSA-CL as a practical and scalable solution for continually evolving zero-shot VLMs in real-world applications.
Anchored Supervised Fine-Tuning
Authors: He Zhu; Junyou Su; Peng Lai; Ren Ma; Wenjia Zhang; Linyi Yang; Guanhua Chen
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Post-training of large language models involves a fundamental trade-off between
supervised fine-tuning (SFT), which efficiently mimics demonstrations but tends
to memorize, and reinforcement learning (RL), which achieves better generaliza-
tion at higher computational cost. Dynamic Fine-Tuning (DFT) recently emerged
as a promising middle ground, reweighting SFT objectives with token probabili-
ties and achieving improvements in certain reasoning domains, though it exhibits
instability in other tasks. We provide a analysis of DFT through the reward-
weighted regression (RWR) framework, revealing that it corresponds to a spe-
cific auxiliary distribution choice that yields provably tighter RL bounds than
standard SFT. However, our analysis also uncovers a critical limitation: this con-
struction lacks distributional anchoring, leading to progressive drift that under-
mines training stability. To address this, we propose Anchored Supervised Fine-
Tuning (ASFT), which augments DFT’s reweighting with lightweight KL regu-
larization to preserve tightness while ensuring stability. Empirically, ASFT con-
sistently outperforms both SFT and DFT across mathematical reasoning, medical
knowledge grounding, and code generation, achieving substantial improvements
with minimal computational overhead. Our RWR framework provides a system-
atic lens for understanding post-training methods and demonstrates that principled
theoretical analysis leads to both stronger guarantees and practical gains.
Barriers for Learning in an Evolving World:  Mathematical Understanding of Loss of Plasticity
Authors: Amir Joudaki; Giulia Lanzillotta; Mohammad Samragh; Seyed Iman Mirzadeh; Keivan Alizadeh; Thomas Hofmann; Mehrdad Farajtabar; Fartash Faghri
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Deep learning models excel in stationary settings but suffer from loss of plasticity (LoP) in non-stationary environments. While prior literature characterizes LoP through symptoms like rank collapse of representations, it often lacks a mechanistic explanation for why gradient descent fails to recover from these states. This work presents a first-principles investigation grounded in dynamical systems theory, formally defining LoP not merely as a statistical degradation, but as an entrapment of gradient dynamics within invariant sub-manifolds of the parameter space. We identify two primary mechanisms that create these traps: frozen units from activation saturation and cloned-unit manifolds from representational redundancy. Crucially, our framework uncovers a fundamental tension: the very mechanisms that promote generalization in static settings, such as low-rank compression, actively steer the network into these LoP manifolds. We validate our theoretical analysis with numerical simulations and demonstrate how architectural interventions can destabilize these manifolds to restore plasticity.
Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
Authors: Xuemiao Zhang; Can Ren; Chengying Tu; Rongxiang Weng; Shuo Wang; Hongfei Yan; Jingang Wang; Xunliang Cai
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Recent progress in large reasoning models for challenging mathematical reasoning has been driven by reinforcement learning (RL). 
Incorporating long chain-of-thought (CoT) data during mid-training has also been shown to substantially improve reasoning depth. 
However, current approaches often utilize CoT data indiscriminately, leaving open the critical question of which data types most effectively enhance model reasoning capabilities.
In this paper, we define the foundation model's reasoning potential for the first time as the inverse of the number of independent attempts required to correctly answer the question, which is strongly correlated with the final model performance.
We then propose utilizing diverse data enriched with high-value reasoning patterns to expand the reasoning potential. 
Specifically, we abstract atomic reasoning patterns from CoT sequences, characterized by commonality and inductive capabilities, and use them to construct a core reference set enriched with valuable reasoning patterns. 
Furthermore, we propose a dual-granularity algorithm involving chains of reasoning patterns and token entropy, efficiently selecting high-value CoT data (CoTP) from the data pool that aligns with the core set, thereby training models to master reasoning effectively. 
Only 10B-token CoTP data enables the 85A6B Mixture-of-Experts (MoE) model to improve by **9.58\%** on the challenging AIME 2024 and 2025, and to raise the upper bound of downstream RL performance by **7.81\%**.
Reconciling Visual Perception and Generation in Diffusion Models
Authors: Liulei Li; Yi Yang; Wenguan Wang
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: We present \textsc{GenRep}, a unified image understanding and synthesis model that jointly conducts  discriminative learning and generative modeling in one training session. By leveraging  Monte Carlo approximation, \textsc{GenRep} distills distributional knowledge embedded in diffusion models to guide the discriminative learning for visual perception tasks. Simultaneously, a semantic-driven image generation process is established, where  high-level semantics learned from perception tasks can be used to inform image synthesis, creating a positive feedback loop for mutual boosts. Moreover, to reconcile the learning process for both tasks, a gradient alignment strategy is proposed to symmetrically modify the optimization directions of perception and generation losses. These designs empower \textsc{GenRep} to be a versatile and powerful model that achieves top-leading performance on both image understanding and generation benchmarks.
Conformal Prediction with Corrupted Labels: Uncertain Imputation and Robust Re-weighting
Authors: Shai Feldman; Stephen Bates; Yaniv Romano
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: We introduce a framework for robust uncertainty quantification in situations where labeled training data are corrupted, through noisy or missing labels. We build on conformal prediction, a statistical tool for generating prediction sets that cover the test label with a pre-specified probability. The validity of conformal prediction, however, holds under the i.i.d assumption, which does not hold in our setting due to the corruptions in the data. To account for this distribution shift, the privileged conformal prediction (PCP) method proposed leveraging privileged information (PI)---additional features available only during training---to re-weight the data distribution, yielding valid prediction sets under the assumption that the weights are accurate. In this work, we analyze the robustness of PCP to inaccuracies in the weights. Our analysis indicates that PCP can still yield valid uncertainty estimates even when the weights are poorly estimated. Furthermore, we introduce uncertain imputation (UI), a new conformal method that does not rely on weight estimation. Instead, we impute corrupted labels in a way that preserves their uncertainty. Our approach is supported by theoretical guarantees and validated empirically on both synthetic and real benchmarks. Finally, we show that these techniques can be integrated into a triply robust framework, ensuring statistically valid predictions as long as at least one underlying method is valid.
Differential Fine-Tuning Large Language Models Towards Better Diverse Reasoning Abilities
Authors: Xiaosong Yuan; Chen Shen; Shaotian Yan; kaiyuan liu; Xiaofeng Zhang; Sinan Fan; Liang Xie; Wenxiao Wang; Renchu Guan; Ying Wang; Jieping Ye
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Reasoning abilities of large language models (LLMs) require explicit derivations compared to general question-answering, supervised fine-tuning (SFT) can empower multiple reasoning abilities in LLMs via learning from various datasets. However, neither training the datasets jointly (mix-up) nor continually can maintain the performance of single-dataset SFT, sometimes better while sometimes even worse, illustrating vanilla SFT can not only facilitate reasoning abilities but also introduce conflicts. In this paper, we propose a novel framework to mitigate the conflicts and preserve benefits among different reasoning tasks, and even surpass each task's single dataset SFT performance. We start by exploring the differences between reasoning fine-tuned and base LLMs by analyzing their parameter variations during model inference, and we discover that each reasoning capability has exclusive parameters that benefit it more evidently than others. In contrast, the overlapped parameters of tasks can bring benefits or conflicts. Inspired by the findings, we propose to update the exclusive and overlapped parameters according to specific reasoning task combinations differentially, thereby avoiding unnecessary conflicts while maintaining benefits. Consistent improvements in mix-up and continual SFT experiments demonstrate that the proposed SFT strategy can achieve better performance on various LLMs (Llama3-8B, Mistral-7B, and Qwen2.5-14B) and diverse reasoning tasks with fewer conflicts, showing the superiority and generality of our analysis findings and the proposed approach.
Short Window Attention Enables Long-Term Memorization
Authors: Loïc Cabannes; Maximilian Beck; Maria Lomeli; Gergely Szilvasy; Matthijs Douze; Jade Copet; Pierre-Emmanuel Mazare; Gabriel Synnaeve; Herve Jegou
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. However, the impact of the window length and the interplay between local layers and global layers remain under-studied. In this work, we first analyze the interaction between short and long term memory by considering SWAX: a hybrid architecture consisting of sliding-window attention and xLSTM linear RNN layers.

A counter-intuitive finding is that larger sliding windows hurts the long-context performance. In fact, short window attention encourages the model to better train the long-term memory of the xLSTM as it cannot rely on the local softmax attention mechanism for long context-retrieval. 
We also validate our findings on local-global architectures alternating short window and full attention layers: the short layers should be small in order not to hinder the usefulness of the long layers. 

However, employing too small sliding windows is detrimental even for short-context tasks, which could be solved with information from moderately larger sliding windows otherwise. Therefore, we train hybrid architectures by stochastically changing the sliding window size, forcing the model to leverage both the short term window and the long-term memory. Training with stochastic window sizes significantly outperforms regular window attention both on short and long-context problems.
NatADiff: Adversarial Boundary Guidance for Natural Adversarial Diffusion
Authors: Max Collins; Jordan Vice; Tim French; Ajmal Saeed Mian
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Adversarial samples exploit irregularities in the manifold "learned" by deep learning models to cause misclassifications. The study of these adversarial samples provides insight into the features a model uses to classify inputs, which can be leveraged to improve robustness against future attacks. However, much of the existing literature focuses on constrained adversarial samples, which do not accurately reflect test-time errors encountered in real-world settings. To address this, we propose `NatADiff', an adversarial sampling scheme that leverages denoising diffusion to generate natural adversarial samples. Our approach is based on the observation that natural adversarial samples frequently contain structural elements from the adversarial class. Deep learning models can exploit these structural elements to shortcut the classification process, rather than learning to genuinely distinguish between classes. To leverage this behavior, we guide the diffusion trajectory towards the intersection of the true and adversarial classes, combining time-travel sampling with augmented classifier guidance to enhance attack transferability while preserving image quality. Our method achieves comparable white-box attack success rates to current state-of-the-art techniques, while exhibiting significantly higher transferability across model architectures and improved alignment with natural test-time errors as measured by FID. These results demonstrate that NatADiff produces adversarial samples that not only transfer more effectively across models, but more faithfully resemble naturally occurring test-time errors when compared with other generative adversarial sampling schemes.
Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
Authors: Han Zhou; Xingchen Wan; Ruoxi Sun; Hamid Palangi; Shariq Iqbal; Ivan Vulić; Anna Korhonen; Sercan O Arik
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Large language models, employed as multiple agents that interact and collaborate with each other, have excelled at solving complex tasks. The agents are programmed with prompts that declare their functionality, along with the topologies that orchestrate interactions across agents. Designing prompts and topologies for multi-agent systems (MAS) is inherently complex. To automate the entire design process, we first conduct an in-depth analysis of the design space aiming to understand the factors behind building effective MAS. We reveal that prompts together with topologies play critical roles in enabling more effective MAS design. Based on the insights, we propose Multi-Agent System Search (MASS), a MAS optimization framework that efficiently exploits the complex MAS design space by interleaving its optimization stages, from local to global, from prompts to topologies, over three stages: 1) block-level (local) prompt optimization; 2) workflow topology optimization; 3) workflow-level (global) prompt optimization, where each stage is conditioned on the iteratively optimized prompts/topologies from former stages. We show that MASS-optimized multi-agent systems outperform a spectrum of existing alternatives by a substantial margin. Based on the MASS-found systems, we finally propose design principles behind building effective multi-agent systems.
Deep Global-sense Hard-negative Discriminative Generation Hashing for Cross-modal Retrieval
Authors: Kun Cheng; Qibing Qin; Wenfeng Zhang; Lei Huang; Jie Nie
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Hard negative generation (HNG) provides valuable signals for deep learning, but existing methods mostly rely on local correlations while neglecting the global geometry of the embedding space. This limitation often leads to weak discrimination, particularly in cross-modal hashing, which obtains compact binary codes.
We propose Deep Global-sense Hard-negative Discriminative Generation Hashing (DGHDGH), a framework that constructs a structured graph with dual-iterative message propagation to capture global correlations, and then performs difficulty-adaptive, channel-wise interpolation to synthesize semantically consistent hard negatives aligned with global Hamming geometry.
Our approach yields more informative negatives, sharpens semantic boundaries in the Hamming co-space, and substantially enhances cross-modal retrieval. Experiments on multiple benchmarks consistently demonstrate improvements in retrieval accuracy, verifying the discriminative advantages brought by global-sense HNG in cross-modal hashing.