GenAI4ELab/papercli-papers huggingface dataset:
How Text Quality Interventions Reshape Neural Scaling Laws for LLMs: Empirical Study
Authors: Newsha Ardalani; Feiyang Kang; Michael Kuchnik; Mostafa Elhoushi; Shubhabrata Sengupta; Shang-Wen Li; Carole-Jean Wu
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Neural scaling laws are widely used for performance projection and resource planning, yet their sensitivity to data quality interventions remains poorly understood. We present the first large-scale empirical study of how interventions—deduplication, heuristic filtering, and LLM-guided rewriting—reshape scaling behavior in large language model training. Using QualityPajama, a suite of 23 systematically curated datasets, we train over 2,000 models (100M–8B parameters, 100M–200B tokens) to measure how text quality interventions affects
scaling-law parameters and compute-optimal design decisions. While prior studies have shown that model architecture primarily shifts coefficients, we demonstrate that data interventions shift both coefficients and exponents, fundamentally changing the fitted scaling laws in ways not anticipated by existing theory. We show that data quality ranking is scale and resource-dependent. Compute-optimal
token–to-parameter ratios vary by orders of magnitude across interventions, revealing a fundamental data quality–quantity trade-off in scaling. These findings pave the way for deeper theoretical understanding of scaling laws, establish scaling-law analysis as a principled framework for data strategy evaluation and ranking, and motivate a data-quality–aware approach to scaling next-generation LLMs.
MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use
Authors: Zijian Wu; Xiangyan Liu; xinyuan zhang; Lingjun Chen; Fanqing Meng; Lingxiao Du; Yiran Zhao; Fanshi Zhang; Yaoqi Ye; Jiawei Wang; Zirui Wang; Jinjie Ni; Yufan Yang; Arvin Xu; Michael Qizhe Shieh
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: The MCP standardizes how LLMs interact with external systems, forming the foundation for general agents.
However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows.
To address this gap, we propose \texttt{MCPMark}, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents, each with a curated initial state and programmatic verification script. These tasks demand diverse CRUD operations and richer environmental interactions.
We evaluate cutting-edge LLMs using a minimal agent framework. The best-performing model, \texttt{gpt-5-medium}, reaches only $52.56$\% pass@1 and $33.86$\% pass\textasciicircum{}4, while other strong models including \texttt{claude-sonnet-4} and \texttt{o3} fall below $30$\% pass@1 and $15$\% pass\textasciicircum{}4. On average, LLMs require $16.2$ turns and $17.4$ tool calls per task, highlighting the stress-testing nature of \texttt{MCPMark}.
Membership Inference Attacks Against Fine-tuned Diffusion Language Models
Authors: Yuetian Chen; Kaiyuan Zhang; Yuntao Du; Edoardo Stoppa; Charles Fleming; Ashish Kundu; Bruno Ribeiro; Ninghui Li
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Diffusion Language Models (DLMs) represent a promising alternative to autoregressive language models, using bidirectional masked token prediction. Yet their susceptibility to privacy leakage via Membership Inference Attacks (MIA) remains critically underexplored. This paper presents the first systematic investigation of MIA vulnerabilities in DLMs. Unlike the autoregressive models' single fixed prediction pattern, DLMs' multiple maskable configurations exponentially increase attack opportunities. This ability to probe many independent masks dramatically improves detection chances. To exploit this, we introduce SAMA (Subset-Aggregated Membership Attack), which addresses the sparse signal challenge through robust aggregation. SAMA samples masked subsets across progressive densities and applies sign-based statistics that remain effective despite heavy-tailed noise. Through inverse-weighted aggregation prioritizing sparse masks' cleaner signals, SAMA transforms sparse memorization detection into a robust voting mechanism. Experiments on nine datasets show SAMA achieves 30\% relative AUC improvement over the best baseline, with up to 8$\times$ improvement at low false positive rates. These findings reveal significant, previously unknown vulnerabilities in DLMs, necessitating the development of tailored privacy defenses.
DiffBED: Scaling Bayesian Experimental Design to High-Dimensions
Authors: Adhi Saravanan; Rik Knowles; Gavin Kerrigan; Tom Rainforth
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Bayesian experimental design (BED) is a principled framework for intelligent data acquisition. However, current approaches do not scale to problems with high–dimensional designs, impeding its uptake. We show that this limitation arises predominantly from the difficulty in specifying a likelihood model that remains accurate throughout the design space, and that without this, standard design optimisation procedures lead to a reward-hacking-like behaviour that exploits deficiencies in the likelihood, producing implausible or unrealistic designs.
To overcome this, we introduce DiffBED, an approach based on a novel BED objective that explicitly rewards realistic designs. Realism is captured by a diffusion model, which we guide using information-theoretic experimental design criteria to generate highly informative yet realistic designs. This enables BED at an unprecedented scale: while existing applications of BED have been restricted to design spaces with a handful of dimensions, we show that DiffBED can successfully scale to designing high–resolution images.
SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables
Authors: Sungho Park; Jueun Kim; Wook-Shin Han
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Real-world Table–Text question answering (QA) tasks require models that can reason across long text and source tables, traversing multiple hops and executing complex operations such as aggregation. Yet existing benchmarks are small, manually curated—and therefore error-prone—and contain shallow questions that seldom demand more than two hops or invoke aggregations, grouping, or other advanced analytical operations expressible in natural-language queries. We present SPARTA, an end-to-end construction framework that automatically generates large-scale Table–Text QA benchmarks with lightweight human validation, requiring only one quarter of the annotation time of HybridQA. The framework first constructs a reference fact database by enriching each source table with grounding tables whose tuples are atomic facts automatically extracted from the accompanying unstructured passages, then synthesizes nested queries whose number of nested predicates matches the desired hop count. To ensure that every SQL statement is executable and that its verbalization yields a fluent, human-sounding question, we propose two novel techniques: provenance-based refinement, which rewrites any syntactically valid query that returns a non-empty result, and realistic-structure enforcement, which confines generation to post-order traversals of the query graph. The resulting pipeline produces thousands of high-fidelity question–answer pairs covering aggregations, grouping, and deep multi-hop reasoning across text and tables. On SPARTA, state-of-the-art models that reach over 70 F1 on HybridQA or over 50 F1 on OTT-QA drop by more than 30 F1 points, exposing fundamental weaknesses in current cross-modal reasoning. We will release the benchmark, construction code, and baseline results to spur progress toward robust, realistic Table–Text QA models.
Learning to Reason over Continuous Tokens with Reinforcement Learning
Authors: Yiran Zhao; Yuhui Xu; Doyen Sahoo; Caiming Xiong; Junnan Li
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Large Language Models (LLMs) have shown strong performance in complex reasoning tasks, especially when guided by Chain-of-Thought (CoT) prompting. However, conventional CoT reasoning in the discrete token space suffers from high computational and memory costs due to verbose intermediate steps. Recent work has explored latent reasoning in the embedding space to improve efficiency, but often at the cost of clarity and performance. In this work, we propose $\underline{Hy}$brid $\underline{Rea}$soning ($\texttt{HyRea}$), a unified framework that enables LLMs to dynamically switch between explicit (token-based) and latent (embedding-based) reasoning during inference. To train the model to make these decisions effectively, we introduce a two-stage training pipeline: (1) a supervised cold-start phase that introduces latent reasoning by replacing low-entropy CoT steps with embeddings, and (2) a reinforcement learning phase using Group Relative Policy Optimization (GRPO) to fine-tune the model’s reasoning strategy based on task-specific rewards. 
Experiments on mathematical reasoning benchmarks show that \texttt{HyRea} achieves significant reductions in token usage while maintaining or improving accuracy, offering an effective and scalable solution for efficient multi-step reasoning in LLMs.
Group Representational Position Encoding
Authors: Yifan Zhang; Zixiang Chen; Yifeng Liu; Zhen Qin; Huizhuo Yuan; Kangping Xu; Yang Yuan; Quanquan Gu; Andrew C Yao
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: We present GRAPE (Group RepresentAtional Position Encoding), a unified framework for positional encoding based on group actions. GRAPE brings together two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $\operatorname{SO}(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $\mathrm{GL}$.
In Multiplicative GRAPE, a position $n\in\mathbb{Z}$ (or $t\in\mathbb{R}$) acts as $\mathbf{G}(n)=\exp(n\,\omega\,\mathbf{L})$ with a rank‑2 skew generator $\mathbf{L} \in \mathbb{R}^{d \times d}$, yielding a relative, compositional, norm‑preserving map with a closed‑form matrix exponential. RoPE is recovered exactly when the $d/2$ planes are the canonical coordinate pairs with log‑uniform spectrum. Learned commuting subspaces and compact non‑commuting mixtures strictly extend this geometry to capture cross-subspace feature coupling at $O(d)$ and $O(rd)$ cost per head, respectively.
In Additive GRAPE, additive logits arise as rank‑1 (or low‑rank) unipotent actions, recovering ALiBi and the Forgetting Transformer (FoX) as exact special cases while preserving an exact relative law and streaming cacheability. Altogether, GRAPE supplies a principled design space for positional geometry in long‑context models, subsuming RoPE and ALiBi as special cases.
Contrastive Predictive Coding Done Right for Mutual Information Estimation
Authors: Jongha Jon Ryu; Pavan Yeddanapudi; Xiangxiang Xu; Gregory W. Wornell
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce a simple modification, which we refer to as *InfoNCE-anchor*, for accurate MI estimation. 
Our modification introduces an auxiliary \emph{anchor} class, enabling consistent density ratio estimation and yielding a plug-in MI estimator with significantly reduced bias.
Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCE-anchor as a special case when the log score is employed. 
This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and $f$-divergence variants, under a single principled framework. 
Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in self-supervised representation learning experiments, we find that the anchor does not improve the downstream task performance.
These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios.
Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence
Authors: Alexander Semenenko; Ivan Butakov; Ivan Oseledets; Alexey Frolov
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence (even under linear transformations designed to enhance the extraction of information), prioritizes redundancy over informative content, and in some cases, performs worse than simpler dependence measures like the correlation coefficient.
Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability
Authors: Taylor Sorensen; Benjamin Newman; Jared Moore; Chan Young Park; Jillian Fisher; Niloofar Mireshghallah; Liwei Jiang; Yejin Choi
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Language model post-training has enhanced instruction-following and performance on many downstream tasks, but also comes with an often-overlooked cost on tasks with many possible valid answers. On many tasks such as creative writing, synthetic data generation, or steering to diverse preferences, models must cover an entire distribution of outputs, rather than a single correct answer. We characterize three desiderata for conditional distributional modeling: in-context steerability, valid output space coverage, and distributional alignment, and document across three model families how current post-training can reduce these properties. In particular, we disambiguate between two kinds of in-context learning: ICL for eliciting existing underlying knowledge or capabilities, and in-context steerability, where a model must use in-context information to override its priors and steer to a novel data generating distribution. To better evaluate and improve these desiderata, we introduce Spectrum Suite, a large-scale resource compiled from $>40$ data sources and spanning $>90$ tasks requiring models to steer to and match diverse distributions ranging from varied human preferences to numerical distributions and more. We find that while current post-training techniques elicit underlying capabilities and knowledge, they hurt models' ability to flexibly steer in-context. To mitigate these issues, we propose Spectrum Tuning, a post-training method using Spectrum Suite to improve steerability and distributional coverage. We find that Spectrum Tuning often improves over pretrained and typical instruction-tuned models, enhancing steerability, spanning more of the output space, and improving distributional alignment on held-out datasets.
Decoupled Q-Chunking
Authors: Qiyang Li; Seohong Park; Sergey Levine
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Bootstrapping bias problem is a long-standing challenge in temporal-difference (TD) methods in off-policy reinforcement learning (RL). Multi-step return backups can alleviate this issue but require delicate importance sampling to correct their off-policy bias. Recent work has proposed to use chunked critics, which estimate the value of short action sequences ("chunks") rather than individual actions, enabling unbiased multi-step backup. However, extracting policies from chunked critics is challenging: policies must output the entire action chunk open-loop, which can be sub-optimal in environments that require policy reactivity and also challenging to model especially when the chunk length grows. Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks. We propose a novel algorithm that achieves this by optimizing the policy against a distilled critic for partial action chunks, constructed by optimistically backing up from the original chunked critic to approximate the maximum value achievable when a partial action chunk is extended to a complete one. This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning policies over long action chunks. We evaluate our method on challenging, long-horizon offline goal-conditioned benchmarks and show that it reliably outperforms prior methods.
Kaleidoscope: In-language Exams for Massively  Multilingual Vision Evaluation
Authors: Israfel Salazar; Manuel Fernández Burda; Shayekh Bin Islam; Arshia Soltani Moakhar; Shivalika Singh; Fabian Farestam; Angelika Romanou; Danylo Boiko; Dipika Khullar; Mike Zhang; Dominik Krzemiński; Jekaterina Novikova; Luísa Shimabucoro; Joseph Marvin Imperial; Rishabh Maheshwary; Sharad Duwal; Alfonso Amayuelas; Swati Rajwal; Jebish Purbey; Ahmed Ruby; Nicholas Popovič; Marek Suppa; Azmine Toushik Wasi; Ram Mohan Rao Kadiyala; Olga Tsymboi; Maksim Kostritsya; Bardia soltani moakhar; Gabriel da Costa Merlin; Otávio Ferracioli Coletti; Maral Jabbarishiviari; MOHAMMADAMIN FARAHANIFARD; Silvia Andrea Fernandez; María Grandury; Dmitry Abulkhanov; Drishti Sharma; Andre Guarnier De Mitri; Leticia Bossatto Marchezi; Setayesh Heydari; Johan Obando-Ceron; Nazar Kohut; Beyza Ermis; Desmond Elliott; Enzo Ferrante; Sara Hooker; Marzieh Fadaee
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.
Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining
Authors: Anirudh Subramanyam; Yuxin Chen; Robert L. Grossman
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Scaling laws for language model training traditionally characterize how performance scales with model size and dataset volume. Prior work has explored architecture variants and data treatments such as dataset filtering and noise injection in language model pretraining; however, these studies have not formalized data quality within a principled scaling law. We introduce a dimensionless data-quality parameter Q, and propose a quality-aware scaling law extending the Chinchilla framework to predict loss as a joint function of model size, data volume, and data quality. The law is motivated by an effective-sample-size and information-theoretic view of noisy or redundant corpora, and it admits two practical estimators for Q: (i) a corruption rate proxy and (ii) a deficiency measure. Through synthetic experiments in neural machine translation and autoregressive modeling--where we systematically control data quality via multiple levels of noise injection variation--we show that loss scales predictably with data quality and that higher-quality data can substantially reduce model size and hence compute requirements. Our results demonstrate a sublinear decay of effective data with quality and robustness to moderate data corruption; out-of-sample evaluations further validate the predictive form of the law. Unlike prior empirical analyses, our work establishes an explicit, generalizable law for data quality, offering concrete guidance for balancing data curation effort and model scale in large-scale pretraining.
Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring.
Authors: Natalia Martinez; Fearghal O'Donncha; Wesley M. Gifford; Nianjun Zhou; Dhaval C Patel; Roman Vaculin
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: We propose a post-hoc adaptive conformal anomaly detection method for monitoring time series that leverages predictions from pre-trained foundation models without requiring additional fine-tuning. Our method yields an interpretable anomaly score directly interpretable as a false alarm rate (p-value), facilitating transparent and actionable decision-making. It employs weighted quantile conformal prediction bounds and adaptively learns optimal weighting parameters from past predictions, enabling calibration under distribution shifts and stable false alarm control, while preserving out-of-sample guarantees. As a model-agnostic solution, it integrates seamlessly with foundation models and supports rapid deployment in resource-constrained environments. This approach addresses key industrial challenges such as limited data availability, lack of training expertise, and the need for immediate inference, while taking advantage of the growing accessibility of time series foundation models. Experiments on both synthetic and real-world datasets show that the proposed approach delivers strong performance, combining simplicity, interpretability, robustness, and adaptivity.
HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series
Authors: Simon A. Lee; Cyrus Tanade; Hao Zhou; Juhyeon Lee; Megha Thukral; Md Sazzad Hissain Khan; Keum San Chun; Baiying Lu; Migyeong Gwak; Mehrab Bin Morshed; Viswam Nathan; Md Mahbubur Rahman; Li Zhu; Subramaniam Venkatraman; Sharanya Arcot Desai
Venue: ICLR 2026 (ICLR 2026 Poster)

Abstract: Wearable sensors provide abundant physiological time series observations, yet the resolution at which we should extract features for downstream tasks remain unclear. We hypothesize that temporal resolution is a fundamental axis of representation learning, with different clinical and behavioral outcomes relying on features at distinct scales. To test this resolution hypothesis, we introduce HiMAE (Hierarchical Masked Autoencoder), a self-supervised framework that combines masked autoencoding with a hierarchical convolutional encoder–decoder. HiMAE produces multi-resolution embeddings across its intermediate layers that enable systematic evaluation of which temporal scales carry predictive signal, transforming resolution from a hyperparameter into a probe for interpretability. Across classification and generative benchmarks, HiMAE consistently outperforms state-of-the-art foundation models that collapse scale, while being orders of magnitude smaller. Due to the convolution based design choices behind HiMAE, the model is also compact enough to run entirely on-device, achieving sub-millisecond inference on smartwatch-class CPUs for true edge inference. Together, these contributions position HiMAE as both an efficient self supervised learning method and a discovery tool for understanding how time resolution contributes to downstream task alignment.