GenAI4ELab/papercli-papers huggingface dataset:
OUNLP at TSAR 2025 Shared Task Multi-Round Text Simplifier via Code Generation
Authors: Cuong Huynh; Jie Cao
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: This paper describes the system submission of our team OUNLP to the TSAR-2025 shared task on readability-controlled text simplification. Based on the analysis of prompt-based text simplification methods, we discovered that simplification performance is highly related to the gap between the source CEFR level and the target CEFR level. Inspired by this finding, we propose two multi-round simplification methods generated via GPT-4o rule-based simplification (MRS-Rule) and jointly rule-based LLM simplification (MRS-Joint). Our submitted systems ranked 7th out of 20 teams. Later improvements with MRS-Joint show that taking the LLM simplified candidates as the starting point could further boost multi-round simplification performance.
HIT-YOU at TSAR 2025 Shared Task Leveraging Similarity-Based Few-Shot Prompting, Round-Trip Translation, and Self-Refinement for Readability-Controlled Text Simplification
Authors: Mao Shimada; Kexin Bian; Zhidong Ling; Mamoru Komachi
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: We describe our submission to the TSAR 2025 shared task on readability-controlled text simplification, which evaluates systems on their ability to adjust linguistic complexity to specified CEFR levels while preserving meaning and coherence. We explored two complementary frameworks leveraging the shared task CEFR classifier as feedback. The first is an ensemble approach generating diverse candidates using multiple LLMs under zero-shot prompting with level-specific instructions and vocabulary lists, one-shot prompting, and round-trip translation. Candidates were filtered by predicted CEFR level before an LLM judge selected the final output. The second framework is a self-refinement loop, where a single candidate is iteratively revised with classifier feedback until matching the target level or reaching a maximum number of iterations. This study is among the first to apply round-trip translation and iterative self-refinement to controlled simplification, broadening the toolkit for adapting linguistic complexity.
SQUREL at TSAR 2025 Shared Task CEFR-Controlled Text Simplification with Prompting and Reinforcement Fine-Tuning
Authors: Daria Sokova; Anastasiia Bezobrazova; Constantin Orasan
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: This paper summarises the submissions of our team to the TSAR 2025 Shared Task on Readability-Controlled Text Simplification, which aims to create text simplifications balancing reduced linguistic complexity, meaning preservation, and fluency while meeting predefined target readability levels. We tested two different methods for CEFR-controlled simplification a conservative lexical pipeline relying on prompting LLMs to simplify sentences, and a setup employing reinforcement fine-tuning.
Archaeology at TSAR 2025 Shared Task Teaching Small Models to do CEFR Simplifications
Authors: Rareş-Alexandru Roşcan; Sergiu Nisioi
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: Large language models (LLMs) have demonstrated strong performance in text simplification tasks, but their high computational cost and proprietary nature often limit practical use, especially in education. We explore open-source LLMs for CEFR-level text simplification. By reducing model size and computational requirements, our approach enables greater accessibility and deployment in educational environments. Our results show some of the lowest error rates in producing CEFR-compliant texts at TSAR 2025, using models with 8 billion and 1 billion parameters. Such approaches have the potential to democratize NLP technologies for real-world applications.
HOPE at TSAR 2025 Shared Task Balancing Control and Complexity in Readability-Controlled Text Simplification
Authors: Sujal Maharjan; Astha Shrestha
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: This paper describes our submissions to the TSAR 2025 Shared Task on Readability-Controlled Text Simplification. We present a comparative study of three architectures a rule-based baseline, a heuristic-driven expert system, and a zero-shot generative T5 pipeline with a semantic guardrail. Our analysis shows a trade-off between the controllability of rule-based systems and the fluency of generative models. In this zero-shot setting, simpler, confined systems achieved superior meaning preservation scores compared to the more powerful but less predictable generative model. We present a diagnostic failure analysis on system outputs, illustrating how different architectures result in distinct error patterns such as under-simplification, information loss via heuristics, and semantic drift.
Know-AI at TSAR 2025 Shared Task Difficulty-aware Text Simplification System
Authors: Yiheng Wu; Anisia Katinskaia; Jue Hou; Roman Yangarber
Venue: EMNLP 2025 (Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025))

Abstract: Text simplification is an active research topic with applications in multiple domains. In a simplification pipeline, assessment of text difficulty plays a crucial role as a quality control mechanism it acts as a critic and guides models to generate text at the difficulty level required by the user. This paper presents our Difficulty-aware Text Simplification System. We evaluate our pipeline using the TSAR shared task dataset and discuss challenges in constructing corpora for training models to assess text difficulty.
Uncertainty-driven Partial Diacritization for Arabic Text
Authors: Humaid Ali Alblooshi; Artem Shelmanov; Hanan Aldarmaki
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: We present an uncertainty-based approach to Partial Diacritization (PD) for Arabic text. We evaluate three uncertainty metrics for this task: Softmax Response, BALD via MC-dropout, and Mahalanobis Distance. We further introduce a lightweight Confident Error Regularizer to improve model calibration. Our preliminary exploration illustrates possible ways to use uncertainty estimation for selectively retaining or discarding diacritics in Arabic text with an analysis of performance in terms of correlation with diacritic error rates. For instance, the model can be used to detect words with high diacritic error rates which tend to have higher uncertainty scores at inference time. On the Tashkeela dataset, the method maintains low Diacritic Error Rate while reducing the amount of visible diacritics on the text by up to 50% with thresholding-based retention.
Phases of Uncertainty: Confidence–Calibration Dynamics in Language Model Training
Authors: Aneesh Durai
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Autoregressive language models achieve strong performance across a wide range of natural language processing (NLP) tasks, yet their uncertainty estimates remain poorly understood, particularly during training. Prior work has primarily evaluated calibration and out-of-distribution (OOD) robustness at the final checkpoint, overlooking the dynamics that unfold earlier. We introduce a phase-based framework for tracking uncertainty metrics—including expected calibration error (ECE) and Kullback–Leibler (KL) divergence—across distinct stages of training. Using GPT-2 models trained across multiple random seeds, we find that uncertainty dynamics follow a consistent set of phases: models begin conservative and relatively well calibrated, but later phases introduce a paradoxical decoupling where confidence increases even as calibration worsens, especially under distribution shift. This paradox implies that the final checkpoint is not always the most reliable for deployment and motivates phase-aware strategies such as dynamic checkpoint selection or targeted calibration. Our findings highlight that uncertainty should be understood as a training-dependent property rather than a static one, opening new directions for scaling this framework to larger models, tasks, and distribution shift scenarios.
Beyond Human Judgment: A Bayesian Evaluation of LLMs’ Moral Values Understanding
Authors: Maciej Skorski; Alina Landowska
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: How do Large Language Models understand moral dimensions compared to humans?This first comprehensive large-scale Bayesian evaluation of leading language models provides the answer. In contrast to prior approaches based on deterministic ground truth (obtained via majority or inclusion consensus), we obtain the labels by modelling annotators’ disagreement to capture both aleatoric uncertainty (inherent human disagreement) and epistemic uncertainty (model domain sensitivity).We evaluated Claude Sonnet 4, DeepSeek-V3, and Llama 4 Maverick across 250K+ annotations from nearly 700 annotators in 100K+ texts spanning social networks, news, and discussion forums. Our GPU-optimized Bayesian framework processed 1M+ model queries, revealing that AI models generally rank among the top 25% of annotators in terms of balanced accuracy, substantially better than average humans.Importantly, we find that AI produces far fewer false negatives than humans, highlighting their more sensitive moral detection capabilities.
Do Large Language Models Know When Not to Answer in Medical QA?
Authors: Sravanthi Machcha; Sushrita Yerra; Sharmin Sultana; Hong Yu; Zonghai Yao
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Uncertainty awareness is essential for large language models (LLMs), particularly in safety-critical domains such as medicine where erroneous or hallucinatory outputs can cause harm. Yet most evaluations remain centered on accuracy, offering limited insight into model confidence and its relation to abstention. In this work, we present preliminary experiments that combine conformal prediction with abstention-augmented and perturbed variants of medical QA datasets. Our early results suggest a positive link between uncertainty estimates and abstention decisions, with this effect amplified under higher difficulty and adversarial perturbations. These findings highlight abstention as a practical handle for probing model reliability in medical QA.
The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models
Authors: Esteban Garces Arias; Julian Rodemann; Christian Heumann
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Understanding uncertainty in large language models remains a fundamental challenge, particularly in creative tasks where multiple valid outputs exist. We present a geometric framework using credal sets—convex hulls of probability distributions—to quantify and decompose uncertainty in neural text generation, calibrated against human creative variation. Analyzing 500 creative writing prompts from the dataset with 10 unique human continuations each, we evaluate four language models across five decoding strategies, generating 100,000 stories. Our credal set analysis reveals substantial gaps in capturing human creative variation, with the best model-human calibration reaching only 0.434 (Gemma-2B with temperature 0.7). We decompose total uncertainty into epistemic and aleatoric components, finding that the choice of decoding strategy contributes 39.4% to 72.0% of total epistemic uncertainty. Model scale shows weak correlation with calibration quality and no significant difference exists between base and instruction-tuned models in calibration quality. Our geometric framework provides actionable insights for improving generation systems for human-AI creative alignment. We release our complete experimental framework at https://github.com/EstebanGarces/uncertainHuman.
Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios
Authors: Autumn Toney; Ryan Wails
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from token logits, with derived probability and entropy values offering insight into performance on the prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between token certainty and alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2) alignment between token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all prompt scenarios, their token-level probability and entropy values consistently diverge from the corresponding theoretical distributions.
The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation
Authors: Timothy Pistotti; Michael J. Witbrock; Dr Padriac Amato Tahua O’Leary; Jason Brown
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation’s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text’s departure from rigid translationese.
Asking a Language Model for Diverse Responses
Authors: Sergey Troshin; Irina Saparina; Antske Fokkens; Vlad Niculae
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Large language models increasingly rely on explicit reasoning chains and can produce multiple plausible responses for a given context. We study the candidate sampler that produces the set of plausible responses contrasting the ancestral (parallel) sampling against two alternatives: enumeration, which asks the model to produce n candidates in one pass, and iterative sampling, which proposes candidates sequentially while conditioning on the currently generated response set. Under matched budgets, we compare these samplers on quality, lexical and computation flow diversity, and efficiency. Our empirical results demonstrate that enumeration and iterative strategies result in higher diversity at comparable quality. Our findings highlight the potential of simple non-independent sampling strategies to improve response diversity without sacrificing generation quality.
Learning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction
Authors: Tobias Groot; Salo Lacunes; Evgenia Ilia
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Natural language generation (NLG) tasks are often subject to inherent variability; e.g. predicting the next word given a context has multiple valid responses, evident when asking multiple humans to complete the task. While having language models (LMs) that are aligned pluralistically, so that they are able to reproduce well the inherent diversity in perspectives of an entire population of interest is clearly beneficial, Ilia and Aziz (2024) show that LMs do not reproduce this type of linguistic variability well. They speculate this inability might stem from the lack of consistent training of LMs with data reflecting this type of inherent variability. As such, we investigate whether training LMs on multiple plausible word continuations per context can improve their ability to reproduce human linguistic variability for next-word prediction. We employ fine-tuning techniques for pre-trained and instruction-tuned models; and demonstrate their potential when fine-tuning GPT-2 and Mistral-7B-IT, using Provo Corpus. Our evaluation, which measures divergence among empirically estimated human and model next-word distributions across contexts before and after fine-tuning, shows that our multi-label fine-tuning improves the LMs’ ability to reproduce linguistic variability; both for contexts that admit higher and lower variability.
HALLUCINOGEN: Benchmarking Hallucination in Implicit Reasoning within Large Vision Language Models
Authors: Ashish Seth; Dinesh Manocha; Chirag Agarwal
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks. However, these models still suffer from hallucinations, particularly when required to implicitly recognize or infer diverse visual entities from images for complex vision-language tasks. To address this challenge, we propose HALLUCINOGEN, a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks to evaluate the extent of hallucination in state-of-the-art LVLMs. Our benchmark provides a comprehensive study of the implicit reasoning capabilities of these models by first categorizing visual entities based on the ease of recognition in an image as either salient (prominent, visibly recognizable objects such as a car) or latent entities (such as identifying a disease from a chest X-ray), which are not readily visible and require domain knowledge or contextual reasoning for accurate inference. Next, we design hallucination attacks for both types of entities to assess hallucinations in LVLMs while performing various vision-language tasks, such as locating or reasoning about specific entities within an image, where models must perform implicit reasoning by verifying the existence of the queried entity within the image before generating responses. Finally, our extensive evaluations of eleven LVLMs, including powerful open-source models (like LLaMA-3.2 and DeepSeek-V2), commercial models like Gemini, and two hallucination mitigation strategies across multiple datasets, demonstrate that current LVLMs remain susceptible to hallucination attacks.
Uncertainty in Semantic Language Modeling with PIXELS
Authors: Stefania Radu; Marco Zullich; Matias Valdenegro-Toro
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Pixel-based language models aim to solve the vocabulary bottleneck problem in language modeling, but the challenge of uncertainty quantification remains open. The novelty of this work consists of analysing uncertainty and confidence in pixel-based language models across 18 languages and 7 scripts, all part of 3 semantically challenging tasks. This is achieved through several methods such as Monte Carlo Dropout, Transformer Attention, and Ensemble Learning. The results suggest that pixel-based models underestimate uncertainty when reconstructing patches. The uncertainty is also influenced by the script, with Latin languages displaying lower uncertainty. The findings on ensemble learning show better performance when applying hyperparameter tuning during the named entity recognition and question-answering tasks across 16 languages.
Confidence Calibration in Large Language Model-Based Entity Matching
Authors: Iris Kamsteeg; Juan Cardenas-Cartagena; Floris van Beers; Tsegaye Misikir Tashu; Matias Valdenegro-Toro
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: This research aims to explore the intersection of Large Language Models and confidence calibration in Entity Matching. To this end, we perform an empirical study to compare baseline RoBERTa confidences for an Entity Matching task against confidences that are calibrated using Temperature Scaling, Monte Carlo Dropout and Ensembles. We use the Abt-Buy, DBLP-ACM, iTunes-Amazon and Company datasets. The findings indicate that the proposed modified RoBERTa model exhibits a slight overconfidence, with Expected Calibration Error scores ranging from 0.0043 to 0.0552 across datasets. We find that this overconfidence can be mitigated using Temperature Scaling, reducing Expected Calibration Error scores by up to 23.83%.
Consensus or Conflict? Fine-Grained Evaluation of Conflicting Answers in Question-Answering
Authors: Eviatar Nachshoni; Arie Cattan; Shmuel Amar; Ori Shapira; Ido Dagan
Venue: EMNLP 2025 (Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025))

Abstract: Large Language Models (LLMs) have demonstrated strong performance in question answering (QA) tasks. However, Multi-Answer Question Answering (MAQA), where a question may have several valid answers, remains challenging. Traditional QA settings often assume consistency across evidences, but MAQA can involve conflicting answers. Constructing datasets that reflect such conflicts is costly and labor-intensive, while existing benchmarks often rely on synthetic data, restrict the task to yes/no questions, or apply unverified automated annotation. To advance research in this area, we extend the conflict-aware MAQA setting to require models not only to identify all valid answers, but also to detect specific conflicting answer pairs, if any. To support this task, we introduce a novel cost-effective methodology for leveraging fact-checking datasets to construct NATCONFQA, a new benchmark for realistic, conflict-aware MAQA, enriched with detailed conflict labels, for all answer pairs. We evaluate eight high-end LLMs on NATCONFQA, revealing their fragility in handling various types of conflicts and the flawed strategies they employ to resolve them.