GenAI4ELab/papercli-papers huggingface dataset:
Pre-annotation Matters: A Comparative Study on POS and Dependency Annotation for an Alsatian Dialect
Authors: Delphine Bernhard; Nathanaël Beiner; Barbara Hoff
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: The annotation of corpora for lower-resource languages can benefit from automatic pre-annotation to increase the throughput of the annotation process in a a context where human resources are scarce. However, this can be hindered by the lack of available pre-annotation tools. In this work, we compare three pre-annotation methods in zero-shot or near-zero-shot contexts for part-of-speech (POS) and dependency annotation of an Alsatian Alemannic dialect. Our study shows that good levels of annotation quality can be achieved, with human annotators adapting their correction effort to the perceived quality of the pre-annotation. The pre-annotation tools also vary in efficiency depending on the task, with better global results for a system trained on closely related languages and dialects.
Where it’s at: Annotating Verb Placement Types in Learner Language
Authors: Josef Ruppenhofer; Annette Portmann; Matthias Schwendemann; Christine Renker; Katrin Wisniewski; Torsten Zesch
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: The annotation of learner language is an often ambiguous and challenging task. It is therefore surprising that in Second Language Acquisition research, information on annotation quality is hardly ever published. This is also true for verb placement, a linguistic feature that has re- ceived much attention within SLA. This paper presents an annotation on verb placement in German learner texts at different proficiency levels. We argue that as part of the annotation process target hypotheses should be provided as ancillary annotations that make explicit each annotator’s interpretation of a learner sentence. Our study demonstrates that verb placement can be annotated with high agreement between multiple annotators, for texts at all proficiency levels and across sentences of varying complex- ity. We release our corpus with annotations by four annotators on more than 600 finite clauses sampled across 5 CEFR levels.
ICLE-RC: International Corpus of Learner English for Relative Clauses
Authors: Debopam Das; Izabela Czerniak; Peter Bourgonje
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: We present the ICLE-RC, a corpus of learner English texts annotated for relative clauses and related phenomena. The corpus contains a collection of 144 academic essays from the International Corpus of Learner English (ICLE; Granger et al., 2002), representing six L1 backgrounds – Finnish, Italian, Polish, Swedish, Turkish, and Urdu. These texts are annotated for over 900 relative clauses, with respect to a wide array of lexical, syntactic, semantic, and discourse features. The corpus also provides annotation of over 400 related phenomena (it-clefts, pseudo-clefts, existential-relatives, etc.). Here, we describe the corpus annotation framework, report on the IAA study, discuss the prospects of (semi-)automating annotation, and present the first results from our corpus analysis. We envisage the ICLE-RC to be used as a valuable resource for research on relative clauses in SLA, language typology, World Englishes, and discourse analysis.
ExpLay: A new Corpus Resource for the Research on Expertise as an Influential Factor on Language Production
Authors: Carmen Schacht; Renate Delucchi Danhier
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: This paper introduces the ExpLay-Pipeline, a novel semi-automated processing tool designed for the analysis of language production data from experts in comparison to the language production of a control group of laypeople. The pipeline combines manual annotation and curation with state-of-the-art machine learning and rule-based methods, following a silver standard approach. It integrates various analysis modules specifically for the syntactic and lexical evaluation of parsed linguistic data. While implemented initially for the creation of the ExpLay-Corpus, it is designed for the processing of linguistic data in general. The paper details the design and implementation of this pipeline.
Towards Resource-Rich Mizo and Khasi in NLP: Resource Development, Synthetic Data Generation and Model Building
Authors: Soumyadip Ghosh; Henry Lalsiam; Dorothy Marbaniang; Gracious Mary Temsen; Rahul Mishra; Parameswari Krishnamurthy
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: In the rapidly evolving field of Natural Language Processing (NLP), Indian regional languages remain significantly underrepresented due to their limited digital presence and lack of annotated resources. This work presents the first comprehensive effort toward developing high quality linguistic datasets for two extremely low resource languages Mizo and Khasi. We introduce human annotated, gold standard datasets for three core NLP tasks: Part-of-Speech (POS) tagging, Named Entity Recognition (NER), and Keyword Identification. To overcome annotation bottlenecks in NER, we further explore a synthetic data generation pipeline involving translation from Hindi and cross lingual word alignment. For POS tagging, we adopt and subsequently modify the Universal Dependencies (UD) framework to better suit the linguistic characteristics of Mizo and Khasi, while custom annotation guidelines are developed for NER and Keyword Identification. The constructed datasets are evaluated using multilingual language models, demonstrating that structured resource development, coupled with gradual fine-tuning, yields significant improvements in performance. This work represents a critical step toward advancing linguistic resources and computational tools for Mizo and Khasi.
Creating Hierarchical Relations in a Multilingual Event-type Ontology
Authors: Zdeňka Urešová; Eva Fučíková; Jan Hajič
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: This paper describes the work on hierarchization of the SynSemClass event-type ontology. The original resource has been extended by a hierarchical structure to model specialization and generalization relations between classes that are formally and technically unrelated in the original ontology. The goal is to enable one to use the ontology enriched by the hierarchical concepts for annotation of running texts in symbolic meaning representations, such as UMR or PDT. The hierarchy is in principle built bottom-up, based on existing SSC classes (concepts). This approach differs from other approaches to semantic classes, such as in WordNet or VerbNet. Although the hierarchical relations are similar, the underlying nodes in the hierarchy are not. In this paper, we describe the challenges related to the principles chosen: single-tree constraint and finding features for the definitions of specificity/generality. Also, a pilot inter-annotator experiment is described that shows the difficulty of the hierarchization task.
Visual Representations of Temporal Relations between Events and Time Expressions in News Stories
Authors: Evelin Amorim; António Leal; Nana Yu; Purificação Moura Silvano; Alipio Mario Jorge
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: High-quality annotation is paramount for effective predictions of machine learning models. When the annotation is dense, achieving superior human labeling can be challenging since the most used annotation tools present an overloaded visualization of labels. Thus, we present a tool for viewing annotations made in corpora, specifically for temporal relations between events and temporal expressions, filling a gap in this type of tool. We focus on narrative text, which is a rich source for these types of elements.
Annotating candy speech in German YouTube comments
Authors: Yulia Clausen; Tatjana Scheffler
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: We describe the phenomenon of candy speech – positive emotional speech in online communication – and introduce a classification of its various types based on the theoretical framework of social interaction by Goffman (1967). We provide a dataset of 46,286 German YouTube comments manually annotated with candy speech
Variety delights (sometimes) - Annotation differences in morphologically annotated corpora
Authors: Andrea Dömötör; Balázs Indig; Dávid Márk Nemeskey
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: The goal of annotation standards is to ensure consistency across different corpora and languages. But do they succeed? In our paper we experiment with morphologically annotated Hungarian corpora of different sizes (ELTE DH gold standard corpus, NYTK-NerKor, and Szeged Treebank) to assess their compatibility as a merged training corpus for morphological analysis and disambiguation. Our results show that combining any two corpora not only failed to improve the results of the trained tagger but even degraded them due the inconsistent annotations. Further analysis of the annotation differences among the corpora revealed inconsistencies of several sources: different theoretical approach, lack of consensus, and tagset conversion issues.
Addressing Variability in Interlinear Glossed Texts with Linguistic Linked Data
Authors: Maxim Ionov; Natalia Patiño Mazzotti
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: In this paper, we identify types of uncertainty in interlinear glossed text (IGT) annotation, a common notation for language data in linguistic research.
Illuminating Logical Fallacies with the CAMPFIRE Corpus
Authors: Austin Blodgett; Claire Bonial; Taylor A. Pellegrin; Melissa Torgbi; Harish Tayyar Madabushi
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: Misinformation detection remains today a challenging task for both annotators and computer systems. While there are many known markers of misinformation—e.g., logical fallacies, propaganda techniques, and improper use of sources—labeling these markers in practice has been shown to produce low agreement as it requires annotators to make several subjective judgments and rely on their own knowledge, external to the text, which may vary between annotators. In this work, we address these challenges with a collection of linguistically-inspired litmus tests. We annotate a schema of 25 logical fallacies, each of which is defined with rigorous tests applied during annotation. Our annotation methodology results in a comparatively high IAA on this task: Cohen’s kappa in the range .69-.86. We release a corpus of 12 documents from various domains annotated with fallacy labels. Additionally, we experiment with a large language model baseline showing that the largest, most advanced models struggle on this challenging task, achieving an F1-score with our gold standard of .08 when excluding non-fallacious examples, compared to human performance of .59-.73. However, we find that prompting methodologies requiring the model to work through our litmus tests improves performance. Our work contributes a robust fallacy annotation schema and annotated corpus, which advance capabilities in this critical research area.
Cheap Annotation of Complex Information: A Study on the Annotation of Information Status in German TEDx Talks
Authors: Carmen Schacht; Tobias Nischk; Oleksandra Yazdanfar; Stefanie Dipper
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: We present an annotation experiment for the annotation of information status in German TEDx Talks with the main goal to reduce annotation costs in terms of time and personnel. We aim for maximizing efficiency while keeping annotation quality constant by testing various different annotation scenarios for an optimal ratio of annotation expenses to resulting quality of the annotations. We choose the RefLex scheme of Riester and Baumann (2017) as a basis for our annotations, refine their annotation guidelines for a more generalizable tagset and conduct the experiment on German Tedx talks, applying different constellations of annotators, curators and correctors to test for an optimal annotation scenario. Our results show that we can achieve equally good and possibly even better results with significantly less effort, by using correctors instead of additional annotators.
Annotating Spatial Descriptions in Literary and Non-Literary Text
Authors: Emilie Sitter; Omar Momen; Florian Steig; J. Berenike Herrmann; Sina Zarrieß
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: Descriptions are a central component of literary texts, yet their systematic identification remains a challenge. This work suggests an approach to identifying sentences describing spatial conditions in literary text. It was developed iteratively on German literary text and extended to non-literary text to evaluate its applicability across textual domains. To assess the robustness of the method, we involved both humans and a selection of state-of-the-art Large Language Models (LLMs) in annotating a collection of sentences regarding their descriptiveness and spatiality. We compare the annotations across human annotators and between humans and LLMs. The main contributions of this paper are: (1) a set of annotation guidelines for identifying spatial descriptions in literary texts, (2) a curated dataset of almost 4,700 annotated sentences of which around 500 are spatial descriptions, produced through in-depth discussion and consensus among annotators, and (3) a pilot study of automating the task of spatial description annotation of German texts. We publish the codes and all human and LLM annotations for the public to be used for research purposes only.
A GitHub-based Workflow for Annotated Resource Development
Authors: Brandon Waldon; Nathan Schneider
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: Computational linguists have long recognized the value of version control systems such as Git (and related platforms, e.g., GitHub) when it comes to managing and distributing computer code. However, the benefits of version control remain under-explored for a central activity within computational linguistics: the development of annotated natural language resources. We argue that researchers can employ version control practices to make development workflows more transparent, efficient, consistent, and participatory. We report a proof-of-concept, GitHub-based solution which facilitated the creation of a legal English treebank.
The Incremental Process of Building an Annotation Scheme for Clinical Narratives in Portuguese: the Contribution of Human Variation Analysis
Authors: Ana Luísa Fernandes; Purificação Silvano; António Leal; Nuno Guimarães; Rita Rb-Silva; Luís Filipe Cunha; Alípio Jorge
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: The development of a robust annotation scheme and corresponding guidelines is crucial for producing annotated datasets that advance both linguistic and computational research. This paper presents a case study that outlines a methodology for designing an annotation scheme and its guidelines, specifically aimed at representing morphosyntactic and semantic information regarding temporal features, as well as medical information in medical reports written in Portuguese. We detail a multi-step process that includes reviewing existing frameworks, conducting an annotation experiment to determine the optimal approach, and designing a model based on these findings. We validated the approach through a pilot experiment where we assessed the reliability and applicability of the annotation scheme and guidelines. In this experiment, two annotators independently annotated a patient’s medical report consisting of six documents using the proposed model, while a curator established the ground truth. The analysis of inter-annotator agreement and the annotation results enabled the identification of sources of human variation and provided insights for further refinement of the annotation scheme and guidelines.
Expanding the UNSC Conflicts Corpus by Incorporating Domain Expert Annotations and LLM Experiments
Authors: Karolina Zaczynska
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: In this work we expand the UN Security Council Conflicts corpus (UNSCon) (Zaczynska at al. 2024) on verbal disputes in diplomatic speeches in English. By including annotations of a UNSC expert, we target the problem of annotating verbal conflicts in a domain with its own culture and rules. On the one hand, we aim to catch all conflicts detected by political domain experts which as a result will be interpretable only by people with advanced political science backgrounds. On the other hand, we target linguistically marked verbalisations that are domain-independent and potentially easier to detect for language models. This balancing act resulted in a refined annotation scheme, and we re-annotate and expand the corpus size by 40% by including new debates. We perform a pilot study using a Large Language Model to include lexical markers of negative evaluation within the conflict spans, which until now were not annotated separately. Classification experiments on the conflict labels in the corpus using Transformer models demonstrate that models trained on the political domain improve the results.
Guidelines for Fine-grained Sentence-level Arabic Readability Annotation
Authors: Nizar Habash; Hanada Taha-Thomure; Khalid N. Elmadani; Zeina Zeino; Abdallah Abushmaes
Venue: ACL 2025 (Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025))

Abstract: This paper presents the annotation guidelines of the Balanced Arabic Readability Evaluation Corpus (BAREC), a large-scale resource for fine-grained sentence-level readability assessment in Arabic. BAREC includes 69,441 sentences (1M+ words) labeled across 19 levels, from kindergarten to postgraduate. Based on the Taha/Arabi21 framework, the guidelines were refined through iterative training with native Arabic-speaking educators. We highlight key linguistic, pedagogical, and cognitive factors in determining readability and report high inter-annotator agreement: Quadratic Weighted Kappa 81.8% (substantial/excellent agreement) in the last annotation phase. We also benchmark automatic readability models across multiple classification granularities (19-, 7-, 5-, and 3-level). The corpus and guidelines are publicly available: http://barec.camel-lab.com.
MultiReflect: Multimodal Self-Reflective RAG-based Automated Fact-Checking
Authors: Uku Kangur; Krish Agrawal; Yashashvi Singh; Ahmed Sabir; Rajesh Sharma
Venue: ACL 2025 (Proceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2025))

Abstract: In this work, we introduce MultiReflect, a novel multimodal self-reflective Retrieval Augmented Generation (RAG)-based automated fact-checking pipeline. MultiReflect is designed to address the challenges of rapidly outdated information, limitations in human query capabilities, and expert knowledge barriers in fact-checking. Our proposed pipeline leverages the latest advancements in Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to enhance fact verification across text and images. Specifically, by integrating multimodal data processing with RAG-based evidence reflection, our system improves the accuracy of fact-checking by utilizing internet-sourced verification. We evaluate our results on the VERITE benchmarks and using several multimodal LLMs, outperforming baselines in binary classification.
CollEX – A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections
Authors: Florian Schneider; Narges Baba Ahmadi; Niloufar Baba Ahmadi; Iris Vogel; Martin Semmann; Chris Biemann
Venue: ACL 2025 (Proceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2025))

Abstract: In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.
VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering
Authors: Zackary Rackauckas; Julia Hirschberg
Venue: ACL 2025 (Proceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2025))

Abstract: We introduce VoxRAG, a modular speech-to-speech retrieval-augmented generation system that bypasses transcription to retrieve semantically relevant audio segments directly from spoken queries. VoxRAG employs silence-aware segmentation, speaker diarization, CLAP audio embeddings, and FAISS retrieval using L2-normalized cosine similarity. We construct a 50-query test set recorded as spoken input by a native English speaker. Retrieval quality was evaluated using LLM-as-a-judge annotations. For very relevant segments, cosine similarity achieved a Recall@10 of 0.34. For somewhat relevant segments, Recall@10 rose to 0.60 and nDCG@10 to 0.27, highlighting strong topical alignment. Answer quality was judged on a 0–2 scale across relevance, accuracy, completeness, and precision, with mean scores of 0.84, 0.58, 0.56, and 0.46 respectively. While precision and retrieval quality remain key limitations, VoxRAG shows that transcription-free speech-to-speech retrieval is feasible in RAG systems.