GenAI4ELab/papercli-papers huggingface dataset:
SRI-B’s Systems for IWSLT 2023 Dialectal and Low-resource Track: Marathi-Hindi Speech Translation
Authors: Balaji Radhakrishnan; Saurabh Agrawal; Raj Prakash Gohil; Kiran Praveen; Advait Vinay Dhopeshwarkar; Abhishek Pandey
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the speech translation systems SRI-B developed for the IWSLT 2023 Evaluation Campaign Dialectal and Low-resource track: Marathi-Hindi Speech Translation. We propose systems for both the constrained (systems are trained only on the datasets provided by the organizers) and the unconstrained conditions (systems can be trained with any resource). For both the conditions, we build end-to-end speech translation networks comprising of a conformer encoder and a transformer decoder. Under both the conditions, we leverage Marathi Automatic Speech Recognition (ASR) data to pre-train the encoder and subsequently train the entire model on the speech translation data. Our results demonstrate that pre-training the encoder with ASR data is a key step in significantly improving the speech translation performance. We also show that conformer encoders are inherently superior to its transformer counterparts for speech translation tasks. Our primary submissions achieved a BLEU% score of 31.2 on the constrained condition and 32.4 on the unconstrained condition. We secured the top position in the constrained condition and second position in the unconstrained condition.
BIT’s System for Multilingual Track
Authors: Zhipeng Wang; Yuhang Guo; Shuoying Chen
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the system we submitted to the IWSLT 2023 multilingual speech translation track, with input being English speech and output being text in 10 target languages. Our system consists of CNN and Transformer, convolutional neural networks downsample speech features and extract local information, while transformer extract global features and output the final results. In our system, we use speech recognition tasks to pre-train encoder parameters, and then use speech translation corpus to train the multilingual speech translation model. We have also adopted other methods to optimize the model, such as data augmentation, model ensemble, etc. Our system can obtain satisfactory results on test sets of 10 languages in the MUST-C corpus.
Matesub: The Translated Subtitling Tool at the IWSLT2023 Subtitling Task
Authors: Simone Perone
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper briefly describes Matesub, the subtitling tool Translated used to participate in the Subtitling shared task at IWSLT 2023. Matesub is a professional web-based tool that combines state-of-the-art AI with a WYSIWYG editor. The automatic generation of subtitles in Matesub is based on a cascade architecture, composed of ASR, text segmenter and MT neural models, which allows covering any pair from about 70 languages and their variants.
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
Authors: Itai Gat; Felix Kreuk; Tu Anh Nguyen; Ann Lee; Jade Copet; Gabriel Synnaeve; Emmanuel Dupoux; Yossi Adi
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: Generative Spoken Language Modeling research focuses on optimizing speech Language Models (LMs) using raw audio recordings without accessing any textual supervision. Such speech LMs usually operate over discrete units obtained from quantizing internal representations of self-supervised models. Although such units show impressive modeling results, their robustness capabilities have not been extensively investigated. This work focuses on improving the robustness of discrete input representations for generative spoken language modeling. First, we formally define how to measure the robustness of such representations to various signal variations that do not alter the spoken information (e.g., time-stretch). Next, we empirically demonstrate how current state-of-the-art representation models lack robustness to such variations. To overcome this, we propose an effective and efficient method to learn robust discrete speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding and modeling metrics. We additionally evaluate our method on the speech-to-speech translation task, considering Spanish-English and French-English translations, and show the proposed approach outperforms the evaluated baselines.
DePA: Improving Non-autoregressive Translation with Dependency-Aware Decoder
Authors: Jiaao Zhan; Qian Chen; Boxing Chen; Wen Wang; Yu Bai; Yang Gao
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: Non-autoregressive machine translation (NAT) models have lower translation quality than autoregressive translation (AT) models because NAT decoders do not depend on previous target tokens in the decoder input. We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input. First, we propose an autoregressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training. Second, we transform the decoder input from the source language representation space to the target language representation space through a novel attentive transformation process, which enables the decoder to better capture target dependencies. DePA can be applied to any fully NAT models. Extensive experiments show that DePA consistently improves highly competitive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.
On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss
Authors: Yihong Liu; Alexandra Chronopoulou; Hinrich Schütze; Alexander Fraser
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: Although unsupervised neural machine translation (UNMT) has achieved success in many language pairs, the copying problem, i.e., directly copying some parts of the input sentence as the translation, is common among distant language pairs, especially when low-resource languages are involved. We find this issue is closely related to an unexpected copying behavior during online back-translation (BT). In this work, we propose a simple but effective training schedule that incorporates a language discriminator loss. The loss imposes constraints on the intermediate translation so that the translation is in the desired language. By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.
Medieval Social Media: Manual and Automatic Annotation of Byzantine Greek Marginal Writing
Authors: Colin Swaelens; Ilse De Vos; Els Lefever
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: In this paper, we present the interim results of a transformer-based annotation pipeline for Ancient and Medieval Greek. As the texts in the Database of Byzantine Book Epigrams have not been normalised, they pose more challenges for manual and automatic annotation than Ancient Greek, normalised texts do. As a result, the existing annotation tools perform poorly. We compiled three data sets for the development of an automatic annotation tool and carried out an inter-annotator agreement study, with a promising agreement score. The experimental results show that our part-of-speech tagger yields accuracy scores that are almost 50 percentage points higher than the widely used rule-based system Morpheus. In addition, error analysis revealed problems related to phenomena also occurring in current social media language.
“Orpheus Came to His End by Being Struck by a Thunderbolt”: Annotating Events in Mythological Sequences
Authors: Franziska Pannach
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: The mythological domain has various ways of expressing events and background knowledge. Using data extracted according to the hylistic approach (Zgoll, 2019), we annotated a data set of 6315 sentences from various mythological contexts and geographical origins, like Ancient Greece and Rome or Mesopotamia, into four categories: single-point events (e.g. actions), durative-constant (background knowledge, continuous states), durative-initial, and durative-resultativ. This data is used to train a classifier, which is able to reliably distinguish event types.
Difficulties in Handling Mathematical Expressions in Universal Dependencies
Authors: Lauren Levine
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: In this paper, we give a brief survey of the difficulties in handling the syntax of mathematical expressions in Universal Dependencies, focusing on examples from English language corpora. We first examine the prevalence and current handling of mathematical expressions in UD corpora. We then examine several strategies for how to approach the handling of syntactic dependencies for such expressions: as multi-word expressions, as a domain appropriate for code-switching, or as approximate to other types of natural language. Ultimately, we argue that mathematical expressions should primarily be analyzed as natural language, and we offer recommendations for the treatment of basic mathematical expressions as analogous to English natural language.
A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement
Authors: Annerose Eichel; Sabine Schulte Im Walde
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: We present a novel dataset for physical and abstract plausibility of events in English. Based on naturally occurring sentences extracted from Wikipedia, we infiltrate degrees of abstractness, and automatically generate perturbed pseudo-implausible events. We annotate a filtered and balanced subset for plausibility using crowd-sourcing, and perform extensive cleansing to ensure annotation quality. In-depth quantitative analyses indicate that annotators favor plausibility over implausibility and disagree more on implausible events. Furthermore, our plausibility dataset is the first to capture abstractness in events to the same extent as concreteness, and we find that event abstractness has an impact on plausibility ratings: more concrete event participants trigger a perception of implausibility.
Annotating and Disambiguating the Discourse Usage of the Enclitic dA in Turkish
Authors: Ebru Ersöyleyen; Deniz Zeyrek; Fırat Öter
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: The Turkish particle dA is a focus-associated enclitic, and it can act as a discourse connective conveying multiple senses, like additive, contrastive, causal etc. Like many other linguistic expressions, it is subject to usage ambiguity and creates a challenge in natural language automatization tasks. For the first time, we annotate the discourse and non-discourse connnective occurrences of dA in Turkish with the PDTB principles. Using a minimal set of linguistic features, we develop binary classifiers to distinguish its discourse connective usage from its other usages. We show that despite its ability to cliticize to any syntactic type, variable position in the sentence and having a wide argument span, its discourse/non-discourse connective usage can be annotated reliably and its discourse usage can be disambiguated by exploiting local cues.
An Active Learning Pipeline for NLU Error Detection in Conversational Agents
Authors: Damian Pascual; Aritz Bercher; Akansha Bhardwaj; Mingbo Cui; Dominic Kohler; Liam Van Der Poel; Paolo Rosso
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: High-quality labeled data is paramount to the performance of modern machine learning models. However, annotating data is a time-consuming and costly process that requires human experts to examine large collections of raw data. For conversational agents in production settings with access to large amounts of user-agent conversations, the challenge is to decide what data should be annotated first. We consider the Natural Language Understanding (NLU) component of a conversational agent deployed in a real-world setup with limited resources. We present an active learning pipeline for offline detection of classification errors that leverages two strong classifiers. Then, we perform topic modeling on the potentially mis-classified samples to ease data analysis and to reveal error patterns. In our experiments, we show on a real-world dataset that by using our method to prioritize data annotation we reach 100% of the performance annotating only 36% of the data. Finally, we present an analysis of some of the error patterns revealed and argue that our pipeline is a valuable tool to detect critical errors and reduce the workload of annotators.
Multi-layered Annotation of Conversation-like Narratives in German
Authors: Magdalena Repp; Petra B. Schumacher; Fahime Same
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: This work presents two corpora based on excerpts from two novels with an informal narration style in German. We performed fine-grained multi-layer annotations of animate referents, assigning local and global prominence-lending features to the annotated referring expressions. In addition, our corpora include annotations of intra-sentential segments, which can serve as a more reliable unit of length measurement. Furthermore, we present two exemplary studies demonstrating how to use these corpora.
Crowdsourcing on Sensitive Data with Privacy-Preserving Text Rewriting
Authors: Nina Mouhammad; Johannes Daxenberger; Benjamin Schiller; Ivan Habernal
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: Most tasks in NLP require labeled data. Data labeling is often done on crowdsourcing platforms due to scalability reasons. However, publishing data on public platforms can only be done if no privacy-relevant information is included. Textual data often contains sensitive information like person names or locations. In this work, we investigate how removing personally identifiable information (PII) as well as applying differential privacy (DP) rewriting can enable text with privacy-relevant information to be used for crowdsourcing. We find that DP-rewriting before crowdsourcing can preserve privacy while still leading to good label quality for certain tasks and data. PII-removal led to good label quality in all examined tasks, however, there are no privacy guarantees given.
Extending an Event-type Ontology: Adding Verbs and Classes Using Fine-tuned LLMs Suggestions
Authors: Jana Straková; Eva Fučíková; Jan Hajič; Zdeňka Urešová
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: In this project, we have investigated the use of advanced machine learning methods, specifically fine-tuned large language models, for pre-annotating data for a lexical extension task, namely adding descriptive words (verbs) to an existing (but incomplete, as of yet) ontology of event types. Several research questions have been focused on, from the investigation of a possible heuristics to provide at least hints to annotators which verbs to include and which are outside the current version of the ontology, to the possible use of the automatic scores to help the annotators to be more efficient in finding a threshold for identifying verbs that cannot be assigned to any existing class and therefore they are to be used as seeds for a new class. We have also carefully examined the correlation of the automatic scores with the human annotation. While the correlation turned out to be strong, its influence on the annotation proper is modest due to its near linearity, even though the mere fact of such pre-annotation leads to relatively short annotation times.
Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling
Authors: Gavin Abercrombie; Dirk Hovy; Vinodkumar Prabhakaran
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: Much work in natural language processing (NLP) relies on human annotation. The majority of this implicitly assumes that annotator’s labels are temporally stable, although the reality is that human judgements are rarely consistent over time. As a subjective annotation task, hate speech labels depend on annotator’s emotional and moral reactions to the language used to convey the message. Studies in Cognitive Science reveal a ‘foreign language effect’, whereby people take differing moral positions and perceive offensive phrases to be weaker in their second languages. Does this affect annotations as well? We conduct an experiment to investigate the impacts of (1) time and (2) different language conditions (English and German) on measurements of intra-annotator agreement in a hate speech labelling task. While we do not observe the expected lower stability in the different language condition, we find that overall agreement is significantly lower than is implicitly assumed in annotation tasks, which has important implications for dataset reproducibility in NLP.
BenCoref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference Annotations
Authors: Shadman Rohan; Mojammel Hossain; Mohammad Mamun Or Rashid; Nabeel Mohammed
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: Coreference Resolution is a well studied problem in NLP. While widely studied for English and other resource-rich languages, research on coreference resolution in Bengali largely remains unexplored due to the absence of relevant datasets. Bengali, being a low-resource language, exhibits greater morphological richness compared to English. In this article, we introduce a new dataset, BenCoref, comprising coreference annotations for Bengali texts gathered from four distinct domains. This relatively small dataset contains 5200 mention annotations forming 502 mention clusters within 48,569 tokens. We describe the process of creating this dataset and report performance of multiple models trained using BenCoref. We anticipate that our work sheds some light on the variations in coreference phenomena across multiple domains in Bengali and encourages the development of additional resources for Bengali. Furthermore, we found poor crosslingual performance at zero-shot setting from English, highlighting the need for more language-specific resources for this task.
Annotators-in-the-loop: Testing a Novel Annotation Procedure on Italian Case Law
Authors: Emma Zanoli; Matilde Barbini; Davide Riva; Sergio Picascia; Emanuela Furiosi; Stefano D’Ancona; Cristiano Chesi
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: The availability of annotated legal corpora is crucial for a number of tasks, such as legal search, legal information retrieval, and predictive justice. Annotation is mostly assumed to be a straightforward task: as long as the annotation scheme is well defined and the guidelines are clear, annotators are expected to agree on the labels. This is not always the case, especially in legal annotation, which can be extremely difficult even for expert annotators. We propose a legal annotation procedure that takes into account annotator certainty and improves it through negotiation. We also collect annotator feedback and show that our approach contributes to a positive annotation environment. Our work invites reflection on often neglected ethical concerns regarding legal annotation.
Annotating Decomposition in Time: Three Approaches for Again
Authors: Martin Kopf; Remus Gergel
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: This submission reports on a three-part series of original methods geared towards producing semantic annotations for the decompositional marker “again”. The three methods are (i) exhaustive expert annotation based on a comprehensive set of guidelines, (ii) extension of expert annotation by predicting presuppositions with a Multinomial Naïve Bayes classifier in the context of a meta-analysis to optimize feature selection and (iii) quality-controlled crowdsourcing with ensuing evaluation and KMeans clustering of annotation vectors.
How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Authors: Shafiuddin Rehan Ahmed; Abhijnan Nath; Michael Regan; Adam Pollins; Nikhil Krishnaswamy; James H. Martin
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97% recall while substantially reducing the workload required by a fully manual annotation process.
Pragmatic Annotation of Articles Related to Police Brutality
Authors: Tess Feyen; Alda Mari; Paul Portner
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: The annotation task we elaborated aims at describing the contextual factors that influence the appearance and interpretation of moral predicates, in newspaper articles on police brutality, in French and in English. The paper provides a brief review of the literature on moral predicates and their relation with context. The paper also describes the elaboration of the corpus and the ontology. Our hypothesis is that the use of moral adjectives and their appearance in context could change depending on the political orientation of the journal. We elaborated an annotation task to investigate the precise contexts discussed in articles on police brutality. The paper concludes by describing the study and the annotation task in details.
The RST Continuity Corpus
Authors: Debopam Das; Markus Egg
Venue: ACL 2023 (Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII))

Abstract: We present the RST Continuity Corpus (RST-CC), a corpus of discourse relations annotated for continuity dimensions. Continuity or discontinuity (maintaining or shifting deictic centres across discourse segments) is an important property of discourse relations, but the two are correlated in greatly varying ways. To analyse this correlation, the relations in the RST-CC are annotated using operationalised versions of Givón’s (1993) continuity dimensions. We also report on the inter-annotator agreement, and discuss recurrent annotation issues. First results show substantial variation of continuity dimensions within and across relation types.