GenAI4ELab/papercli-papers huggingface dataset:
On using distribution-based compositionality assessment to evaluate compositional generalisation in machine translation
Authors: Anssi Moisio; Mathias Creutz; Mikko Kurimo
Venue: EMNLP 2023 (Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP)

Abstract: Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems’ capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca.
Shifted PAUQ: Distribution shift in text-to-SQL
Authors: Oleg Somov; Elena Tutubalina
Venue: EMNLP 2023 (Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP)

Abstract: Semantic parsing plays a pivotal role in advancing the accessibility of human-computer interaction on a large scale. Spider, a widely recognized dataset for text2SQL, contains a wide range of natural language (NL) questions in English and corresponding SQL queries. Original splits of Spider and its adapted to Russian language and improved version, PAUQ, assume independence and identical distribution of training and testing data (i.i.d split). In this work, we propose a target length split and multilingual i.i.d split to measure compositionality and cross-language generalization. We present experimental results of popular text2SQL models on original, multilingual, and target length splits. We also construct a context-free grammar for the evaluation of compositionality in text2SQL in an out-of-distribution setting. We make the splits publicly available on HuggingFace hub via https://huggingface.co/datasets/composite/pauq
Literary Intertextual Semantic Change Detection: Application and Motivation for Evaluating Models on Small Corpora
Authors: Jackson Ehrenworth; Katherine A. Keith
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Lexical semantic change detection is the study of how words change meaning between corpora. While Schlechtweg et al. (2020) standardized both datasets and evaluation metrics for this shared task, for those interested in applying semantic change detection models to small corpora—e.g., in the digital humanities—there is a need for evaluation involving much smaller datasets. We present a method and open-source code pipeline for downsampling the SemEval-2020 Task 1 corpora while preserving gold standard measures of semantic change. We then evaluate several state-of-the-art models trained on these downsampled corpora and find both dramatically decreased performance (average 67% decrease) and high variance. We also propose a novel application to the digital humanities and provide a case study demonstrating that semantic change detection can be used in an exploratory manner to produce insightful avenues of investigation for literary scholars.
Domain-Adapting BERT for Attributing Manuscript, Century and Region in Pre-Modern Slavic Texts
Authors: Piroska Lendvai; Uwe Reichel; Anna Jouravel; Achim Rabus; Elena Renje
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Our study presents a stratified dataset compiled from six different Slavic bodies of text, for cross-linguistic and diachronic analyses of Slavic Pre-Modern language variants. We demonstrate unsupervised domain adaptation and supervised finetuning of BERT on these low-resource, historical Slavic variants, for the purposes of provenance attribution in terms of three downstream tasks: manuscript, century and copying region classification.The data compilation aims to capture diachronic as well as regional language variation and change: the texts were written in the course of roughly a millennium, incorporating language variants from the High Middle Ages to the Early Modern Period, and originate from a variety of geographic regions. Mechanisms of language change in relatively small portions of such data have been inspected, analyzed and typologized by Slavists manually; our contribution aims to investigate the extent to which the BERT transformer architecture and pretrained models can benefit this process. Using these datasets for domain adaptation, we could attribute temporal, geographical and manuscript origin on the level of text snippets with high F-scores. We also conducted a qualitative analysis of the models’ misclassifications.
Representing and Computing Uncertainty in Phonological Reconstruction
Authors: Johann-Mattis List; Nathan Hill; Robert Forkel; Frederic Blum
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Despite the inherently fuzzy nature of reconstructions in historical linguistics, most scholars do not represent their uncertainty when proposing proto-forms. With the increasing success of recently proposed approaches to automating certain aspects of the traditional comparative method, the formal representation of proto-forms has also improved. This formalization makes it possible to address both the representation and the computation of uncertainty. Building on recent advances in supervised phonological reconstruction, during which an algorithm learns how to reconstruct words in a given proto-language relying on previously annotated data, and inspired by improved methods for automated word prediction from cognate sets, we present a new framework that allows for the representation of uncertainty in linguistic reconstruction and also includes a workflow for the computation of fuzzy reconstructions from linguistic data.
GHisBERT – Training BERT from scratch for lexical semantic investigations across historical German language stages
Authors: Christin Beck; Marisa Köllner
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: While static embeddings have dominated computational approaches to lexical semantic change for quite some time, recent approaches try to leverage the contextualized embeddings generated by the language model BERT for identifying semantic shifts in historical texts. However, despite their usability for detecting changes in the more recent past, it remains unclear how well language models scale to investigations going back further in time, where the language differs substantially from the training data underlying the models. In this paper, we present GHisBERT, a BERT-based language model trained from scratch on historical data covering all attested stages of German (going back to Old High German, c. 750 CE). Given a lack of ground truth data for investigating lexical semantic change across historical German language stages, we evaluate our model via a lexical similarity analysis of ten stable concepts. We show that, in comparison with an unmodified and a fine-tuned German BERT-base model, our model performs best in terms of assessing inter-concept similarity as well as intra-concept similarity over time. This in turn argues for the necessity of pre-training historical language models from scratch when working with historical linguistic data.
A longitudinal study about gradual changes in the Iranian Online Public Sphere pre and post of ‘Mahsa Moment’: Focusing on Twitter
Authors: Sadegh Jafari; Amin Fathi; Abolfazl Hajizadegan; Amirmohammad Kazemeini; Sauleh Eetemadi
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Mahsa Amini’s death shocked Iranian society. The effects of this event and the subsequent tragedies in Iran not only in realspace but also in cyberspace, including Twitter, were tremendous and unimaginable. We explore how Twitter has changed after Mahsa Amini’s death by analyzing the sentiments of Iranian users in the 90 days after this event. Additionally, we track the change in word meaning and each word’s neighboring words. Finally, we use word clustering methods for topic modeling.
Political dogwhistles and community divergence in semantic change
Authors: Max Boholm; Asad Sayeed
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: We test whether the development of political dogwhistles can be observed using language change measures; specifically, does the development of a “hidden” message in a dogwhistle show up as differences in semantic change between communities over time? We take Swedish-language dogwhistles related to the on-going immigration debate and measure differences over time in their rate of semantic change between two Swedish-language community forums, Flashback and Familjeliv, the former representing an in-group for understanding the “hidden” meaning of the dogwhistles. We find that multiple measures are sensitive enough to detect differences over time, in that the meaning changes in Flashback over the relevant time period but not in Familjeliv. We also examine the sensitivity of multiple modeling approaches to semantic change in the matter of community divergence.
EvoSem: A database of polysemous cognate sets
Authors: Mathieu Dehouck; Alex François; Siva Kalyan; Martial Pastor; David Kletz
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Polysemies, or “colexifications”, are of great interest in cognitive and historical linguistics, since meanings that are frequently expressed by the same lexeme are likely to be conceptually similar, and lie along a common pathway of semantic change. We argue that these types of inferences can be more reliably drawn from polysemies of cognate sets (which we call “dialexifications”) than from polysemies of lexemes. After giving a precise definition of dialexification, we introduce Evosem, a cross-linguistic database of etymologies scraped from several online sources. Based on this database, we measure for each pair of senses how many cognate sets include them both — i.e. how often this pair of senses is “dialexified”. This allows us to construct a weighted dialexification graph for any set of senses, indicating the conceptual and historical closeness of each pair. We also present an online interface for browsing our database, including graphs and interactive tables. We then discuss potential applications to NLP tasks and to linguistic research.
Multi-lect automatic detection of Swadesh list items from raw corpus data in East Slavic languages
Authors: Ilia Afanasev
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: The article introduces a novel task of multi-lect automatic detection of Swadesh list items from raw corpora. The task aids the early stageof historical linguistics study by helping the researcher compile word lists for further analysis.In this paper, I test multi-lect automatic detection on the East Slavic lects’ data. The training data consists of Ukrainian, Belarusian, and Russian material. I introduce a new dataset for the Ukrainian language. I implement data augmentation techniques to give automatic tools a better understanding of the searched value. The test data consists of the Old East Slavic texts.I train HMM, CRF, and mBERT models, then test and evaluate them by harmonic F1 score. The baseline is a Random Forest classifier. I introduce two different subtasks: the search for new Swadesh list items, and the search for the known Swadesh list items in new lects of the well-established group. The first subtask, given the simultaneously diverse and vague nature of the Swadesh list, currently presents an almost unbeatable challenge for machine learning methods. The second subtask, on the other hand, is easier, and the mBERT model achieves a 0.57 F1 score. This is an impressive result, given how hard it is to formalise the token belonging to a very specific and thematically diverse set of concepts.
Anchors in Embedding Space: A Simple Concept Tracking Approach to Support Conceptual History Research
Authors: Jetske Adams; Martha Larson; Jaap Verheul; Michael Boyden
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: We introduce a simple concept tracking approach to support conceptual history research. Building on the existing practices of conceptual historians, we use dictionaries to identify “anchors”, which represent primary dimensions of meaning of a concept. Then, we create a plot showing how a key concept has evolved over time in a historical corpus in relation to these dimensions. We demonstrate the approach by plotting the change of several key concepts in the COHA corpus.
ChiWUG: A Graph-based Evaluation Dataset for Chinese Lexical Semantic Change Detection
Authors: Jing Chen; Emmanuele Chersoni; Dominik Schlechtweg; Jelena Prokic; Chu-Ren Huang
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: Recent studies suggested that language models are efficient tools for measuring lexical semantic change. In our paper, we present the compilation of the first graph-based evaluation dataset for lexical semantic change in the context of the Chinese language, specifically covering the periods of pre- and post- Reform and Opening Up. Exploiting the existing framework DURel, we collect over 61,000 human semantic relatedness judgments for 40 targets. The inferred word usage graphs and semantic change scores provide a basis for visualization and evaluation of semantic change.
Towards Detecting Lexical Change of Hate Speech in Historical Data
Authors: Sanne Hoeken; Sophie Spliethoff; Silke Schwandt; Sina Zarrieß; Özge Alacam
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: The investigation of lexical change has predominantly focused on generic language evolution, not suited for detecting shifts in a particular domain, such as hate speech. Our study introduces the task of identifying changes in lexical semantics related to hate speech within historical texts. We present an interdisciplinary approach that brings together NLP and History, yielding a pilot dataset comprising 16th-century Early Modern English religious writings during the Protestant Reformation. We provide annotations for both semantic shifts and hatefulness on this data and, thereby, combine the tasks of Lexical Semantic Change Detection and Hate Speech Detection. Our framework and resulting dataset facilitate the evaluation of our applied methods, advancing the analysis of hate speech evolution.
Changing usage of Low Saxon auxiliary and modal verbs
Authors: Janine Siewert; Martijn Wieling; Yves Scherrer
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: We investigate the usage of auxiliary and modal verbs in Low Saxon dialects from both Germany and the Netherlands based on word vectors, and compare developments in the modern language to Middle Low Saxon. Although most of these function words have not been affected by lexical replacement, changes in usage that likely at least partly result from contact with the state languages can still be observed.
Semantic Shifts in Mental Health-Related Concepts
Authors: Naomi Baes; Nick Haslam; Ekaterina Vylomova
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: The present study evaluates semantic shifts in mental health-related concepts in two diachronic corpora spanning 1970-2016, one academic and one general. It evaluates whether their meanings have broadened to encompass less severe phenomena and whether they have become more pathology related. It applies a recently proposed methodology (Baes et al., 2023) to examine whether words collocating with a sample of mental health concepts have become less emotionally intense and develops a new way to examine whether the concepts increasingly co-occur with pathology-related terms. In support of the first hypothesis, mental health-related concepts became associated with less emotionally intense language in the psychology corpus (addiction, anger, stress, worry) and in the general corpus (addiction, grief, stress, worry). In support of the second hypothesis, mental health-related concepts came to be more associated with pathology-related language in psychology (addiction, grief, stress, worry) and in the general corpus (grief, stress). Findings demonstrate that some mental health concepts have become normalized and/or pathologized, a conclusion with important social and cultural implications.
Automating Sound Change Prediction for Phylogenetic Inference: A Tukanoan Case Study
Authors: Kalvin Chang; Nathaniel Robinson; Anna Cai; Ting Chen; Annie Zhang; David Mortensen
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: We describe a set of new methods to partially automate linguistic phylogenetic inference given (1) cognate sets with their respective protoforms and sound laws, (2) a mapping from phones to their articulatory features and (3) a typological database of sound changes.We train a neural network on these sound change data to weight articulatory distances between phones and predict intermediate sound change steps between historical protoforms and their modern descendants, replacing a linguistic expert in part of a parsimony-based phylogenetic inference algorithm. In our best experiments on Tukanoan languages, this method produces trees with a Generalized Quartet Distance of 0.12 from a tree that used expert annotations, a significant improvement over other semi-automated baselines. We discuss potential benefits and drawbacks to our neural approach and parsimony-based tree prediction. We also experiment with a minimal generalization learner for automatic sound law induction, finding it less effective than sound laws from expert annotation. Our code is publicly available.
Scent and Sensibility: Perception Shifts in the Olfactory Domain
Authors: Teresa Paccosi; Stefano Menini; Elisa Leonardelli; Ilaria Barzon; Sara Tonelli
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: In this work, we investigate olfactory perception shifts, analysing how the description of the smells emitted by specific sources has changed over time. We first create a benchmark of selected smell sources, relying upon existing historical studies related to olfaction. We also collect an English text corpus by retrieving large collections of documents from freely available resources, spanning from 1500 to 2000 and covering different domains. We label such corpus using a system for olfactory information extraction inspired by frame semantics, where the semantic roles around the smell sources in the benchmark are marked. We then analyse how the roles describing Qualities of smell sources change over time and how they can contribute to characterise perception shifts, also in comparison with more standard statistical approaches.
From Diachronic to Contextual Lexical Semantic Change: Introducing Semantic Difference Keywords (SDKs) for Discourse Studies
Authors: Isabelle Gribomont
Venue: EMNLP 2023 (Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change)

Abstract: This paper introduces the concept of Semantic Difference Keywords (SDKs). We define SDKs as keywords selected because of a comparatively high semantic difference between their use in two or more corpora. They are extracted by applying methods developed to identify diachronic Lexical Semantic Change. Like statistical keywords, most commonly used in quantitative discourse studies, SDKs capture the distinctiveness of a target corpus. However, they do not do so because they are used significantly more often or more consistently, but because they are used significantly differently. The case study presented in this paper shows that SDKs are successful in identifying concepts which are contested, i.e., sites of “semantic struggles” (CITATION). SDKs are therefore a useful contribution to (computational) discourse studies and text-based Digital Humanities more broadly.
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Authors: Jiaan Wang; Yunlong Liang; Fandong Meng; Zengkui Sun; Haoxiang Shi; Zhixu Li; Jinan Xu; Jianfeng Qu; Jie Zhou
Venue: EMNLP 2023 (Proceedings of the 4th New Frontiers in Summarization Workshop)

Abstract: Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.