Reviewerly/exHarmony huggingface dataset:
Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis

We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations.Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label.We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users-historical linguists, lexicographers, or social scientists-to explore and intuitively explain diachronic trajectories of word meaning.Semantic change analysis is only one of many possible applications of the 'definitions as representations' paradigm.Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.
Stubborn Lexical Bias in Data and Models

In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior.However, the presence and effect of such correlations are typically examined feature by feature.We investigate the cumulative impact on a model of many such intersecting features.Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data.We select two tasks-natural language inference and duplicate-question detection-for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment.The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels.We then apply an optimization approach to reweight the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data.Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams).We close with discussion about the implications of our results on what it means to "debias" training data, and how issues of data quality can affect model bias.
PingAnLifeInsurance at SemEval-2023 Task 10: Using Multi-Task Learning to Better Detect Online Sexism

This paper describes our system used in the SemEval-2023 Task 10: Towards Explainable Detection of Online Sexism (Kirk et al., 2023).The harmful effects of sexism on the internet have impacted both men and women, yet current research lacks a fine-grained classification of sexist content.The task involves three hierarchical sub-tasks, which we addressed by employing a multitask-learning framework.To further enhance our system's performance, we pre-trained the roberta-large (Liu et al., 2019b) and deberta-v3-large (He et al., 2021) models on two million unlabeled data, resulting in significant improvements on sub-tasks A and C. In addition, the multitask-learning approach boosted the performance of our models on subtasks A and B. Our system exhibits promising results in achieving explainable detection of online sexism, attaining a test f1-score of 0.8746 on sub-task A (ranking 1st on the leaderboard), and ranking 5th on sub-tasks B and C.
Sequential Path Signature Networks for Personalised Longitudinal Language Modeling

Longitudinal user modeling can provide a strong signal for various downstream tasks.Despite the rapid progress in representation learning, dynamic aspects of modelling individuals' language have only been sparsely addressed.We present a novel extension of neural sequential models using the notion of path signatures from rough path theory, which constitute graduated summaries of continuous paths and have the ability to capture non-linearities in trajectories.By combining path signatures of users' history with contextual neural representations and recursive neural networks we can produce compact time-sensitive user representations.Given the magnitude of mental health conditions with symptoms manifesting in language, we show the applicability of our approach on the task of identifying changes in individuals' mood by analysing their online textual content.By directly integrating signature transforms of users' history in the model architecture we jointly address the two most important aspects of the task, namely sequentiality and temporality.Our approach 1 achieves state-of-the-art performance on macro-average F1 score on the two available datasets for the task, outperforming or performing on-par with state-of-the-art models utilising only historical posts and even outperforming prior models which also have access to future posts of users.
What Knowledge Is Needed? Towards Explainable Memory for kNN-MT Domain Adaptation

kNN-MT presents a new paradigm for domain adaptation by building an external datastore, which usually saves all target language token occurrences in the parallel corpus.As a result, the constructed datastore is usually large and possibly redundant.In this paper, we investigate the interpretability issue of this approach: what knowledge does the NMT model need?We propose the notion of local correctness (LAC) as a new angle, which describes the potential translation correctness for a single entry and for a given neighborhood.Empirical study shows that our investigation successfully finds the conditions where the NMT model could easily fail and need related knowledge.Experiments on six diverse target domains and two language-pairs show that pruning according to local correctness brings a light and more explainable memory for kNN-MT domain adaptation 1 .
Pruning Pre-trained Language Models Without Fine-Tuning

To overcome the overparameterized problem in Pre-trained Language Models (PLMs), pruning is widely used as a simple and straightforward compression method by directly removing unimportant weights.Previous first-order methods successfully compress PLMs to extremely high sparsity with little performance drop.These methods, such as movement pruning, use first-order information to prune PLMs while fine-tuning the remaining weights.In this work, we argue fine-tuning is redundant for first-order pruning, since first-order pruning is sufficient to converge PLMs to downstream tasks without fine-tuning.Under this motivation, we propose Static Model Pruning (SMP), which only uses first-order pruning to adapt PLMs to downstream tasks while achieving the target sparsity level.In addition, we also design a new masking function and training objective to further improve SMP.Extensive experiments at various sparsity levels show SMP has significant improvements over firstorder and zero-order methods.Unlike previous first-order methods, SMP is also applicable to low sparsity and outperforms zero-order methods.Meanwhile, SMP is more parameter efficient than other methods due to it does not require fine-tuning.
Uppsala University at SemEval-2023 Task12: Zero-shot Sentiment Classification for Nigerian Pidgin Tweets

While sentiment classification has been considered a practically solved task for high-resource languages such as English, the scarcity of data for many languages still makes it a challenging task.The AfriSenti-SemEval shared task aims to classify sentiment on Twitter data for 14 low-resource African languages.In our participation, we focus on Nigerian Pidgin as the target language.We have investigated the effect of English monolingual and multilingual pre-trained models on the sentiment classification task for Nigerian Pidgin.Our setup includes zero-shot models (using English, Igbo and Hausa data) and a Nigerian Pidgin finetuned model.Our results show that English fine-tuned models perform slightly better than models fine-tuned on other Nigerian languages, which could be explained by the lexical and structural closeness between Nigerian Pidgin and English.The best results were reported on the monolingual Nigerian Pidgin data.The model pre-trained on English and fine-tuned on Nigerian Pidgin was submitted to Task A Track 4 of the AfriSenti-SemEval Shared Task 12, and scored 25 out of 32 in the ranking.
Curating Datasets for Better Performance with Example Training Dynamics

The landscape of NLP research is dominated by large-scale models training on colossal datasets, relying on data quantity rather than quality.As an alternative to this landscape, we propose a method for weighing the relative importance of examples in a dataset based on their Example Training dynamics (ETD; Swayamdipta et al., 2020), a set of metrics computed during training.We propose a new way of computing the ETD of a dataset, and show that they can be used to improve performance in both in-distribution and out-of-distribution testing.We show that ETD can be transferable, i.e., they can be computed once and used for training different models, effectively reducing their computation cost.Finally, we suggest an active learning approach for computing ETD during training rather than as a preprocessing step-an approach that is not as effective, but dramatically reduces the extra computational costs.
Acquiring Frame Element Knowledge with Deep Metric Learning for Semantic Frame Induction

The semantic frame induction tasks are defined as a clustering of words into the frames that they evoke, and a clustering of their arguments according to the frame element roles that they should fill.In this paper, we address the latter task of argument clustering, which aims to acquire frame element knowledge, and propose a method that applies deep metric learning.In this method, a pre-trained language model is fine-tuned to be suitable for distinguishing frame element roles through the use of frame-annotated data, and argument clustering is performed with embeddings obtained from the fine-tuned model.Experimental results on FrameNet demonstrate that our method achieves substantially better performance than existing methods.
Structured Mean-Field Variational Inference for Higher-Order Span-Based Semantic Role

In this work, we enhance higher-order graphbased approaches for span-based semantic role labeling (SRL) by means of structured modeling.To decrease the complexity of higher-order modeling, we decompose the edge from predicate word to argument span into three different edges, predicate-to-head (P2H), predicateto-tail (P2T), and head-to-tail (H2T), where head/tail means the first/last word of the semantic argument span.As such, we use a CRF-based higher-order dependency parser and leverage Mean-Field Variational Inference (MFVI) for higher-order inference.Moreover, since semantic arguments of predicates are often constituents within a constituency parse tree, we can leverage such nice structural property by defining a TreeCRF distribution over all H2T edges, using the idea of partial marginalization to define structural training loss.We further leverage structured MFVI to enhance inference.We experiment on spanbased SRL benchmarks, showing the effectiveness of both higher-order and structured modeling and the combination thereof.In addition, we show superior performance of structured MFVI against vanilla MFVI.Our code is publicly available at https://github.com/VPeterV/Structured-MFVI.
The Turing Quest: Can Transformers Make Good NPCs?

In this paper, we investigate the potential of using large pre-trained language models to generate non-playable character (NPC) scripts in video games.We introduce a novel pipeline that automatically constructs believable NPC scripts for various game genres and specifications using Transformer-based models.Moreover, we develop a self-diagnosis method, inspired by prior research, that is tailored to essential NPC characteristics such as coherence, believability, and variety in dialogue.To evaluate our approach, we propose a new benchmark, The Turing Quest, which demonstrates that our pipeline, when applied to GPT-3, generates NPC scripts across diverse game genres and contexts that can successfully deceive judges into believing they were written by humans.Our findings hold significant implications for the gaming industry and its global community, as the current reliance on manually-curated scripts is resource-intensive and can limit the immersiveness and enjoyment of players.
Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty

This study investigates learning with disagreement in NLP tasks and evaluates its performance on four datasets.The results suggest that the model performs best on the experimental dataset and faces challenges in minority languages.Furthermore, the analysis indicates that annotator demographics play a significant role in the interpretation of such tasks.This study suggests the need for greater consideration of demographic differences in annotators and more comprehensive evaluation metrics for NLP models.
DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation

Self-training (ST) has prospered again in language understanding by augmenting the finetuning of big pre-trained models when labeled data is insufficient.However, it remains challenging to incorporate ST into attributecontrollable language generation.Augmented only by self-generated pseudo text, generation models over-exploit the previously learned text space and fail to explore a larger one, suffering from a restricted generalization boundary and limited controllability.In this work, we propose DuNST, a novel ST framework to tackle these problems.DuNST jointly models text generation and classification as a dual process and further perturbs and escapes from the collapsed space by adding two kinds of flexible noise.In this way, our model could construct and utilize both pseudo text generated from given labels and pseudo labels predicted from available unlabeled text, which are gradually refined during the ST phase.Theoretically, we show that DuNST can be viewed as enhancing the exploration of the potentially larger real text space while maintaining exploitation, guaranteeing improved performance.Experiments on three controllable generation tasks show that DuNST significantly boosts control accuracy with comparable generation fluency and diversity against several strong baselines.
Neighboring Words Affect Human Interpretation of Saliency Explanations

Word-level saliency explanations (“heat maps over words”) are often used to communicate feature-attribution in text-based models. Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores. We conduct a user study to investigate how the marking of a word’s *neighboring words* affect the explainee’s perception of the word’s importance in the context of a saliency explanation.We find that neighboring words have significant effects on the word’s importance rating.Concretely, we identify that the influence changes based on neighboring direction (left vs. right) and a-priori linguistic and computational measures of phrases and collocations (vs. unrelated neighboring words).Our results question whether text-based saliency explanations should be continued to be communicated at word level, and inform future research on alternative saliency explanation methods.
Biomedical Language Models are Robust to Sub-optimal Tokenization

As opposed to general English, many concepts in biomedical terminology have been designed in recent history by biomedical professionals with the goal of being precise and concise.This is often achieved by concatenating meaningful biomedical morphemes to create new semantic units.Nevertheless, most modern biomedical language models (LMs) are pretrained using standard domain-specific tokenizers derived from large scale biomedical corpus statistics without explicitly leveraging the agglutinating nature of biomedical language.In this work, we first find that standard opendomain and biomedical tokenizers are largely unable to segment biomedical terms into meaningful components.Therefore, we hypothesize that using a tokenizer which segments biomedical terminology more accurately would enable biomedical LMs to improve their performance on downstream biomedical NLP tasks, especially ones which involve biomedical terms directly such as named entity recognition (NER) and entity linking.Surprisingly, we find that pre-training a biomedical LM using a more accurate biomedical tokenizer does not improve the entity representation quality of a language model as measured by several intrinsic and extrinsic measures such as masked language modeling prediction (MLM) accuracy as well as NER and entity linking performance.These quantitative findings, along with a case study which explores entity representation quality more directly, suggest that the biomedical pretraining process is quite robust to instances of sub-optimal tokenization. 1
Decoder Tuning: Efficient Language Understanding as Decoding

With the evergrowing sizes of pre-trained models (PTMs), it has been an emerging practice to only provide the inference APIs for users, namely model-as-a-service (MaaS) setting.To adapt PTMs with model parameters frozen, most current approaches focus on the input side, seeking for powerful prompts to stimulate models for correct answers.However, we argue that input-side adaptation could be arduous due to the lack of gradient signals and they usually require thousands of API queries, resulting in high computation and time costs.In light of this, we present Decoder Tuning (DecT), which in contrast optimizes task-specific decoder networks on the output side.Specifically, DecT first extracts prompt-stimulated output scores for initial predictions.On top of that, we train an additional decoder network on the output representations to incorporate posterior data knowledge.By gradientbased optimization, DecT can be trained within several seconds and requires only one PTM query per sample.Empirically, we conduct extensive natural language understanding experiments and show that DecT significantly outperforms state-of-the-art algorithms with a 200× speed-up.Our codes are available at https://github.com/thunlp/DecT.
Annotating and Disambiguating the Discourse Usage of the Enclitic dA in Turkish

The Turkish particle dA is a focus-associated enclitic, and it can act as a discourse connective conveying multiple senses, like additive, contrastive, causal etc.Like many other linguistic expressions, it is subject to usage ambiguity and creates a challenge in natural language automatization tasks.For the first time, we annotate the discourse and non-discourse connnective occurrences of dA in Turkish with the PDTB principles.Using a minimal set of linguistic features, we develop binary classifiers to distinguish its discourse connective usage from its other usages.We show that despite its ability to cliticize to any syntactic type, variable position in the sentence and having a wide argument span, its discourse/non-discourse connective usage can be annotated reliably and its discourse usage can be disambiguated by exploiting local cues.
Long-Tailed Question Answering in an Open World

Real-world data often have an open long-tailed distribution, and building a unified QA model supporting various tasks is vital for practical QA applications.However, it is non-trivial to extend previous QA approaches since they either require access to seen tasks of adequate samples or do not explicitly model samples from unseen tasks.In this paper, we define Open Long-Tailed QA (OLTQA) as learning from long-tailed distributed data and optimizing performance over seen and unseen QA tasks.We propose an OLTQA model that encourages knowledge sharing between head, tail and unseen tasks, and explicitly mines knowledge from a large pre-trained language model (LM).Specifically, we organize our model through a pool of fine-grained components and dynamically combine these components for an input to facilitate knowledge sharing.A retrieve-then-rerank frame is further introduced to select in-context examples, which guild the LM to generate text that express knowledge for QA tasks.Moreover, a twostage training approach is introduced to pretrain the framework by knowledge distillation (KD) from the LM and then jointly train the frame and a QA model through an adaptive mutual KD method.On a large-scale OLTQA dataset we curate from 43 existing QA datasets, our model consistently outperforms the stateof-the-art.We release the code and data at
Long to reign over us: A Case Study of Machine Translation and a New Monarch

Novel terminology and changes in terminology are often a challenge for machine translation systems.The passing of Queen Elizabeth II and the accession of King Charles III provide a striking example of translation shift in the real world, particularly in translation contexts that have ambiguity.Examining translation between French and English, we present a focused case-study of translations about King Charles III as produced both by publicly-available MT systems and by a neural machine translation system trained specifically on Canadian parliamentary text.We find that even in cases where human translators would have adequate context to disambiguate terms from the source language, machine translation systems do not always produce the expected output.Where we are able to analyze the training data, we note that this may represent artifacts in the data, raising important questions about machine translation updates in light of real world events.
Data Sampling and (In)stability in Machine Translation Evaluation

We analyze the different data sampling approaches used in selecting data for human evaluation and ranking of machine translation systems at the highly influential Conference on Machine Translation (WMT).By using automatic evaluation metrics, we are able to focus on the impact of the data sampling procedure as separate from questions about human annotator consistency.We provide evidence that the latest data sampling approach used at WMT skews the annotated data toward shorter documents, not necessarily representative of the full test set.Lastly, we examine a new data sampling method that uses the available labour budget to sample data in a more representative manner, with the goals of improving representation of various document lengths in the sample and producing more stable rankings of system translation quality.
Recognizing Learner Handwriting Retaining Orthographic Errors for Enabling Fine-Grained Error Feedback

This paper addresses the problem of providing automatic feedback on orthographic errors in handwritten text.Despite the availability of automatic error detection systems, the practical problem of digitizing the handwriting remains.Current handwriting recognition (HWR) systems produce highly accurate transcriptions but normalize away the very errors that are essential for providing useful feedback, e.g.orthographic errors.Our contribution is twofold: First, we create a comprehensive dataset of handwritten text with transcripts retaining orthographic errors by transcribing 1,350 pages from the German learner dataset FD-LEX.Second, we train a simple HWR system on our dataset, allowing it to transcribe words with orthographic errors.Thereby, we evaluate the effect of different dictionaries on recognition output, highlighting the importance of addressing spelling errors in these dictionaries.