Reviewerly/exHarmony huggingface dataset:
Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation

Word sense disambiguation (WSD), which aims to determine an appropriate sense for a target word given its context, is crucial for natural language understanding.Existing supervised methods treat WSD as a classification task and have achieved remarkable performance.However, they ignore uncertainty estimation (UE) in the real-world setting, where the data is always noisy and out of distribution.This paper extensively studies UE on the benchmark designed for WSD.Specifically, we first compare four uncertainty scores for a state-of-the-art WSD model and verify that the conventional predictive probabilities obtained at the final layer of the model are inadequate to quantify uncertainty.Then, we examine the capability of capturing data and model uncertainties by the model with the selected UE score on well-designed test scenarios and discover that the model adequately reflects data uncertainty but underestimates model uncertainty.Furthermore, we explore numerous lexical properties that intrinsically affect data uncertainty and provide a detailed analysis of four critical aspects: the syntactic category, morphology, sense granularity, and semantic relations.
UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

Vision-language tasks, such as VQA, SNLI-VE, and VCR are challenging because they require the model’s reasoning ability to understand the semantics of the visual world and natural language. Supervised methods working for vision-language tasks have been well-studied. However, solving these tasks in a zero-shot setting is less explored. Since Contrastive Language-Image Pre-training (CLIP) has shown remarkable zero-shot performance on image-text matching, previous works utilized its strong zero-shot ability by converting vision-language tasks into an image-text matching problem, and they mainly consider global-level matching (e.g., the whole image or sentence). However, we find visual and textual fine-grained information, e.g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding. Inspired by this, we propose a unified framework to take advantage of the fine-grained information for zero-shot vision-language learning, covering multiple tasks such as VQA, SNLI-VE, and VCR. Our experiments show that our framework outperforms former zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR. Furthermore, our ablation studies confirm the effectiveness and generalizability of our proposed method.
Reducing Sensitivity on Speaker Names for Text Generation from Dialogues

Changing speaker names consistently throughout a dialogue should not affect its meaning and corresponding outputs for text generation from dialogues.However, pre-trained language models, serving as the backbone for dialogue-processing tasks, have shown to be sensitive to nuances.This may result in unfairness in real-world applications.No comprehensive analysis of this problem has been done in the past.In this work, we propose to quantitatively measure a model's sensitivity on speaker names, and comprehensively evaluate a number of known methods for reducing speaker name sensitivity, including a novel approach of our own.Extensive experiments on multiple datasets provide a benchmark for this problem and show the favorable performance of our approach in sensitivity reduction and quality of generation.
A Natural Bias for Language Generation Models

After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens.Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus.The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity?Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model's final linear layer with the log-unigram distribution.We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.
Ensemble Transfer Learning for Multilingual Coreference Resolution

Entity coreference resolution is an important research problem with many applications, including information extraction and question answering.Coreference resolution for English has been studied extensively.However, there is relatively little work for other languages.A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data.To overcome this challenge, we design a simple but effective ensemble-based framework that combines various transfer learning (TL) techniques.We first train several models using different TL methods.Then, during inference, we compute the unweighted average scores of the models' predictions to extract the final set of predicted clusters.Furthermore, we also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.Leveraging the idea that the coreferential links naturally exist between anchor texts pointing to the same article, our method builds a sizeable distantly-supervised dataset for the target language that consists of tens of thousands of documents.We can pre-train a model on the pseudo-labeled dataset before finetuning it on the final target dataset.Experimental results on two benchmark datasets, OntoNotes and Se-mEval, confirm the effectiveness of our methods.Our best ensembles consistently outperform the baseline approach of simple training by up to 7.68% in the F1 score.These ensembles also achieve new state-of-the-art results for three languages: Arabic, Dutch, and Spanish 1 .
What Makes a Good Counter-Stereotype? Evaluating Strategies for Automated Responses to Stereotypical Text

Content Warning: This paper presents examples of societal stereotypes that may be offensive or upsetting.When harmful social stereotypes are expressed on a public platform, they must be addressed in a way that educates and informs both the original poster and other readers, without causing offence or perpetuating new stereotypes.In this paper, we synthesize findings from psychology and computer science to propose a set of potential counter-stereotype strategies.We then automatically generate such counter-stereotypes using ChatGPT, and analyze their correctness and expected effectiveness at reducing stereotypical associations.We identify the strategies of denouncing stereotypes, warning of consequences, and using an empathetic tone as three promising strategies to be further tested.
Robustness-Aware Word Embedding Improves Certified Robustness to Adversarial Word Substitutions

Natural Language Processing (NLP) models have gained great success on clean texts, but they are known to be vulnerable to adversarial examples typically crafted by synonym substitutions.In this paper, we target to solve this problem and find that word embedding is important to the certified robustness of NLP models.Given the findings, we propose the Embedding Interval Bound Constraint (EIBC) triplet loss to train robustness-aware word embeddings for better certified robustness.We optimize the EIBC triplet loss to reduce distances between synonyms in the embedding space, which is theoretically proven to make the verification boundary tighter.Meanwhile, we enlarge distances among non-synonyms, maintaining the semantic representation of word embeddings.Our method is conceptually simple and componentized.It can be easily combined with IBP training and improves the certified robust accuracy from 76.73% to 84.78% on the IMDB dataset.Experiments demonstrate that our method outperforms various state-of-the-art certified defense baselines and generalizes well to unseen substitutions.The code is available at https://github.
Task-adaptive Label Dependency Transfer for Few-shot Named Entity Recognition

Named Entity Recognition (NER), as a crucial subtask in natural language processing (NLP), suffers from limited labeled samples (a.k.a.few-shot).Meta-learning methods are widely used for few-shot NER, but these existing methods overlook the importance of label dependency for NER, resulting in suboptimal performance.However, applying meta-learning methods to label dependency learning faces a special challenge, that is, due to the discrepancy of label sets in different domains, the label dependencies can not be transferred across domains.In this paper, we propose the Taskadaptive Label Dependency Transfer (TLDT) method to make label dependency transferable and effectively adapt to new tasks by a few samples.TLDT improves the existing optimizationbased meta-learning methods by learning general initialization and individual parameter update rule for label dependency.Extensive experiments show that TLDT achieves significant improvement over the state-of-the-art methods.
CPIC at SemEval-2023 Task 7: GPT2-Based Model for Multi-evidence Natural Language Inference for Clinical Trial Data

This paper describes our system submitted for SemEval Task 7, Multi-Evidence Natural Language Inference for Clinical Trial Data.The task consists of 2 subtasks.Subtask 1 is to determine the relationships between clinical trial data (CTR) and statements.Subtask 2 is to output a set of supporting facts extracted from the premises with the input of CTR premises and statements.Through experiments, we found that our GPT2-based pre-trained models can obtain good results in Subtask 2. Therefore, we use the GPT2-based pre-trained model to fine-tune Subtask 2. We transform the evidence retrieval task into a binary class task by combining premises and statements as input, and the output is whether the premises and statements match.We obtain a top-5 score in the evaluation phase of Subtask 2.
Tucker Decomposition with Frequency Attention for Temporal Knowledge Graph Completion

Temporal Knowledge Graph Completion aims to complete missing entities or relations under temporal constraints.Previous tensor decomposition-based models for TKGC only independently consider the combination of one single relation with one single timestamp, ignoring the global nature of the embedding.We propose a Frequency Attention (FA) model to capture the global temporal dependencies between one relation and the entire timestamp.Specifically, we use Discrete Cosine Transform (DCT) to capture the frequency of the timestamp embedding and further compute the frequency attention weight to scale embedding.Meanwhile, the previous temporal tucker decomposition method uses a simple norm regularization to constrain the core tensor, which limits the optimization performance.Thus, we propose Orthogonal Regularization (OR) variants for the core tensor, which can limit the non-superdiagonal elements of the 3-rd core tensor.Experiments on three standard TKGC datasets demonstrate that our method outperforms the state-of-the-art results on several metrics.The results suggest that the direct-current component is not the best feature for TKG representation learning.Additional analysis shows the effectiveness of our FA and OR models, even with smaller embedding dimensions.
Morphological Inflection with Phonological Features

Recent years have brought great advances into solving morphological tasks, mostly due to powerful neural models applied to various tasks as (re)inflection and analysis.Yet, such morphological tasks cannot be considered solved, especially when little training data is available or when generalizing to previously unseen lemmas.This work explores effects on performance obtained through various ways in which morphological models get access to subcharacter phonological features that are often the targets of morphological processes.We design two methods to achieve this goal: one that leaves models as is but manipulates the data to include features instead of characters, and another that manipulates models to take phonological features into account when building representations for phonemes.We elicit phonemic data from standard graphemic data using language-specific grammars for languages with shallow grapheme-to-phoneme mapping, and we experiment with two reinflection models over eight languages.Our results show that our methods yield comparable results to the grapheme-based baseline overall, with minor improvements in some of the languages.All in all, we conclude that patterns in character distributions are likely to allow models to infer the underlying phonological characteristics, even when phonemes are not explicitly represented.
Unlearning Bias in Language Models by Partitioning Gradients

Recent research has shown that large-scale pretrained language models, specifically transformers, tend to exhibit issues relating to racism, sexism, religion bias, and toxicity in general.Unfortunately, these pretrained language models are used almost universally in downstream tasks, and natural language processing is often applied to make real-world predictions.Thus, debiasing these language models as early in development as possible is increasingly crucial for preventing unintentional harms caused by natural language systems.To this end, we propose a new technique called partitioned contrastive gradient unlearning (PCGU), a gray-box method for debiasing pretrained masked language models.PCGU aims to optimize only the weights that contribute most to a specific domain of bias, doing so by computing a first-order approximation based on the gradients of contrastive sentence pairs.Our experiments show that PCGU is both low-cost and seems particularly effective at pinpointing the sources of implicit social bias in large pretrained transformers.Although we train using PCGU in the genderprofession domain only, we find that doing so can also partially mitigate bias across other domains.All code for our implementation and experiments can be found at https: //github.com/CharlesYu2000/PCGU-UnlearningBias.
Measuring the Instability of Fine-Tuning

Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets.Many previous studies have investigated this instability and proposed methods to mitigate it.However, most studies only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability.In this paper, we analyze SD and six other measures quantifying instability at different levels of granularity.Moreover, we propose a systematic framework to evaluate the validity of these measures.Finally, we analyze the consistency and difference between different measures by reassessing existing instability mitigation methods.We hope our results will inform the development of better measurements of fine-tuning instability. 1
Native Language Prediction from Gaze: a Reproducibility Study

Numerous studies found that the linguistic properties of a person's native language affect the cognitive processing of other languages.However, only one study has shown that it was possible to identify the native language based on eye-tracking records of natural L2 reading using machine learning.A new corpus allows us to replicate these results on a more interrelated and larger set of native languages.Our results show that comparable classification performance is maintained despite using less data.However, analysis shows that the correlation between L2 eye movements and native language similarity may be more complex than the original study found.
Peer-Label Assisted Hierarchical Text Classification

Hierarchical text classification (HTC) is a challenging task, in which the labels of texts can be organized into a category hierarchy.To deal with the HTC problem, many existing works focus on utilizing the parent-child relationships that are explicitly shown in the hierarchy.However, texts with a category hierarchy also have some latent relevancy among labels in the same level of the hierarchy.We refer to these labels as peer labels, from which the peer effects are originally utilized in our work to improve the classification performance.To fully explore the peer-label relationship, we develop a PeerHTC method.This method innovatively measures the latent relevancy of peer labels through several metrics and then encodes the relevancy with a Graph Convolutional Neural Network.We also propose a sample importance learning method to ameliorate the side effects raised by modelling the peer label relevancy.Our experiments on several standard datasets demonstrate the evidence of peer labels and the superiority of PeerHTC over other state-of-the-art HTC methods in terms of classification accuracy.
It is a Bird Therefore it is a Robin: On BERT’s Internal Consistency Between Hypernym Knowledge and Logical Words

The lexical knowledge of NLP systems should be tested (i) for their internal consistency (avoiding groundedness issues) and (ii) both for content words and logical words.In this paper we propose a new method to test the understanding of the hypernymy relationship by measuring its antisymmetry according to the models.Previous studies often rely only on the direct question (e.g., A robin is a ...), where we argue a correct answer could only rely on collocational cues, rather than hierarchical cues.We show how to control for this, and how it is important.We develop a method to ask similar questions about logical words that encode an entailment-like relation (e.g., because or therefore).Our results show important weaknesses of BERT-like models on these semantic tasks.
Improving Diachronic Word Sense Induction with a Nonparametric Bayesian method

Diachronic Word Sense Induction (DWSI) is the task of inducing the temporal representations of a word meaning from the context, as a set of senses and their prevalence over time.We introduce two new models for DWSI, based on topic modelling techniques: one is based on Hierarchical Dirichlet Processes (HDP), a nonparametric model; the other is based on the Dynamic Embedded Topic Model (DETM), a recent dynamic neural model.We evaluate these models against two state of the art DWSI models, using a time-stamped labelled dataset from the biomedical domain.We demonstrate that the two proposed models perform better than the state of the art.In particular, the HDP-based model drastically outperforms all the other models, including the dynamic neural models. 1
Joint End-to-end Semantic Proto-role Labeling

Semantic proto-role labeling (SPRL) assigns properties to arguments based on a series of binary labels.While multiple studies have evaluated various approaches to SPRL, it has only been studied in-depth as a standalone task using gold predicate/argument pairs.How do SPRL systems perform as part of an information extraction pipeline?We model SPRL jointly with predicate-argument extraction using a deep transformer model.We find that proto-role labeling is surprisingly robust in this setting, with only a small decrease when using predicted arguments.We include a detailed analysis of each component of the joint system, and an error analysis to understand correlations in errors between system stages.Finally, we study the effects of annotation errors on SPRL.
Guiding Computational Stance Detection with Expanded Stance Triangle Framework

Stance detection determines whether the author of a piece of text is in favor of, against, or neutral towards a specified target, and can be used to gain valuable insights into social media.The ubiquitous indirect referral of targets makes this task challenging, as it requires computational solutions to model semantic features and infer the corresponding implications from a literal statement.Moreover, the limited amount of available training data leads to subpar performance in out-of-domain and cross-target scenarios, as data-driven approaches are prone to rely on superficial and domain-specific features.In this work, we decompose the stance detection task from a linguistic perspective, and investigate key components and inference paths in this task.The stance triangle is a generic linguistic framework previously proposed to describe the fundamental ways people express their stance.We further expand it by characterizing the relationship between explicit and implicit objects.We then use the framework to extend one single training corpus with additional annotation.Experimental results show that strategically-enriched data can significantly improve the performance on out-of-domain and cross-target evaluation.
GunadarmaXBRIN at SemEval-2023 Task 12: Utilization of SVM and AfriBERTa for Monolingual, Multilingual, and Zero-shot Sentiment Analysis in African Languages

This paper describes our participation in Task 12: AfriSenti-SemEval 2023, i.e., Track 12 of subtask A, Track 16 of subtask B, and Track 18 of subtask C. To deal with these three tracks, we utilize Support Vector Machine (SVM) + One vs Rest, SVM + One vs Rest with SMOTE, and AfriBERTa-large models.In particular, our SVM + One vs Rest with SMOTE model could obtain the highest weighted F1-Score for Tracks 16 and 18 in the evaluation phase, that is, 65.14% and 33.49%, respectively.Meanwhile, our SVM + One vs Rest model could perform better than other models for Track 12 in the evaluation phase.
BLOCSUM: Block Scope-based Source Code Summarization via Shared Block Representation

Code summarization, which aims to automatically generate natural language descriptions of the source code, has become an essential task in software development for better program understanding.Abstract Syntax Tree (AST), which represents the syntax structure of the source code, is helpful when utilized together with the sequence of code tokens to improve the quality of code summaries.Recent works on code summarization attempted to capture the sequential and structural information of the source code, but they considered less the property that source code consists of multiple code blocks.In this paper, we propose BLOCSUM, BLOck scope-based source Code SUMmarization via shared block representation that utilizes block-scope information by representing various structures of the code block.We propose a shared block position embedding to effectively represent the structure of code blocks and merge both code and AST.Furthermore, we develop variant ASTs to learn rich information such as block and global dependencies of the source code.To prove our approach, we perform experiments on two real-world datasets, the Java dataset and the Python dataset.We demonstrate the effectiveness of BLOCSUM through various experiments, including ablation studies and a human evaluation.