Reviewerly/exHarmony huggingface dataset:
General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation

Training keyphrase generation (KPG) models require a large amount of annotated data, which can be prohibitively expensive and often limited to specific domains.In this study, we first demonstrate that large distribution shifts among different domains severely hinder the transferability of KPG models.We then propose a three-stage pipeline, which gradually guides KPG models' learning focus from general syntactical features to domain-related semantics, in a data-efficient manner.With domain-general phrase pre-training, we pre-train Sequence-to-Sequence models with generic phrase annotations that are widely available on the web, which enables the models to generate phrases in a wide range of domains.The resulting model is then applied in the Transfer Labeling stage to produce domain-specific pseudo keyphrases, which help adapt models to a new domain.Finally, we fine-tune the model with limited data with true labels to fully adapt it to the target domain.Our experiment results show that the proposed process can produce good quality keyphrases in new domains and achieve consistent improvements after adaptation with limited in-domain annotated data 12 .
This prompt is measuring &lt;mask&gt;: evaluating bias evaluation in language models

Bias research in NLP seeks to analyse models for social biases, thus helping NLP practitioners uncover, measure, and mitigate social harms.We analyse the body of work that uses prompts and templates to assess bias in language models.We draw on a measurement modelling framework to create a taxonomy of attributes that capture what a bias test aims to measure and how that measurement is carried out.By applying this taxonomy to 90 bias tests, we illustrate qualitatively and quantitatively that core aspects of bias test conceptualisations and operationalisations are frequently unstated or ambiguous, carry implicit assumptions, or be mismatched.Our analysis illuminates the scope of possible bias types the field is able to measure, and reveals types that are as yet under-researched.We offer guidance to enable the community to explore a wider section of the possible bias space, and to better close the gap between desired outcomes and experimental design, both for bias and for evaluating language models more broadly.
Robust Representation Learning with Reliable Pseudo-labels Generation via Self-Adaptive Optimal Transport for Short Text Clustering

Short text clustering is challenging since it takes imbalanced and noisy data as inputs.Existing approaches cannot solve this problem well, since (1) they are prone to obtain degenerate solutions especially on heavy imbalanced datasets, and (2) they are vulnerable to noises.To tackle the above issues, we propose a Robust Short Text Clustering (RSTC) model to improve robustness against imbalanced and noisy data.RSTC includes two modules, i.e., pseudo-label generation module and robust representation learning module.The former generates pseudo-labels to provide supervision for the later, which contributes to more robust representations and correctly separated clusters.To provide robustness against the imbalance in data, we propose self-adaptive optimal transport in the pseudo-label generation module.To improve robustness against the noise in data, we further introduce both class-wise and instance-wise contrastive learning in the robust representation learning module.Our empirical studies on eight short text clustering datasets demonstrate that RSTC significantly outperforms the state-of-the-art models.
Holographic CCG Parsing

We propose a method for formulating CCG as a recursive composition in a continuous vector space.Recent CCG supertagging and parsing models generally demonstrate high performance, yet rely on black-box neural architectures to implicitly model phrase structure dependencies.Instead, we leverage the method of holographic embeddings (Nickel et al., 2016) as a compositional operator to explicitly model the dependencies between words and phrase structures in the embedding space.Experimental results revealed that holographic composition effectively improves the supertagging accuracy to achieve state-of-the-art parsing performance when using a C&C parser.The proposed span-based parsing algorithm using holographic composition achieves performance comparable to state-of-the-art neural parsing with Transformers.Furthermore, our model can semantically and syntactically infill text at the phrase level due to the decomposability of holographic composition.
OD-RTE: A One-Stage Object Detection Framework for Relational Triple Extraction

The Relational Triple Extraction (RTE) task is a fundamental and essential information extraction task.Recently, the table-filling RTE methods have received lots of attention.Despite their success, they suffer from some inherent problems such as underutilizing regional information of triple.In this work, we treat the RTE task based on table-filling method as an Object Detection task and propose a one-stage Object Detection framework for Relational Triple Extraction (OD-RTE).In this framework, the vertices-based bounding box detection, coupled with auxiliary global relational triple region detection, ensuring that regional information of triple could be fully utilized.Besides, our proposed decoding scheme could extract all types of triples.In addition, the negative sampling strategy of relations in the training stage improves the training efficiency while alleviating the imbalance of positive and negative relations.The experimental results show that 1) OD-RTE achieves the state-of-the-art performance on two widely used datasets (i.e., NYT and WebNLG).2) Compared with the best performing table-filling method, OD-RTE achieves faster training and inference speed with lower GPU memory usage.To facilitate future research in this area, the codes are publicly available at https://github.com/NingJinzhong/ODRTE.
mCPT at SemEval-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection

This paper presents the winning system for the zero-shot Spanish framing detection task, which also achieves competitive places in eight additional languages.The challenge of the framing detection task lies in identifying a set of 14 frames when only a few or zero samples are available, i.e., a multilingual multi-label few-or zero-shot setting.Our developed solution employs a pre-training procedure based on multilingual Transformers using a label-aware contrastive loss function.In addition to describing the system, we perform an embedding space analysis and ablation study to demonstrate how our pre-training procedure supports framing detection to advance computational framing analysis.
Large-Scale Correlation Analysis of Automated Metrics for Topic Models

Automated coherence metrics constitute an important and popular way to evaluate topic models.Previous works present a mixed picture of their presumed correlation with human judgement.In this paper, we conduct a large-scale correlation analysis of coherence metrics.We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated.Moreover, we extend the analysis to measure topical differences between corpora.Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes.Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.
Fact-Checking Complex Claims with Program-Guided Reasoning

Fact-checking real-world claims often requires collecting multiple pieces of evidence and applying complex multi-step reasoning.In this paper, we present Program-Guided Fact-Checking (PROGRAMFC), a novel factchecking model that decomposes complex claims into simpler sub-tasks that can be solved using a shared library of specialized functions.We first leverage the in-context learning ability of large language models to generate reasoning programs to guide the verification process.Afterward, we execute the program by delegating each sub-task to the corresponding sub-task handler.This process makes our model both explanatory and data-efficient, providing clear explanations of its reasoning process and requiring minimal training data.We evaluate PRO-GRAMFC on two challenging fact-checking datasets and show that it outperforms seven fact-checking baselines across different settings of evidence availability, with explicit output programs that benefit human debugging. 1
MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement

This paper describes the participation of the research laboratory MIND, at the University of Milano-Bicocca, in the SemEval 2023 task related to Learning With Disagreements (Le-Wi-Di).The main goal is to identify the level of agreement/disagreement from a collection of textual datasets with different characteristics in terms of style, language, and task.The proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold label.
Multi-modal Sarcasm Generation: Dataset and Solution

As an interesting and challenging task, sarcasm generation has attracted widespread attention.Although very recent studies have made promising progress, none of them considers generating a sarcastic description for a given image -as what people usually do on Twitter.In this paper, we present a Multi-modal Sarcasm Generation (MSG) task: Given an image with hashtags that provide the sarcastic target, MSG aims to generate sarcastic descriptions like humans.Compared with textual sarcasm generation, MSG is more challenging as it is difficult to accurately capture the key information from images, hashtags, and OCR tokens and exploit multi-modal incongruity to generate sarcastic descriptions.To support the research on MSG, we develop MuSG, a new dataset with 5000 images and related Twitter text.We also propose a multi-modal Transformer-based method as a solution to this MSG task.The input features are embedded in the common space and passed through the multi-modal Transformer layers to generate the sarcastic descriptions by the auto-regressive paradigm.Both automatic and manual evaluations demonstrate the superiority of our method.The dataset and code will be available at github.com/lukakupolida/MSG.
Billie-Newman at SemEval-2023 Task 5: Clickbait Classification and Question Answering with Pre-Trained Language Models, Named Entity Recognition and Rule-Based Approaches

In this paper, we describe the implementations of our systems for the SemEval-2023 Task 5 ‘Clickbait Spoiling’, which involves the classification of clickbait posts in sub-task 1 and the spoiler generation and question answering of clickbait posts in sub-task 2, ultimately achieving a balanced accuracy of 0.593 and a BLEU score of 0.322 on the test datasets in sub-task 1 and sub-task 2 respectively.For this, we propose the usage of RoBERTa transformer models and modify them for each specific downstream task.In sub-task 1, we use the pre-trained RoBERTa model and use it in conjunction with NER, a spoiler-title ratio, a regex check for enumerations and lists and the use of input reformulation.In sub-task 2, we propose the usage of the RoBERTa-SQuAD2.0 model for extractive question answering in combination with a contextual rule-based approach for multi-type spoilers in order to generate spoiler answers.
From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models

Warning: content in this paper may be upsetting or offensive to some readers.Dogwhistles are coded expressions that simultaneously convey one meaning to a broad audience and a second one, often hateful or provocative, to a narrow in-group; they are deployed to evade both political repercussions and algorithmic content moderation.For example, in the sentence "we need to end the cosmopolitan experiment," the word "cosmopolitan" likely means "worldly" to many, but secretly means "Jewish" to a select few.We present the first large-scale computational investigation of dogwhistles.We develop a typology of dogwhistles, curate the largest-to-date glossary of over 300 dogwhistles with rich contextual information and examples, and analyze their usage in historical U.S. politicians' speeches.We then assess whether a large language model (GPT-3) can identify dogwhistles and their meanings, and find that GPT-3's performance varies widely across types of dogwhistles and targeted groups.Finally, we show that harmful content containing dogwhistles avoids toxicity detection, highlighting online risks of such coded language.This work sheds light on the theoretical and applied importance of dogwhistles in both NLP and computational social science, and provides resources for future research in modeling dogwhistles and mitigating their online harms.
Petals: Collaborative Inference and Fine-tuning of Large Models

Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters.With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale.Still, using these models requires high-end hardware unavailable to many researchers.In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs.However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits.In this work, we propose PETALS -a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties.We demonstrate that this strategy outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second, which is enough for many interactive LLM applications.Unlike most inference APIs, PETALS also natively exposes hidden states of served models, allowing to train and share custom model extensions based on efficient fine-tuning methods.The system, its source code, and documentation are available at https://petals.ml.
Solving Cosine Similarity Underestimation between High Frequency Words by ℓ2 Norm Discounting Norm Discounting

Cosine similarity between two words, computed using their contextualised token embeddings obtained from masked language models (MLMs) such as BERT has shown to underestimate the actual similarity between those words (Zhou et al., 2022).This similarity underestimation problem is particularly severe for highly frequent words.Although this problem has been noted in prior work, no solution has been proposed thus far.We observe that the ℓ 2 norm of contextualised embeddings of a word correlates with its log-frequency in the pretraining corpus.Consequently, the larger ℓ 2 norms associated with the highly frequent words reduce the cosine similarity values measured between them, thus underestimating the similarity scores.To solve this issue, we propose a method to discount the ℓ 2 norm of a contextualised word embedding by the frequency of that word in a corpus when measuring the cosine similarities between words.We show that the so called stop words behave differently from the rest of the words, which require special consideration during their discounting process.Experimental results on a contextualised word similarity dataset show that our proposed discounting method accurately solves the similarity underestimation problem.
Answering Unanswered Questions through Semantic Reformulations in Spoken QA

Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems.Users ask questions via spontaneous speech which can contain disfluencies, errors, and informal syntax or phrasing.This is a major challenge in QA, causing unanswered questions or irrelevant answers, and leading to bad user experiences.We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity.We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering.Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%).Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.
Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

Knowledge distillation (KD) is a promising technique for model compression in neural machine translation.However, where the knowledge hides in KD is still not clear, which may hinder the development of KD.In this work, we first unravel this mystery from an empirical perspective and show that the knowledge comes from the top-1 predictions of teachers, which also helps us build a potential connection between word-and sequence-level KD.Further, we point out two inherent issues in vanilla word-level KD based on this finding.Firstly, the current objective of KD spreads its focus to whole distributions to learn the knowledge, yet lacks special treatment on the most crucial top-1 information.Secondly, the knowledge is largely covered by the golden information due to the fact that most top-1 predictions of teachers overlap with ground-truth tokens, which further restricts the potential of KD.To address these issues, we propose a novel method named Top-1 Information Enhanced Knowledge Distillation (TIE-KD).Specifically, we design a hierarchical ranking loss to enforce the learning of the top-1 information from the teacher.Additionally, we develop an iterative KD procedure to infuse more additional knowledge by distilling on the data without ground-truth targets.Experiments on WMT'14 English-German, WMT'14 English-French and WMT'16 English-Romanian demonstrate that our method can respectively boost Transformer base students by +1.04, +0.60 and +1.11BLEU scores and significantly outperform the vanilla word-level KD baseline.Besides, our method shows higher generalizability on different teacher-student capacity gaps than existing KD techniques.
ACTA: Short-Answer Grading in High-Stakes Medical Exams

This paper presents the ACTA system, which performs automated short-answer grading in the domain of high-stakes medical exams.The system builds upon previous work on neural similarity-based grading approaches by applying these to the medical domain and utilizing contrastive learning as a means to optimize the similarity metric.ACTA is evaluated against three strong baselines and is developed in alignment with operational needs, where low-confidence responses are flagged for human review.Learning curves are explored to understand the effects of training data on performance.The results demonstrate that ACTA leads to substantially lower number of responses being flagged for human review, while maintaining high classification accuracy.
LSJSP at SemEval-2023 Task 2: FTBC: A FastText based framework with pre-trained BERT for NER

This study introduces the system submitted to the SemEval 2022 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) by the LSJSP team.We propose FTBC, a FastTextbased framework with pre-trained Bert for NER tasks with complex entities and over a noisy dataset.Our system achieves an average of 58.27% F1 score (fine-grained) and 75.79%F1 score (coarsegrained) across all languages.FTBC outperforms the baseline BERT-CRF model on all 12 monolingual tracks.
Label efficient semi-supervised conversational intent classification

To provide a convenient shopping experience and to answer user queries at scale, conversational platforms are essential for e-commerce.The user queries can be prepurchase questions, such as product specifications and delivery time related, or postpurchase queries, such as exchange and return.A chatbot should be able to understand and answer a variety of such queries to help users with relevant information.One of the important modules in the chatbot is automated intent identification, i.e., understanding the user's intention from the query text.Due to non-English speaking users interacting with the chatbot, we often get a significant percentage of code mix queries and queries with grammatical errors, which makes the problem more challenging.This paper proposes a simple yet competent Semi-Supervised Learning (SSL) approach for label-efficient intent classification.We use a small labeled corpus and relatively larger unlabeled query data to train a transformer model.For training the model with labeled data, we explore supervised MixUp data augmentation.To train with unlabeled data, we explore label consistency with dropout noise.We experiment with different pre-trained transformer architectures, such as BERT and sentence-BERT.Experimental results demonstrate that the proposed approach significantly improves over the supervised baseline, even with a limited labeled set.A variant of the model is currently deployed in production.
Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts.It can hurt the performance of pre-trained models on text simplification tasks.In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts.We continue pre-training BART, a representative model, to obtain SimpleBART.It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART.At the end, we compare Sim-pleBART with several representative large language models (LLMs).
Compounding Geometric Operations for Knowledge Graph Completion

Geometric transformations including translation, rotation, and scaling are commonly used operations in image processing.Besides, some of them are successfully used in developing effective knowledge graph embedding (KGE).Inspired by the synergy, we propose a new KGE model by leveraging all three operations in this work.Since translation, rotation, and scaling operations are cascaded to form a composite one, the new model is named Com-poundE.By casting CompoundE in the framework of group theory, we show that quite a few distanced-based KGE models are special cases of CompoundE.CompoundE extends the simple distance-based scoring functions to relation-dependent compound operations on head and/or tail entities.To demonstrate the effectiveness of CompoundE, we perform three prevalent KG prediction tasks including link prediction, path query answering, and entity typing, on a range of datasets.CompoundE outperforms extant models consistently, demonstrating its effectiveness and flexibility.