Reviewerly/exHarmony huggingface dataset:
Concise Answers to Complex Questions: Summarization of Long-form Answers

Long-form question answering systems provide rich information by presenting paragraph-level answers, often containing optional background or auxiliary information.While such comprehensive answers are helpful, not all information is required to answer the question (e.g.users with domain knowledge do not need an explanation of background).Can we provide a concise version of the answer by summarizing it, while still addressing the question?We conduct a user study on summarized answers generated from state-of-the-art models and our newly proposed extract-and-decontextualize approach.We find a large proportion of long-form answers (over 90%) in the ELI5 domain can be adequately summarized by at least one system, while complex and implicit answers are challenging to compress.We observe that decontextualization improves the quality of the extractive summary, exemplifying its potential in the summarization task.To promote future work, we provide an extractive summarization dataset covering 1K long-form answers and our user study annotations.Together, we present the first study on summarizing long-form answers, taking a step forward for QA agents that can provide answers at multiple granularities.
Query Refinement Prompts for Closed-Book Long-Form QA

Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, Shashi Narayan. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation

Distilling knowledge from a high-resource task, e.g., machine translation, is an effective way to alleviate the data scarcity problem of endto-end speech translation.However, previous works simply use the classical knowledge distillation that does not allow for adequate transfer of knowledge from machine translation.In this paper, we propose a comprehensive knowledge distillation framework for speech translation, CKDST, which is capable of comprehensively and effectively distilling knowledge from machine translation to speech translation from two perspectives: cross-modal contrastive representation distillation and simultaneous decoupled knowledge distillation.In the former, we leverage a contrastive learning objective to optimize the mutual information between speech and text representations for representation distillation in the encoder.In the later, we decouple the non-target class knowledge from target class knowledge for logits distillation in the decoder.Experiments on the MuST-C benchmark dataset demonstrate that our CKDST substantially improves the baseline by 1.2 BLEU on average in all translation directions, and outperforms previous state-ofthe-art end-to-end and cascaded speech translation models.The source code is available at https://github.com/ethanyklei/CKDST.
PMAES: Prompt-mapping Contrastive Learning for Cross-prompt Automated Essay Scoring

Current cross-prompt automated essay scoring (AES) is a challenging task due to the large discrepancies between different prompts, such as different genres and expressions.The main goal of current cross-prompt AES systems is to learn enough shared features between the source and target prompts to grade well on the target prompt.However, because the features are captured based on the original prompt representation, they may be limited by being extracted directly between essays.In fact, when the representations of two prompts are more similar, we can gain more shared features between them.Based on this motivation, in this paper, we propose a learning strategy called "prompt-mapping" to learn about more consistent representations of source and target prompts.In this way, we can obtain more shared features between the two prompts and use them to better represent the essays for the target prompt.Experimental results on the ASAP++ dataset demonstrate the effectiveness of our method.We also design experiments in different settings to show that our method can be applied in different scenarios.Our code is available at https://github.com/gdufsnlp/PMAES.
Type Enhanced BERT for Correcting NER Errors

We introduce the task of correcting named entity recognition (NER) errors without retraining the model.After a NER model is trained and deployed in production, it makes prediction errors, which usually need to be fixed quickly.To address this problem, we firstly construct a gazetteer containing named entities and corresponding possible entity types.And then, we propose type-enhanced BERT (TyBERT), a method that integrates the named entity's type information into BERT by an adapter layer.When errors are identified, we can repair the model by updating the gazetteer.In other words, the gazetteer becomes a trigger to control the NER model's output.The experiment results in multiple corpus show the effectiveness of our method, which outperforms strong baselines.
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models

While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use.Therefore, effectively compressing large-scale PLMs becomes an increasingly important problem.Quantization, which represents high-precision tensors with low-bit fix-point format, is a viable solution.However, most existing quantization methods are task-specific, requiring customized training and quantization with a large number of trainable parameters on each individual task.Inspired by the observation that the overparameterization nature of PLMs makes it possible to freeze most of the parameters during the fine-tuning stage, in this work, we propose a novel "quantize before fine-tuning" framework, PreQuant, that differs from both quantizationaware training and post-training quantization.PreQuant is compatible with various quantization strategies, with outlier-aware parameterefficient fine-tuning incorporated to correct the induced quantization error.We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5.We also provide an empirical investigation into the workflow of PreQuant, which sheds light on its efficacy.
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning

Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time.To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in its past.In this paper, we study catastrophic forgetting, as well as methods to minimize this, in a massively multilingual continual learning framework involving up to 51 languages and covering both classification and sequence labeling tasks.We present LR ADJUST, a learning rate scheduling method that is simple, yet effective in preserving new information without strongly overwriting past knowledge.Furthermore, we show that this method is effective across multiple continual learning approaches.Finally, we provide further insights into the dynamics of catastrophic forgetting in this massively multilingual setup.
Adversarial Training for Low-Resource Disfluency Correction

Disfluencies commonly occur in conversational speech.Speech with disfluencies can result in noisy Automatic Speech Recognition (ASR) transcripts, which affects downstream tasks like machine translation.In this paper, we propose an adversarially-trained sequencetagging model for Disfluency Correction (DC) that utilizes a small amount of labeled real disfluent data in conjunction with a large amount of unlabeled data.We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languages-Bengali, Hindi, and Marathi (all from the Indo-Aryan family).Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments.We achieve an average 6.15 points improvement in F1-score over competitive baselines across all three languages mentioned.To the best of our knowledge, we are the first to utilize adversarial training for DC and use it to correct stuttering disfluencies in English, establishing a new benchmark for this task.
Target-Oriented Relation Alignment for Cross-Lingual Stance Detection

Stance detection is an important task in text mining and social media analytics, aiming to automatically identify the user's attitude toward a specific target from text, and has wide applications in a variety of domains.Previous work on stance detection has mainly focused on monolingual setting.To address the problem of imbalanced language resources, crosslingual stance detection is proposed to transfer the knowledge learned from a high-resource (source) language (typically English) to another low-resource (target) language.However, existing research on cross-lingual stance detection has ignored the inconsistency in the occurrences and distributions of targets between languages, which consequently degrades the performance of stance detection in lowresource languages.In this paper, we first identify the target inconsistency issue in crosslingual stance detection, and propose a finegrained Target-oriented Relation Alignment (TaRA) method for the task, which considers both target-level associations and languagelevel alignments.Specifically, we propose the Target Relation Graph to learn the in-language and cross-language target associations.We further devise the relation alignment strategy to enable knowledge transfer between semantically correlated targets across languages.Experimental results on the representative datasets demonstrate the effectiveness of our method compared to competitive methods under variant settings.
Quantifying Train-Evaluation Overlap with Nearest Neighbors

Characterizing benchmark datasets is crucial to interpreting model performance.In this work, we study train-evaluation overlap as a measure of an individual dataset's adequacy to evaluate model generalization over a wide range of datasets.We quantify the overlap with a simple novel metric based on a nearest neighbors approach between the training and evaluation sets.We identify nearest training examples for each evaluation example by mapping instances with generic and task-specific embedding methods.Our study on eleven classification and extractive QA tasks reveals a wide range of trainevaluation overlap, and we show that the data collection method of the dataset and the difficulty of the task may play a role in the amount of overlap.Lastly, we use our nearest neighbor analysis to identify challenging or potentially mislabeled examples.Our analysis quantifies train-evaluation overlap, providing insights for constructing datasets to study generalization.
GeoDRL: A Self-Learning Framework for Geometry Problem Solving using Reinforcement Learning in Deductive Reasoning

Ensuring both interpretability and correctness is a great challenge in automated geometry problem solving (GPS), and the scarcity of labeled data hinders learning mathematical reasoning from samples.Therefore, we present GeoDRL, a self-learning geometry problem solving framework that integrates logic graph deduction and Deep Reinforcement Learning (DRL) to optimize geometry reasoning as a Markov Decision Process.GeoDRL employs a Graph Neural Network on a Geometry Logic Graph, updating the problem state using a symbolic system.Incorporating DRL into deductive reasoning enables GeoDRL to achieve unsupervised self-learning while maintaining correctness.GeoDRL, through unsupervised learning, exhibits enhanced accuracy in the Geome-try3K dataset, improving by 11.1% over previous SOTA methods, and simultaneously boosts efficiency and interpretability.
“Knowledge is Power”: Constructing Knowledge Graph of Abdominal Organs and Using Them for Automatic Radiology Report Generation

In conventional radiology practice, the radiologist dictates the diagnosis to the transcriptionist, who then prepares a preliminary formatted report referring to the notes, after which the radiologist reviews the report, corrects the errors, and signs off.This workflow is prone to delay and error.In this paper, we report our work on automatic radiology report generation from radiologists' dictation, which is in collaboration with a startup about to become Unicorn.A major contribution of our work is the set of knowledge graphs (KGs) of ten abdominal organs-Liver, Kidney, Gallbladder, Uterus, Urinary bladder, Ovary, Pancreas, Prostate, Biliary Tree, and Bowel.Our method for constructing these KGs relies on extracting entity1relation-entity2 triplets from a large collection (about 10,000) of free-text radiology reports.The quality and coverage of the KGs are verified by two experienced radiologists (practicing for the last 30 years and 8 years, respectively).The dictation of the radiologist is automatically converted to what is called a pathological description which is the clinical description of the findings of the radiologist during ultrasonography (USG).Our knowledge-enhanced deep learning model improves the reported BLEU-3, ROUGE-L, METEOR, and CIDEr scores of the pathological description generation by 2%, 4%, 2% and 2% respectively.To the best of our knowledge, this is the first attempt at representing the abdominal organs in the form of knowledge graphs and utilising these graphs for the automatic generation of USG reports.A Minimum Viable Product (MVP) has been made available to the beta users, i.e., radiologists of reputed hospitals, for testing and evaluation.Our solution guarantees report generation within 30 seconds of running a scan.
Dual Class Knowledge Propagation Network for Multi-label Few-shot Intent Detection

Multi-label intent detection aims to assign multiple labels to utterances and attracts increasing attention as a practical task in task-oriented dialogue systems.As dialogue domains change rapidly and new intents emerge fast, the lack of annotated data motivates multi-label few-shot intent detection.However, previous studies are confused by the identical representation of the utterance with multiple labels and overlook the intrinsic intra-class and inter-class interactions.To address these two limitations, we propose a novel dual class knowledge propagation network in this paper.In order to learn well-separated representations for utterances with multiple intents, we first introduce a labelsemantic augmentation module incorporating class name information.For better consideration of the inherent intra-class and inter-class relations, an instance-level and a class-level graph neural network are constructed, which not only propagate label information but also propagate feature structure.And we use a simple yet effective method to predict the intent count of each utterance.Extensive experimental results on two multi-label intent datasets have demonstrated that our proposed method outperforms strong baselines by a large margin.
Text Style Transfer with Contrastive Transfer Pattern Mining

Text style transfer (TST) is an important task in natural language generation, which aims to alter the stylistic attributes (e.g., sentiment) of a sentence and keep its semantic meaning unchanged.Most existing studies mainly focus on the transformation between styles, yet ignore that this transformation can be actually carried out via different hidden transfer patterns.To address this problem, we propose a novel approach, contrastive transfer pattern mining (CTPM), which automatically mines and utilizes inherent latent transfer patterns to improve the performance of TST.Specifically, we design an adaptive clustering module to automatically discover hidden transfer patterns from the data, and introduce contrastive learning based on the discovered patterns to obtain more accurate sentence representations, and thereby benefit the TST task.To the best of our knowledge, this is the first work that proposes the concept of transfer patterns in TST, and our approach can be applied in a plug-andplay manner to enhance other TST methods to further improve their performance.Extensive experiments on benchmark datasets verify the effectiveness and generality of our approach.
Zero-shot Cross-lingual Transfer With Learned Projections Using Unlabeled Target-Language Data

Adapters have emerged as a parameter-efficient Transformer-based framework for cross-lingual transfer by inserting lightweight languagespecific modules (language adapters) and taskspecific modules (task adapters) within pretrained multilingual models.Zero-shot transfer is enabled by pairing the language adapter in the target language with an appropriate task adapter in a source language.If our target languages are known apriori, we explore how zeroshot transfer can be further improved within the adapter framework by utilizing unlabeled text during task-specific finetuning.We construct language-specific subspaces using standard linear algebra constructs and selectively project source-language representations into the target language subspace during task-specific finetuning using two schemes.Our experiments on three cross-lingual tasks, Named Entity Recognition (NER), Question Answering (QA) and Natural Language Inference (NLI) yield consistent benefits compared to adapter baselines over a wide variety of target languages with up to 11% relative improvement in NER, 2% relative improvement in QA and 5% relative improvement in NLI.
E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition

Most named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty, which is critical to the reliability of NER systems in open environments.Evidential deep learning (EDL) has recently been proposed as a promising solution to explicitly model predictive uncertainty for classification tasks.However, directly applying EDL to NER applications faces two challenges, i.e., the problems of sparse entities and OOV/OOD entities in NER tasks.To address these challenges, we propose a trustworthy NER framework named E-NER 1 by introducing two uncertainty-guided loss terms to the conventional EDL, along with a series of uncertainty-guided training strategies.Experiments show that E-NER can be applied to multiple NER paradigms to obtain accurate uncertainty estimation.Furthermore, compared to state-of-the-art baselines, the proposed method achieves a better OOV/OOD detection performance and better generalization ability on OOV entities.
Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Visual Word Sense Disambiguation (VWSD) is a task to find the image that most accurately depicts the correct sense of the target word for the given context.Previously, image-text matching models often suffered from recognizing polysemous words.This paper introduces an unsupervised VWSD approach that uses gloss information of an external lexical knowledge-base, especially the sense definitions.Specifically, we suggest employing Bayesian inference to incorporate the sense definitions when sense information of the answer is not provided.In addition, to ameliorate the out-of-vocabulary (OOV) issue, we propose a context-aware definition generation with GPT-3.Experimental results show that VWSD performance increased significantly with our Bayesian inference-based approach.In addition, our context-aware definition generation achieved prominent performance improvement in OOV examples exhibiting better performance than the existing definition generation method.!
Gender-Inclusive Grammatical Error Correction through Augmentation

In this paper we show that GEC systems display gender bias related to the use of masculine and feminine terms and the gender-neutral singular “they”. We develop parallel datasets of texts with masculine and feminine terms, and singular “they”, and use them to quantify gender bias in three competitive GEC systems. We contribute a novel data augmentation technique for singular “they” leveraging linguistic insights about its distribution relative to plural “they”. We demonstrate that both this data augmentation technique and a refinement of a similar augmentation technique for masculine and feminine terms can generate training data that reduces bias in GEC systems, especially with respect to singular “they” while maintaining the same level of quality.
ECNU_MIV at SemEval-2023 Task 1: CTIM - Contrastive Text-Image Model for Multilingual Visual Word Sense Disambiguation

This paper describes the performance of MIV team in SemEval-2023 Task-1-Visual Word Sense Disambiguation (Visual-WSD).This task is to give a potentially ambiguous word and some limited textual context and select among a set of candidate images the one which corresponds to the intended meaning of the target word, which plays a critical role in human language understanding.Our team focuses on the multimodal domain of images and texts, we propose a model that can learn the matching relationship between text-image pairs by contrastive learning.More specifically, We train the model from the labeled data provided by the official organizer, after pre-training, texts are used to reference learned visual concepts enabling visual word sense disambiguation tasks.In addition, the top results our teams got have been released showing the effectiveness of our solution.We release our code at https://github.com/Insaner1004/VWSD.
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training

Improving text representation has attracted much attention to achieve expressive text-tospeech (TTS).However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low training efficiency and difficulty in prosody modeling.We propose CLAPSpeech, a cross-modal contrastive pre-training framework that explicitly learns the prosody variance of the same text token under different contexts.Specifically, 1) We encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space with the elaborate design of the encoder inputs and contrastive loss; 2) We introduce a multi-scale pretraining pipeline to capture prosody patterns in multiple levels.We show how to incorporate CLAPSpeech into existing TTS models for better prosody.Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker TTS.We also deeply analyze the principle behind the performance of CLAPSpeech.Ablation studies demonstrate the necessity of each component in our method.
Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting

Knowledge Graph Completion (KGC) often requires both KG structural and textual information to be effective.Pre-trained Language Models (PLMs) have been used to learn the textual information, usually under the fine-tune paradigm for the KGC task.However, the finetuned PLMs often overwhelmingly focus on the textual information and overlook structural knowledge.To tackle this issue, this paper proposes CSProm-KG (Conditional Soft Prompts for KGC) which maintains a balance between structural information and textual knowledge.CSProm-KG only tunes the parameters of Conditional Soft Prompts that are generated by the entities and relations representations.We verify the effectiveness of CSProm-KG on three popular static KGC benchmarks WN18RR, FB15K-237 and Wikidata5M, and two temporal KGC benchmarks ICEWS14 and ICEWS05-15.CSProm-KG outperforms competitive baseline models and sets new state-of-the-art on these benchmarks.We conduct further analysis to show (i) the effectiveness of our proposed components, (ii) the efficiency of CSProm-KG, and (iii) the flexibility of CSProm-KG 1 .