Reviewerly/exHarmony huggingface dataset:
Text Augmented Open Knowledge Graph Completion via Pre-Trained Language Models

The mission of open knowledge graph (KG) completion is to draw new findings from known facts.Existing works that augment KG completion require either (1) factual triples to enlarge the graph reasoning space or (2) manually designed prompts to extract knowledge from a pre-trained language model (PLM), exhibiting limited performance and requiring expensive efforts from experts.To this end, we propose TAGREAL that automatically generates quality query prompts and retrieves support information from large text corpora to probe knowledge from PLM for KG completion.The results show that TAGREAL achieves state-of-the-art performance on two benchmark datasets.We find that TAGREAL has superb performance even with limited training data, outperforming existing embedding-based, graph-based, and PLM-based methods.
Improving Pretraining Techniques for Code-Switched NLP

Pretrained models are a mainstay in modern NLP applications.Pretraining requires access to large volumes of unlabeled text.While monolingual text is readily available for many of the world's languages, access to large quantities of code-switched text (i.e., text with tokens of multiple languages interspersed within a sentence) is much more scarce.Given this resource constraint, the question of how pretraining using limited amounts of code-switched text could be altered to improve performance for code-switched NLP becomes important to tackle.In this paper, we explore different masked language modeling (MLM) pretraining techniques for code-switched text that are cognizant of language boundaries prior to masking.The language identity of the tokens can either come from human annotators, trained language classifiers, or simple relative frequencybased estimates.We also present an MLM variant by introducing a residual connection from an earlier layer in the pretrained model that uniformly boosts performance on downstream tasks.Experiments on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), involving four code-switched language pairs (Hindi-English, Spanish-English, Tamil-English, Malayalam-English) yield relative improvements of up to 5.8 and 2.7 F1 scores on QA (Hindi-English) and SA (Tamil-English), respectively, compared to standard pretraining techniques.To understand our task improvements better, we use a series of probes to study what additional information is encoded by our pretraining techniques and also introduce an auxiliary loss function that explicitly models language identification to further aid the residual MLM variants.
Generating User-Engaging News Headlines

The potential choices for news article headlines are enormous, and finding the right balance between conveying the essential message and capturing the reader's attention is key to effective headlining.However, presenting the same news headline to all readers is a suboptimal strategy, because it does not take into account the different preferences and interests of diverse readers, who may be confused about why a particular article has been recommended to them and do not see a clear connection between their interests and the recommended article.In this paper, we present a novel framework that addresses these challenges by incorporating user profiling to generate personalized headlines, and a combination of automated and human evaluation methods to determine user preference for personalized headlines.Our framework utilizes a learnable relevance function to assign personalized signature phrases to users based on their reading histories, which are then used to personalize headline generation.Through extensive evaluation, we demonstrate the effectiveness of our proposed framework in generating personalized headlines that meet the needs of a diverse audience.Our framework has the potential to improve the efficacy of news recommendations and facilitate creation of personalized content.
Storyline-Centric Detection of Aphasia and Dysarthria in Stroke Patient Transcripts

Aphasia and dysarthria are both common symptoms of stroke, affecting around 30% and 50% of acute ischemic stroke patients. In this paper, we propose a storyline-centric approach to detect aphasia and dysarthria in acute stroke patients using transcribed picture descriptions alone. Our pipeline enriches the training set with healthy data to address the lack of acute stroke patient data and utilizes knowledge distillation to significantly improve upon a document classification baseline, achieving an AUC of 0.814 (aphasia) and 0.764 (dysarthria) on a patient-only validation set.
Improving Syntactic Probing Correctness and Robustness with Control Tasks

Syntactic probing methods have been used to examine whether and how pre-trained language models (PLMs) encode syntactic relations.However, the probing methods are usually biased by the PLMs' memorization of common word co-occurrences, even if they do not form syntactic relations.This paper presents a random-word-substitution and random-labelmatching control task to reduce these biases and improve the robustness of syntactic probing methods.Our control tasks are also shown to notably improve the consistency of probing results between different probing methods and make the methods more robust with respect to the text attributes of the probing instances.Our control tasks make syntactic probing methods better at reconstructing syntactic relations and more generalizable to unseen text domains.Our experiments show that our proposed control tasks are effective on different PLMs, probing methods, and syntactic relations.
The NiuTrans End-to-End Speech Translation System for IWSLT23 English-to-Chinese Offline Task

This paper describes the NiuTrans end-to-end speech translation system submitted for the IWSLT 2023 English-to-Chinese offline task.Our speech translation models are composed of pre-trained ASR and MT models under the stacked acoustic and textual encoding framework.Several pre-trained models with diverse architectures and input representations (e.g., log Mel-filterbank and waveform) were utilized.We proposed an iterative data augmentation method to iteratively improve the performance of the MT models and generate the pseudo ST data through MT systems.We then trained ST models with different structures and data settings to enhance ensemble performance.Experimental results demonstrate that our NiuTrans system achieved a BLEU score of 29.22 on the MuST-C En-Zh tst-COMMON set, outperforming the previous year's submission by 0.12 BLEU despite using less MT training data.
Revisiting Token Dropping Strategy in Efficient BERT Pretraining

Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at several middle layers.It can effectively reduce the training time without degrading much performance on downstream tasks.However, we empirically find that token dropping is prone to a semantic loss problem and falls short in handling semantic-intense tasks ( §2).Motivated by this, we propose a simple yet effective semantic-consistent learning method (SCTD) to improve the token dropping.SCTD aims to encourage the model to learn how to preserve the semantic information in the representation space.Extensive experiments on 12 tasks show that, with the help of our SCTD, token dropping can achieve consistent and significant performance gains across all task types and model sizes.More encouragingly, SCTD saves up to 57% of pretraining time and brings up to +1.56% average improvement over the vanilla token dropping.
Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation

Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in endto-end speech translation (E2E ST).The commonplace "modality gap" between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning.However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance.On the other hand, we find that there has another gap, which we call the "capacity gap": high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the overfitting.In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for enfr on the MuST-C dataset.Code and models are available at https://github.com/hannlp/TAB.
Hard Sample Aware Prompt-Tuning

Prompt-tuning based few-shot learning has garnered increasing attention in recent years due to its efficiency and promising capability.To achieve the best performance for natural language processing (NLP) tasks with just a few samples, it is vital to include as many informative samples as possible and to avoid misleading ones.However, there is no work in prompttuning literature addressing the problem of differentiating informative hard samples from misleading ones in model training, which is challenging due to the lack of supervision signals about the quality of the samples to train a wellperformed model.We propose a framework named Hard Sample Aware Prompt-Tuning (HardPT) to solve the non-differentiable problem in hard sample identification with reinforcement learning, and to strengthen the discrimination of the feature space without changing the original data distribution via an adaptive contrastive learning method.An extensive empirical study on a series of NLP tasks demonstrates the capability of HardPT in few-shot scenarios.HardPT obtains new state-of-the-art results on all evaluated NLP tasks, including pushing the SST-5 accuracy to 49.5% (1.1% point absolute improvement), QNLI accuracy to 74.6% (1.9% absolute improvement), NMLI accuracy to 71.5 (0.7% absolute improvement), TACREV F 1 -score to 28.2 (1.0 absolute improvement), and i2b2/VA F 1 -score to 41.2 (1.3 absolute improvement).
PAD-Net: An Efficient Framework for Dynamic Networks

Dynamic networks, e.g., Dynamic Convolution (DY-Conv) and the Mixture of Experts (MoE), have been extensively explored as they can considerably improve the model's representation power with acceptable computational cost.The common practice in implementing dynamic networks is to convert the given static layers into fully dynamic ones where all parameters are dynamic (at least within a single layer) and vary with the input.However, such a fully dynamic setting may cause redundant parameters and high deployment costs, limiting the applicability of dynamic networks to a broader range of tasks and models.The main contributions of our work are challenging the basic commonsense in dynamic networks and proposing a partially dynamic network, namely PAD-Net, to transform the redundant dynamic parameters into static ones.Also, we further design Iterative Mode Partition to partition dynamic and static parameters efficiently.Our method is comprehensively supported by large-scale experiments with two typical advanced dynamic architectures, i.e., DY-Conv and MoE, on both image classification and GLUE benchmarks.Encouragingly, we surpass the fully dynamic networks by +0.7% top-1 acc with only 30% dynamic parameters for ResNet-50 and +1.9% average score in language understanding with only 50% dynamic parameters for BERT.
Explanation-based Finetuning Makes Models More Robust to Spurious Cues

Large Language Models (LLMs) are so powerful that they sometimes learn correlations between labels and features that are irrelevant to the task, leading to poor generalization on outof-distribution data.We propose explanationbased finetuning as a general approach to mitigate LLMs' reliance on spurious correlations.Unlike standard finetuning where the model only predicts the answer given the input, we finetune the model to additionally generate a free-text explanation supporting its answer.To evaluate our method, we finetune the model on artificially constructed training sets containing different types of spurious cues, and test it on a test set without these cues.Compared to standard finetuning, our method makes GPT-3 (davinci) remarkably more robust against spurious cues in terms of accuracy drop across four classification tasks: ComVE (+1.2),CREAK (+9.1), e-SNLI (+15.4), and SBIC (+6.5).The efficacy generalizes across multiple model families and scales, with greater gains for larger models.Finally, our method also works well with explanations generated by the model, implying its applicability to more datasets without human-written explanations. 1,2
Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data

Relation extraction (RE) aims to extract relations from sentences and documents.Existing relation extraction models typically rely on supervised machine learning.However, recent studies showed that many RE datasets are incompletely annotated.This is known as the false negative problem in which valid relations are falsely annotated as no_relation.Models trained with such data inevitably make similar mistakes during the inference stage.Selftraining has been proven effective in alleviating the false negative problem.However, traditional self-training is vulnerable to confirmation bias and exhibits poor performance in minority classes.To overcome this limitation, we proposed a novel class-adaptive re-sampling self-training framework.Specifically, we resampled the pseudo-labels for each class by precision and recall scores.Our re-sampling strategy favored the pseudo-labels of classes with high precision and low recall, which improved the overall recall without significantly compromising precision.We conducted experiments on document-level and biomedical relation extraction datasets, and the results showed that our proposed self-training framework consistently outperforms existing competitive methods on the Re-DocRED and ChemDisgene datasets when the training data are incompletely annotated 1 .
Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model

We use both Bayesian and neural models to dissect a data set of Chinese learners' pre-and post-interventional responses to two tests measuring their understanding of English prepositions.The results mostly replicate previous findings from frequentist analyses and reveal new and crucial interactions between student ability, task type, and stimulus sentence.Given the sparsity of the data as well as high diversity among learners, the Bayesian method proves most useful; but we also see potential in using language model probabilities as predictors of grammaticality and learnability. 1
Class Lifelong Learning for Intent Detection via Structure Consolidation Networks

Intent detection, which estimates diverse intents behind user utterances, is an essential component of task-oriented dialogue systems.Previous intent detection models are usually trained offline, which can only handle predefined intent classes.In the real world, new intents may keep challenging deployed models.For example, with the prevalence of the COVID-19 pandemic, users may pose various issues related to the pandemic to conversational systems, which brings many new intents.A general intent detection model should be intelligent enough to continually learn new data and recognize new arriving intent classes.Therefore, this work explores Class Lifelong Learning for Intent Detection (CLL-ID), where the model continually learns new intent classes from new data while avoiding catastrophic performance degradation on old data.To this end, we propose a novel lifelong learning method, called Structure Consolidation Networks (SCN), which consists of structure-based retrospection and contrastive knowledge distillation to handle the problems of expression diversity and class imbalance in the CLL-ID task.In addition to formulating the new task, we construct 3 benchmarks based on 8 intent detection datasets.Experimental results demonstrate the effectiveness of SCN, which significantly outperforms previous lifelong learning methods on the three benchmarks.
Learning Optimal Policy for Simultaneous Machine Translation via Binary Search

Simultaneous machine translation (SiMT) starts to output translation while reading the source sentence and needs a precise policy to decide when to output the generated translation.Therefore, the policy determines the number of source tokens read during the translation of each target token.However, it is difficult to learn a precise translation policy to achieve good latency-quality trade-offs, because there is no golden policy corresponding to parallel sentences as explicit supervision.In this paper, we present a new method for constructing the optimal policy online via binary search.By employing explicit supervision, our approach enables the SiMT model to learn the optimal policy, which can guide the model in completing the translation during inference.Experiments on four translation tasks show that our method can exceed strong baselines across all latency scenarios 1
Emotion Cause Extraction on Social Media without Human Annotation

In social media, there is a vast amount of information pertaining to people's emotions and the corresponding causes.The emotion cause extraction (ECE) from social media data is an important research area that has not been thoroughly explored due to the lack of fine-grained annotations.Early studies referred to either unsupervised rule-based methods or supervised machine learning methods using a number of manually annotated data in specific domains.However, the former suffers from limitations in extraction performance, while the latter is constrained by the availability of fine-grained annotations and struggles to generalize to diverse domains.To address these issues, this paper proposes a new ECE framework on Chinese social media that achieves high extraction performance and generalizability without relying on human annotation.Specifically, we design a more dedicated rule-based system based on constituency parsing tree to discover causal patterns in social media.This system enables us to acquire large amounts of fine-grained annotated data.Next, we train a neural model on the rule-annotated dataset with a specific training strategy to further improve the model's generalizability.Extensive experiments demonstrate the superiority of our approach over other methods in unsupervised and weakly-supervised settings.
Multimodal Persona Based Generation of Comic Dialogs

We focus on the novel problem of persona based dialogue generation for comic strips.Dialogs in comic strips is a unique and unexplored area where every strip contains utterances from various characters with each one building upon the previous utterances and the associated visual scene.Previous works like Di-aloGPT, PersonaGPT and other dialog generation models encode two-party dialogues and do not account for the visual information.To the best of our knowledge we are the first to propose the paradigm of multimodal persona based dialogue generation.We contribute a novel dataset, COMSET, consisting of 54K strips, harvested from 13 popular comics available online.Further, we propose a multimodal persona-based architecture, MPDIALOG, to generate dialogues for the next panel in the strip which decreases the perplexity score by ∼10 points over strong dialogue generation baseline models.We demonstrate that there is still ample opportunity for improvement, highlighting the importance of building stronger dialogue systems that are able to generate persona-consistent dialogues and understand the context through various modalities.
DynaMiTE: Discovering Explosive Topic Evolutions with User Guidance

Dynamic topic models (DTMs) analyze text streams to capture the evolution of topics.Despite their popularity, existing DTMs are either fully supervised, requiring expensive human annotations, or fully unsupervised, producing topic evolutions that often do not cater to a user's needs.Further, the topic evolutions produced by DTMs tend to contain generic terms that are not indicative of their designated time steps.To address these issues, we propose the task of discriminative dynamic topic discovery.This task aims to discover topic evolutions from temporal corpora that distinctly align with a set of user-provided category names and uniquely capture topics at each time step.We solve this task by developing DynaMiTE, a framework that ensembles semantic similarity, category indicative, and time indicative scores to produce informative topic evolutions.Through experiments on three diverse datasets, including the use of a newly-designed human evaluation experiment, we demonstrate that DynaMiTE is a practical and efficient framework for helping users discover high-quality topic evolutions suited to their interests.
Value type: the bridge to a better DST model

Value type of the slots can provide lots of useful information for DST tasks.However, it has been ignored in most previous works.In this paper, we propose a new framework for DST task based on these value types.Firstly, we extract the type of token from each turn.Specifically, we divide the slots in the dataset into 9 categories according to the type of slot value, and then train a Ner model to extract the corresponding type-entity from each turn of conversation according to the token.Secondly, we improve the attention mode which is integrated into value type information between the slot and the conversation history to help each slot pay more attention to the turns that contain the same value type.Meanwhile, we introduce a sampling strategy to integrate these types into the attention formula, which decrease the error of Ner model.Finally, we conduct a comprehensive experiment on two multi-domain taskoriented conversation datasets, MultiWOZ 2.1 and MultiWOZ 2.4.The ablation experimental results show that our method is effective on both datasets, which verify the necessity of considering the type of slot value.⇤ The first two authors contribute equally.Weiran Xu is the corresponding author.Sys: yes, the autumn house is on the east part of time, the prices are cheap and it is 4 stars.is there anything else you would like to know?Usr: no, i just want to book it for 2 people for 5 nights starting wednesday.Turn_label: hotel-area=east, hotel-book day=wednesday, hotel-people=2, hotel-book stay=5, hotel-princerange=cheap, hotel-stars=4Sys: your friend has good taste
Information-Transport-based Policy for Simultaneous Translation

Simultaneous translation (ST) outputs translation while receiving the source inputs, and hence requires a policy to determine whether to translate a target token or wait for the next source token.The major challenge of ST is that each target token can only be translated based on the current received source tokens, where the received source information will directly affect the translation quality.So naturally, how much source information is received for the translation of the current target token is supposed to be the pivotal evidence for the ST policy to decide between translating and waiting.In this paper, we treat the translation as information transport from source to target and accordingly propose an Information-Transport-based Simultaneous Translation (ITST).ITST quantifies the transported information weight from each source token to the current target token, and then decides whether to translate the target token according to its accumulated received information.Experiments on both text-to-text ST and speech-to-text ST (a.k.a., streaming speech translation) tasks show that ITST outperforms strong baselines and achieves state-of-the-art performance 1 .