Reviewerly/exHarmony huggingface dataset:
DKAF: KB Arbitration for Learning Task-Oriented Dialog Systems with Dialog-KB Inconsistencies

Task-oriented dialog (TOD) agents often ground their responses on external knowledge bases (KBs).These KBs can be dynamic and may be updated frequently.Existing approaches for learning TOD agents assume the KB snapshot contemporary to each individual dialog is available during training.However, in real-world scenarios, only the latest KB snapshot is available during training and as a result, the train dialogs may contain facts conflicting with the latest KB.These dialog-KB inconsistencies in the training data may potentially confuse the TOD agent learning algorithm.In this work, we define the novel problem of learning a TOD agent with dialog-KB inconsistencies in the training data.We propose a Dialog-KB Arbitration Framework (DKAF) which reduces the dialog-KB inconsistencies by predicting the contemporary KB snapshot for each train dialog.These predicted KB snapshots are then used for training downstream TOD agents.As there are no existing datasets with dialog-KB inconsistencies, we systematically introduce inconsistencies in two publicly available dialog datasets.We show that TOD agents trained with DKAF perform better than existing baselines on both these datasets.
Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore

Toxic content is a global problem, but most resources for detecting toxic content are in English.When datasets are created in other languages, they often focus exclusively on one language or dialect.In many cultural and geographical settings, however, it is common to code-mix languages, combining and interchanging them throughout conversations.To shine a light on this practice, and enable more research into code-mixed toxic content, we introduce SOA, a new multilingual dataset of online attacks.Using the multilingual city-state of Singapore as a starting point, we collect a large corpus of Reddit comments in Indonesian, Malay, Singlish, and other languages, and provide fine-grained hierarchical labels for online attacks.We publish the corpus with rich metadata, as well as additional unlabelled data for domain adaptation.We share comprehensive baseline results, show how the metadata can be used for granular error analysis, and demonstrate the benefits of domain adaptation for detecting multilingual online attacks.
MoNET: Tackle State Momentum via Noise-Enhanced Training for Dialogue State Tracking

Dialogue state tracking (DST) aims to convert the dialogue history into dialogue states which consist of slot-value pairs.As condensed structural information memorizes all dialogue history, the dialogue state in the previous turn is typically adopted as the input for predicting the current state by DST models.However, these models tend to keep the predicted slot values unchanged, which is defined as state momentum in this paper.Specifically, the models struggle to update slot values that need to be changed and correct wrongly predicted slot values in the previous turn.To this end, we propose MoNET to tackle state momentum via noise-enhanced training.First, the previous state of each turn in the training data is noised via replacing some of its slot values.Then, the noised previous state is used as the input to learn to predict the current state, improving the model's ability to update and correct slot values.Furthermore, a contrastive context matching framework is designed to narrow the representation distance between a state and its corresponding noised variant, which reduces the impact of noised state and makes the model better understand the dialogue history.Experimental results on MultiWOZ datasets show that MoNET outperforms previous DST methods.Ablations and analysis verify the effectiveness of MoNET in alleviating state momentum issues and improving the antinoise ability 1 .
PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism

We investigate response generation for multiturn dialogue in generative chatbots.Existing generative models based on RNNs (Recurrent Neural Networks) usually employ the last hidden state to summarize the history, which makes models unable to capture the subtle variability observed in different dialogues and cannot distinguish the differences between dialogues that are similar in composition.In this paper, we propose Pseudo-Variational Gated Recurrent Unit (PVGRU).The key novelty of PVGRU is a recurrent summarizing variable that aggregates the accumulated distribution variations of subsequences.We train PVGRU without relying on posterior knowledge, thus avoiding the training-inference inconsistency problem.PVGRU can perceive subtle semantic variability through summarizing variables that are optimized by two objectives we employ for training: distribution consistency and reconstruction.In addition, we build a Pseudo-Variational Hierarchical Dialogue (PVHD) model based on PVGRU.Experimental results demonstrate that PVGRU can broadly improve the diversity and relevance of responses on two benchmark datasets.
Curriculum Learning for Graph Neural Networks: A Multiview Competence-based Approach

A curriculum is a planned sequence of learning materials and an effective one can make learning efficient and effective for both humans and machines.Recent studies developed effective data-driven curriculum learning approaches for training graph neural networks in language applications.However, existing curriculum learning approaches often employ a single criterion of difficulty in their training paradigms.In this paper, we propose a new perspective on curriculum learning by introducing a novel approach that builds on graph complexity formalisms (as difficulty criteria) and model competence during training.The model consists of a scheduling scheme which derives effective curricula by accounting for different views of sample difficulty and model competence during training.The proposed solution advances existing research in curriculum learning for graph neural networks with the ability to incorporate a fine-grained spectrum of graph difficulty criteria in their training paradigms.Experimental results on real-world link prediction and node classification tasks illustrate the effectiveness of the proposed approach. 1
Neural Unsupervised Reconstruction of Protolanguage Word Forms

We present a state-of-the-art neural approach to the unsupervised reconstruction of ancient word forms.Previous work in this domain used expectation-maximization to predict simple phonological changes between ancient word forms and their cognates in modern languages.We extend this work with neural models that can capture more complicated phonological and morphological changes.At the same time, we preserve the inductive biases from classical methods by building monotonic alignment constraints into the model and deliberately underfitting during the maximization step.We evaluate our performance on the task of reconstructing Latin from a dataset of cognates across five Romance languages, achieving a notable reduction in edit distance from the target word forms compared to previous methods.
Analyzing Bias in Large Language Model Solutions for Assisted Writing Feedback Tools: Lessons from the Feedback Prize Competition Series

This paper analyzes winning solutions from the Feedback Prize competition series hosted from 2021-2022. The competition sought to improve Assisted Writing Feedback Tools (AWFTs) by crowdsourcing Large Language Model (LLM) solutions for evaluating student writing. The winning models are freely available for incorporation into educational applications, but the models need to be assessed for performance and other factors. This study reports the performance accuracy of Feedback Prize-winning models based on demographic factors such as student race/ethnicity, economic disadvantage, and English Language Learner status. Two competitions are analyzed. The first, which focused on identifying discourse elements, demonstrated minimal bias based on students’ demographic factors. However, the second competition, which aimed to predict discourse effectiveness, exhibited moderate bias.
Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses

A human decision-maker benefits the most from an AI assistant that corrects for their biases. For problems such as generating interpretation of a radiology report given findings, a system predicting only highly likely outcomes may be less useful, where such outcomes are already obvious to the user. To alleviate biases in human decision-making, it is worth considering a broad differential diagnosis, going beyond the most likely options. We introduce a new task, "less likely brainstorming," that asks a model to generate outputs that humans think are relevant but less likely to happen. We explore the task in two settings: a brain MRI interpretation generation setting and an everyday commonsense reasoning setting. We found that a baseline approach of training with less likely hypotheses as targets generates outputs that humans evaluate as either likely or irrelevant nearly half of the time; standard MLE training is not effective. To tackle this problem, we propose a controlled text generation method that uses a novel contrastive learning strategy to encourage models to differentiate between generating likely and less likely outputs according to humans. We compare our method with several state-of-the-art controlled text generation models via automatic and human evaluations and show that our models' capability of generating less likely outputs is improved.
Distinguishing Romanized Hindi from Romanized Urdu

We examine the task of distinguishing between Hindi and Urdu when those languages are romanized, i.e., written in the Latin script.Both languages are widely informally romanized, and to the extent that they are identified in the Latin script by language identification systems, they are typically conflated.In the absence of large labeled collections of such text, we consider methods for generating training data.Beginning with a small set of seed words, each of which are strongly indicative of one of the languages versus the other, we prompt a pretrained large language model (LLM) to generate romanized text.Treating text generated from an Urdu prompt as one class and text generated from a Hindi prompt as the other class, we build a binary language identification (LangID) classifier.We demonstrate that the resulting classifier distinguishes manually romanized Urdu Wikipedia text from manually romanized Hindi Wikipedia text far better than chance.We use this classifier to estimate the prevalence of Urdu in a large collection of text labeled as romanized Hindi that has been used to train large language models.These techniques can be applied to bootstrap classifiers in other cases where a dataset is known to contain multiple distinct but related classes, such as different dialects of the same language, but for which labels cannot easily be obtained.
Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks

Coherence is an important aspect of text quality, and various approaches have been applied to coherence modeling.However, existing methods solely focus on a single document's coherence patterns, ignoring the underlying correlation between documents.We investigate a GCN-based coherence model that is capable of capturing structural similarities between documents.Our model first creates a graph structure for each document, from where we mine different subgraph patterns.We then construct a heterogeneous graph for the training corpus, connecting documents based on their shared subgraphs.Finally, a GCN is applied to the heterogeneous graph to model the connectivity relationships.We evaluate our method on two tasks, assessing discourse coherence and automated essay scoring.Results show that our GCN-based model outperforms all baselines, achieving a new state-of-the-art on both tasks.
mCLIP: Multilingual CLIP via Cross-lingual Transfer

Large-scale vision-language pretrained (VLP) models like CLIP have shown remarkable performance on various downstream cross-modal tasks.However, they are usually biased towards English due to the lack of sufficient non-English image-text pairs.Existing multilingual VLP methods often learn retrievalinefficient single-stream models by translationaugmented non-English image-text pairs.In this paper, we introduce mCLIP, a retrievalefficient dual-stream multilingual VLP model, trained by aligning the CLIP model and a Multilingual Text Encoder (MTE) through a novel Triangle Cross-modal Knowledge Distillation (TriKD) method.It is parameter-efficient as only two light projectors on the top of them are updated during distillation.Furthermore, to enhance the token-and sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization.Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval task.
Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning

Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering.To better facilitate research on syllogistic reasoning, we develop a benchmark called SYLLOBASE that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks.We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet.To improve our dataset's naturalness and challenge, we apply GPT-3 to paraphrase the templatebased data and further manually rewrite 1,000 samples as the test set.State-of-the-art pretrained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77% by RoBERTa on SYLLOBASE, which indicates the great challenge of learning diverse syllogistic reasoning types on SYLLOBASE.Our datasets are released at https://github.com/casually-PYlearner/SYLLOBASE.
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence

Sentence-level representations are beneficial for various natural language processing tasks.It is commonly believed that vector representations can capture rich linguistic properties.Currently, large language models (LMs) achieve state-of-the-art performance on sentence embedding.However, some recent works suggest that vector representations from LMs can cause information leakage (Song and Raghunathan, 2020;Pan et al., 2020).In this work, we further investigate the information leakage issue and propose a generative embedding inversion attack (GEIA) that aims to reconstruct input sequences based only on their sentence embeddings.Given the black-box access to a language model, we treat sentence embeddings as initial tokens' representations and train or finetune a powerful decoder model to decode the whole sequences directly.We conduct extensive experiments to demonstrate that our generative inversion attack outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as the original inputs.
A Crosslingual Investigation of Conceptualization in 1335 Languages

Languages differ in how they divide up the world into concepts and words; e.g., in contrast to English, Swahili has a single concept for 'belly' and 'womb'.We investigate these differences in conceptualization across 1,335 languages by aligning concepts in a parallel corpus.To this end, we propose Conceptualizer, a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings.In a detailed linguistic analysis across all languages for one concept ('bird') and an evaluation on gold standard data for 32 Swadesh concepts, we show that Conceptualizer has good alignment accuracy.We demonstrate the potential of research on conceptualization in NLP with two experiments.(1) We define crosslingual stability of a concept as the degree to which it has 1-1 correspondences across languages, and show that concreteness predicts stability.(2) We represent each language by its conceptualization pattern for 83 concepts, and define a similarity measure on these representations.The resulting measure for the conceptual similarity between two languages is complementary to standard genealogical, typological, and surface similarity measures.For four out of six language families, we can assign languages to their correct family based on conceptual similarity with accuracies between 54% and 87%. 1
Exploring Partial Knowledge Base Inference in Biomedical Entity Linking

Biomedical entity linking (EL) consists of named entity recognition (NER) and named entity disambiguation (NED).EL models are trained on corpora labeled by a predefined KB.However, it is a common scenario that only entities within a subset of the KB are precious to stakeholders.We name this scenario partial knowledge base inference: training an EL model with one KB and inferring on the part of it without further training.In this work, we give a detailed definition and evaluation procedures for this practically valuable but significantly understudied scenario and evaluate methods from three representative EL paradigms.We construct partial KB inference benchmarks and witness a catastrophic degradation in EL performance due to dramatically precision drop.Our findings reveal these EL paradigms can not correctly handle unlinkable mentions (NIL), so they are not robust to partial KB inference.We also propose two simple-and-effective redemption methods to combat the NIL issue with little computational overhead.Codes are released at https://github.com/Yuanhy1997/PartialKB-EL.
Alleviating Over-smoothing for Unsupervised Sentence Representation

Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities.Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task.Experimentally, we observe that the over-smoothing problem reduces the capacity of these powerful PLMs, leading to sub-optimal sentence representations.In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate layers, improving the quality of the sentence representation.Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting, which can be seen as a plug-andplay contrastive framework for learning unsupervised sentence representation.Extensive results prove that SSCL brings the superior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets.Our codes are available at https: //github.com/nuochenpku/SSCL.
A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models

Compression techniques for deep learning have become increasingly popular, particularly in settings where latency and memory constraints are imposed.Several methods, such as pruning, distillation, and quantization, have been adopted for compressing models, each providing distinct advantages.However, existing literature demonstrates that compressing deep learning models could affect their fairness.Our analysis involves a comprehensive evaluation of pruned, distilled, and quantized language models, which we benchmark across a range of intrinsic and extrinsic metrics for measuring bias in text classification.We also investigate the impact of using multilingual models and evaluation measures.Our findings highlight the significance of considering both the pre-trained model and the chosen compression strategy in developing equitable language technologies.The results also indicate that compression strategies can have an adverse effect on fairness measures.
Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Section

Recent advances in large language models have led to renewed interest in natural language processing in healthcare using the free text of clinical notes.One distinguishing characteristic of clinical notes is their long time span over multiple long documents.The unique structure of clinical notes creates a new design choice: when the context length for a language model predictor is limited, which part of clinical notes should we choose as the input?Existing studies either choose the inputs with domain knowledge or simply truncate them.We propose a framework to analyze the sections with high predictive power.Using MIMIC-III, we show that: 1) predictive power distribution is different between nursing notes and discharge notes and 2) combining different types of notes could improve performance when the context length is large.Our findings suggest that a carefully selected sampling function could enable more efficient information extraction from clinical notes.
Ideology Prediction from Scarce and Biased Supervision: Learn to Disregard the “What” and Focus on the “How”!

We propose a novel supervised learning approach for political ideology prediction (PIP) that is capable of predicting out-of-distribution inputs.This problem is motivated by the fact that manual data-labeling is expensive, while self-reported labels are often scarce and exhibit significant selection bias.We propose a novel statistical model that decomposes the document embeddings into a linear superposition of two vectors; a latent neutral context vector independent of ideology, and a latent position vector aligned with ideology.We train an end-to-end model that has intermediate contextual and position vectors as outputs.At deployment time, our model predicts labels for input documents by exclusively leveraging the predicted position vectors.On two benchmark datasets we show that our model is capable of outputting predictions even when trained with as little as 5% biased data, and is significantly more accurate than the state-of-the-art.Through crowdsourcing we validate the neutrality of contextual vectors, and show that context filtering results in ideological concentration, allowing for prediction on out-of-distribution examples.
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification.While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks.In this study, we introduce a language model called ESCOXLM-R, based on XLM-R large , which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages.The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations.We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets.Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R large on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.