Reviewerly/exHarmony huggingface dataset:
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages

The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages.We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages.An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m.We evaluate Glot500-m on five diverse tasks across these languages.We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline.Our analysis shows that no single factor explains the quality of multilingual LLM representations.Rather, a combination of factors determines quality including corpus size, script, "help" from related languages and the total capacity of the model.Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world's languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures.Code, data and models are available at https://github.com/cisnlp/Glot500.
Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users

Recent work has shown that large-scale annotated datasets are essential for training state-ofthe-art Question Answering (QA) models.Unfortunately, creating this data is expensive and requires a huge amount of annotation work.An alternative and cheaper source of supervision is given by feedback data collected from deployed QA systems.This data can be collected from tens of millions of user with no additional cost, for real-world QA services, e.g., Alexa, Google Home, and etc.The main drawback is the noise affecting feedback on individual examples.Recent literature on QA systems has shown the benefit of training models even with noisy feedback.However, these studies have multiple limitations: (i) they used uniform random noise to simulate feedback responses, which is typically an unrealistic approximation as noise follows specific patterns, depending on target examples and users; and (ii) they do not show how to aggregate feedback for improving training signals.In this paper, we first collect a large scale (16M) QA dataset with real feedback sampled from the QA traffic of a popular Virtual Assistant.Second, we use this data to develop two strategies for filtering unreliable users and thus de-noise feedback: (i) ranking users with an automatic classifier, and (ii) aggregating feedback over similar instances and comparing users between each other.Finally, we train QA models on our filtered feedback data, showing a significant improvement over the state of the art.
TransGEC: Improving Grammatical Error Correction with Translationese

Data augmentation is an effective way to improve model performance of grammatical error correction (GEC).This paper identifies a critical side-effect of GEC data augmentation, which is due to the style discrepancy between the data used in GEC tasks (i.e., texts produced by non-native speakers) and data augmentation (i.e., native texts).To alleviate this issue, we propose to use an alternative data source, translationese (i.e., human-translated texts), as input for GEC data augmentation, which 1) is easier to obtain and usually has better quality than non-native texts, and 2) has a more similar style to non-native texts.Experimental results on the CoNLL14 and BEA19 English, NLPCC18 Chinese, Falko-MERLIN German, and RULEC-GEC Russian GEC benchmarks show that our approach consistently improves correction accuracy over strong baselines.Further analyses reveal that our approach is helpful for overcoming mainstream correction difficulties such as the corrections of frequent words, missing words, and substitution errors.Data, code, models and scripts are freely available at https://github.com/NLP2CT/TransGEC.
On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Loss

Although unsupervised neural machine translation (UNMT) has achieved success in many language pairs, the copying problem, i.e., directly copying some parts of the input sentence as the translation, is common among distant language pairs, especially when low-resource languages are involved.We find this issue is closely related to an unexpected copying behavior during online back-translation (BT).In this work, we propose a simple but effective training schedule that incorporates a language discriminator loss.The loss imposes constraints on the intermediate translation so that the translation is in the desired language.By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.
Entity Coreference and Co-occurrence Aware Argument Mining from Biomedical Literature

Biomedical argument mining (BAM) aims at automatically identifying the argumentative structure in biomedical texts.However, identifying and classifying argumentative relations (AR) between argumentative components (AC) is challenging since it not only needs to understand the semantics of ACs but also need to capture the interactions between them.We argue that entities can serve as bridges that connect different ACs since entities and their mentions convey significant semantic information in biomedical argumentation.For example, it is common that related AC pairs share a common entity.Capturing such entity information can be beneficial for the Relation Identification (RI) task.In order to incorporate this entity information into BAM, we propose an Entity Coreference and Co-occurrence aware Argument Mining (ECCAM) framework based on an edge-oriented graph model for BAM.We evaluate our model on a benchmark dataset and from the experimental results we find that our method improves upon state-ofthe-art methods.
Uncovering Hidden Consequences of Pre-training Objectives in Sequence-to-Sequence Models

Some variants of self-supervised denoising objectives for pre-training encoder-decoder language models have been reported to have a negligible impact on downstream performance.Yet the design of these pre-training objectives leads to behavioural differences that can be uncovered with specific manipulations.We reproduce a recently proposed zero-shot control method and find that it is only successful on a subset of models.To understand what causes the difference in its effectiveness, we perform a set of controlled experiments, varying only the pre-training objective, and find unexpected interactions between the pre-training method and downstream controllability of models after fine-tuning.Our results show that different pretraining objectives have consequences that may not be visible in standard downstream evaluation, but which should be taken into account when developing models with controllability in mind.
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives

Sustaining coherent and engaging narratives requires dialogue or storytelling agents to understand how the personas of speakers or listeners ground the narrative.Specifically, these agents must infer personas of their listeners to produce statements that cater to their interests.They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story.However, personas are diverse and complex: they entail large quantities of rich interconnected world knowledge that is challenging to robustly represent in general narrative systems (e.g., a singer is good at singing, and may have attended conservatoire).In this work, we construct a new large-scale persona commonsense knowledge graph, PEACOK, containing ∼100K human-validated persona facts.Our knowledge graph schematizes five dimensions of persona knowledge identified in previous studies of human interactive behaviours, and distils facts in this schema from both existing commonsense knowledge graphs and largescale pretrained language models.Our analysis indicates that PEACOK contains rich and precise world persona inferences that help downstream systems generate more consistent and engaging narratives.
Towards Efficient Dialogue Processing in the Emergency Response Domain

In this paper we describe the task of adapting NLP models to dialogue processing in the emergency response domain.Our goal is to provide a recipe for building a system that performs dialogue act classification and domain-specific slot tagging while being efficient, flexible and robust.We show that adapter models (Pfeiffer et al., 2020) perform well in the emergency response domain and benefit from additional dialogue context and speaker information.Comparing adapters to standard fine-tuned Transformer models we show that they achieve competitive results and can easily accommodate new tasks without significant memory increase since the base model can be shared between the adapters specializing on different tasks.We also address the problem of scarce annotations in the emergency response domain and evaluate different data augmentation techniques in a low-resource setting.
Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings

Ambiguity is a major obstacle to providing services based on sentence classification.However, because of the structural limitations of the service, there may not be sufficient contextual information to resolve the ambiguity.In this situation, we focus on ambiguity detection so that service design considering ambiguity is possible.We utilize similarity in a semantic space to detect ambiguity in service scenarios 1 and training data.In addition, we apply taskspecific embedding to improve performance.Our results demonstrate that ambiguities and resulting labeling errors in training data or scenarios can be detected.Additionally, we confirm that it can be used to debug services.
A New Direction in Stance Detection: Target-Stance Extraction in the Wild

Stance detection aims to detect the stance toward a corresponding target.Existing works have achieved promising progress on stance detection tasks in which the goal is to predict the stance given both a target and a text.However, they all work under the assumption that the target is known in advance, which is often not the case in the wild.Given a text from social media platforms, the target information is often unknown due to implicit mentions in the source text and it is infeasible to have manual target annotations at a large scale.Therefore, in this paper, we propose a new task Target-Stance Extraction (TSE) that aims to extract the (target, stance) pair from the text.We benchmark the task by proposing a two-stage framework that first identifies the relevant target in the text and then detects the stance given the predicted target and text.Specifically, we first propose two different settings: Target Classification and Target Generation, to identify the potential target from a given text.Then we propose a multitask approach that takes target prediction as the auxiliary task to detect the stance toward the predicted target.We evaluate the proposed framework on both in-target stance detection in which the test target is always seen in the training stage and zero-shot stance detection that needs to detect the stance for the unseen target during the inference stage.The new TSE task can facilitate future research in the field of stance detection.We publicly release our code.
DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation

Despite the recent advances in open-domain dialogue systems, building a reliable evaluation metric is still a challenging problem.Recent studies proposed learnable metrics based on classification models trained to distinguish the correct response.However, neural classifiers are known to make overly confident predictions for examples from unseen distributions.We propose DENSITY, which evaluates a response by utilizing density estimation on the feature space derived from a neural classifier.Our metric measures how likely a response would appear in the distribution of human conversations.Moreover, to improve the performance of DEN-SITY, we utilize contrastive learning to further compress the feature space.Experiments on multiple response evaluation datasets show that DENSITY correlates better with human evaluations than the existing metrics.
Cross-lingual Science Journalism: Select, Simplify and Rewrite Summaries for Non-expert Readers

Automating Cross-lingual Science Journalism (CSJ) aims to generate popular science summaries from English scientific texts for nonexpert readers in their local language.We introduce CSJ as a downstream task of text simplification and cross-lingual scientific summarization to facilitate science journalists' work.We analyze the performance of possible existing solutions as baselines for the CSJ task.Based on these findings, we propose to combine the three components -SELECT, SIMPLIFY and REWRITE (SSR) to produce cross-lingual simplified science summaries for non-expert readers.Our empirical evaluation on the WIKIPEDIA dataset shows that SSR significantly outperforms the baselines for the CSJ task and can serve as a strong baseline for future work.We also perform an ablation study investigating the impact of individual components of SSR.Further, we analyze the performance of SSR on a high-quality, real-world CSJ dataset with human evaluation and in-depth analysis, demonstrating the superior performance of SSR for CSJ.
University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation

Modeling the most likely label when an annotation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves.We present three approaches to modeling the controversiality of a particular text.First, we explicitly represented annotators using annotator embeddings to predict the training signals of each annotator's selections in addition to a majority class label.This method leads to reduction in error relative to models without these features, allowing the overall result to influence the weights of each annotator on the final prediction.In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annotators.Overall, we found that aggregating and explicitly comparing annotators' responses to a static document representation produced highquality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators.
A dynamic programming algorithm for span-based nested named-entity recognition in O(n2)

Span-based nested named-entity recognition (NER) has a cubic-time complexity using a variant of the CYK algorithm.We show that by adding a supplementary structural constraint on the search space, nested NER has a quadratictime complexity, that is the same asymptotic complexity than the non-nested case.The proposed algorithm covers a large part of three standard English benchmarks and delivers comparable experimental results.
Parameter-Efficient Finetuning for Robust Continual Multilingual Learning

We introduce and study the problem of Continual Multilingual Learning (CML) where a previously trained multilingual model is periodically updated using new data arriving in stages.If the new data is present only in a subset of languages, we find that the resulting model shows improved performance only on the languages included in the latest update (and a few closely related languages) while its performance on all the remaining languages degrade significantly.We address this challenge by proposing LAFT-URIEL, a parameter-efficient finetuning strategy which aims to increase the number of languages on which the model improves after an update, while reducing the magnitude of loss in performance for the remaining languages.LAFT-URIEL uses linguistic knowledge to balance overfitting and knowledge sharing across languages, allowing for an additional 25% of task languages to see an improvement in performance after an update, while also reducing the average magnitude of losses on the remaining languages by 78% relative.
Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering

Explainable question answering (XQA) aims to answer a given question and provide an explanation why the answer is selected.Existing XQA methods focus on reasoning on a single knowledge source, e.g., structured knowledge bases, unstructured corpora, etc.However, integrating information from heterogeneous knowledge sources is essential to answer complex questions.In this paper, we propose to leverage question decomposing for heterogeneous knowledge integration, by breaking down a complex question into simpler ones, and selecting the appropriate knowledge source for each sub-question.To facilitate reasoning, we propose a novel two-stage XQA framework, Reasoning over Hierarchical Question Decomposition Tree (RoHT).First, we build the Hierarchical Question Decomposition Tree (HQDT) to understand the semantics of a complex question; then, we conduct probabilistic reasoning over HQDT from root to leaves recursively, to aggregate heterogeneous knowledge at different tree levels and search for a best solution considering the decomposing and answering probabilities.The experiments on complex QA datasets KQA Pro and Musique show that our framework outperforms SOTA methods significantly, demonstrating the effectiveness of leveraging question decomposing for knowledge integration and our RoHT framework.
Structural Contrastive Pretraining for Cross-Lingual Comprehension

Multilingual language models trained using various pre-training tasks like mask language modeling (MLM) have yielded encouraging results on a wide range of downstream tasks.Despite the promising performances, structural knowledge in cross-lingual corpus is less explored in current works, leading to the semantic misalignment.In this paper, we propose a new pre-training task named Structural Contrast Pretraining (SCP) to align the structural words in a parallel sentence, improving the models' linguistic versatility and their capacity to understand representations in multilingual languages.Concretely, SCP treats each structural word in source and target languages as a positive pair.We further propose Crosslingual Momentum Contrast (CL-MoCo) to optimize negative pairs by maintaining a large size of the queue.CL-MoCo extends the original MoCo approach into cross-lingual training and jointly optimizes the source-to-target language and target-to-source language representations in SCP, resulting in a more suitable encoder for cross-lingual transfer learning.We conduct extensive experiments and prove the effectiveness of our resulting model, named XLM-SCP, on three cross-lingual tasks across five datasets such as MLQA, WikiAnn.Our codes are available at https://github.com/nuochenpku/SCP.
RCLN at SemEval-2023 Task 1: Leveraging Stable Diffusion and Image Captions for Visual WSD

This paper describes the participation of the RCLN team at the Visual Word Sense Disambiguation task at SemEval 2023.The participation was focused on the use of CLIP as a base model for the matching between text and images with additional information coming from captions generated from images and the generation of images from the prompt text using Stable Diffusion.The results we obtained are not particularly good, but interestingly enough, we were able to improve over the CLIP baseline in Italian by recurring simply to the generated images.
Explanation Regeneration via Information Bottleneck

Explaining the black-box predictions of NLP models naturally and accurately is an important open problem in natural language generation.These free-text explanations are expected to contain sufficient and carefully-selected evidence to form supportive arguments for predictions.Thanks to the superior generative capacity of large pretrained language models (PLM), recent work built on prompt engineering enables explanations generated without specific training.However, explanations generated through single-pass prompting often lack sufficiency and conciseness, due to the prompt complexity and hallucination issues.To discard the dross and take the essence of current PLM's results, we propose to produce sufficient and concise explanations via the information bottleneck (EIB) theory.EIB regenerates explanations by polishing the single-pass output of PLM but retaining the information that supports the contents being explained by balancing two information bottleneck objectives.Experiments on two different tasks verify the effectiveness of EIB through automatic evaluation and thoroughly-conducted human evaluation.
Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity.Moreover, methods and labels in dialogue evaluation are not fully standardized, especially for opendomain chats, with a lack of work to compare and assess the validity of those approaches.The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it.Thus, a dimensional evaluation of chat-oriented opendomain dialogue systems that reliably measures several aspects of dialogue capabilities is desired.This paper presents a novel human evaluation method to estimate the rates of many dialogue system behaviors.Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches.The analysis demonstrates that our behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.
Toward Human-Like Evaluation for Natural Language Generation with Error Analysis

The pretrained language model (PLM) based metrics have been successfully used in evaluating language generation tasks.Recent studies of the human evaluation community show that considering both major errors (e.g.mistranslated tokens) and minor errors (e.g.imperfections in fluency) can produce high-quality judgments.This inspires us to approach the final goal of the automatic metrics (human-like evaluations) by fine-grained error analysis.In this paper, we argue that the ability to estimate sentence confidence is the tip of the iceberg for PLM-based metrics.And it can be used to refine the generated sentence toward higher confidence and more reference-grounded, where the costs of refining and approaching reference are used to determine the major and minor errors, respectively.To this end, we take BARTScore as the testbed and present an innovative solution to marry the unexploited sentence refining capacity of BARTScore and human-like error analysis, where the final score consists of both the evaluations of major and minor errors.Experiments show that our solution consistently improves BARTScore, outperforming top-scoring metrics in 19/25 test settings.Analyses demonstrate our method robustly and efficiently approaches human-like evaluations, enjoying better interpretability.Our code and scripts will be publicly released in https: //github.com/Coldmist-Lu/ErrorAnalysis_NLGEvaluation.