Reviewerly/exHarmony huggingface dataset:
Building an Ensemble of Transformer Models for Arabic Dialect Classification and Sentiment Analysis

In this paper, we describe the approaches we developed for the Nuanced Arabic Dialect Identification (NADI) 2022 shared task, which consists of two subtasks: the identification of country-level Arabic dialects and sentiment analysis.Our team, UniManc, developed approaches to the two subtasks which are underpinned by the same model: a pre-trained MAR-BERT language model.For Subtask 1, we applied undersampling to create versions of the training data with a balanced distribution across classes.For Subtask 2, we further trained the original MARBERT model for the masked language modelling objective using a NADIprovided dataset of unlabelled Arabic tweets.For each of the subtasks, a MARBERT model was fine-tuned for sequence classification, using different values for hyperparameters such as seed and learning rate.This resulted in multiple model variants, which formed the basis of an ensemble model for each subtask.Based on the official NADI evaluation, our ensemble model obtained a macro-F1-score of 26.863, ranking second overall in the first subtask.In the second subtask, our ensemble model also ranked second, obtaining a macro-F1-PN score (macro-averaged F1-score over the Positive and Negative classes) of 73.544.
Improving Machine Translation with Phrase Pair Injection and Corpus Filtering

In this paper, we show that the combination of Phrase Pair Injection and Corpus Filtering boosts the performance of Neural Machine Translation (NMT) systems.We extract parallel phrases and sentences from the pseudoparallel corpus and augment it with the parallel corpus to train the NMT models.With the proposed approach, we observe an improvement in the Machine Translation (MT) system for 3 low-resource language pairs, Hindi-Marathi, English-Marathi, and English-Pashto, and 6 translation directions by up to 2.7 BLEU points, on the FLORES test data.These BLEU score improvements are over the models trained using the whole pseudo-parallel corpus augmented with the parallel corpus.
NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation.Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which has not been studied so far.We analyze these measures in the common framework of multilingual NMT, releasing the NMTSCORE library.Compared to baselines such as sentence embeddings, translation-based measures prove competitive in paraphrase identification and are more robust against adversarial or multilingual input, especially if proper normalization is applied.When used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages, translation-based measures show a relatively high correlation to human judgments.
DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages

We introduce DivEMT, the first publicly available post-editing study of Neural Machine Translation (NMT) over a typologically diverse set of target languages.Using a strictly controlled setup, 18 professional translators were instructed to translate or post-edit the same set of English documents into Arabic, Dutch, Italian, Turkish, Ukrainian, and Vietnamese.During the process, their edits, keystrokes, editing times and pauses were recorded, enabling an in-depth, cross-lingual evaluation of NMT quality and post-editing effectiveness.Using this new dataset, we assess the impact of two state-of-the-art NMT systems, Google Translate and the multilingual mBART-50 model, on translation productivity.We find that post-editing is consistently faster than translation from scratch.However, the magnitude of productivity gains varies widely across systems and languages, highlighting major disparities in post-editing effectiveness for languages at different degrees of typological relatedness to English, even when controlling for system architecture and training data size.We publicly release the complete dataset 1 including all collected behavioral data, to foster new research on the translation capabilities of NMT systems for typologically diverse languages.
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning

Contrastive learning has been demonstrated to be effective in enhancing pre-trained language models (PLMs) to derive superior universal sentence embeddings.However, existing contrastive methods still have two limitations.Firstly, previous works may acquire poor performance under domain shift settings, thus hindering the application of sentence representations in practice.We attribute this low performance to the over-parameterization of PLMs with millions of parameters.To alleviate it, we propose PromCSE (Prompt-based Contrastive Learning for Sentence Embeddings), which only trains small-scale Soft Prompt (i.e., a set of trainable vectors) while keeping PLMs fixed.Secondly, the commonly used NT-Xent loss function of contrastive learning does not fully exploit hard negatives in supervised learning settings.To this end, we propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power, inspired by the connection between the NT-Xent loss and the Energy-based Learning paradigm.Empirical results on seven standard semantic textual similarity (STS) tasks and a domain-shifted STS task both show the effectiveness of our method compared with the current state-of-theart sentence embedding models. 1
SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings

Adding interpretability to word embeddings represents an area of active research in text representation.Recent work has explored the potential of embedding words via so-called polar dimensions (e.g.good vs. bad, correct vs. wrong).Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp.Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words.To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables wordsense aware interpretability for pre-trained contextual word embeddings.The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks.Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.
CARE: Causality Reasoning for Empathetic Responses by Conditional Graph Generation

Recent approaches to empathetic response generation incorporate emotion causalities to enhance comprehension of both the user's feelings and experiences.However, these approaches suffer from two critical issues.First, they only consider causalities between the user's emotion and the user's experiences, and ignore those between the user's experiences.Second, they neglect interdependence among causalities and reason them independently.To solve the above problems, we expect to reason all plausible causalities interdependently and simultaneously, given the user's emotion, dialogue history, and future dialogue content.Then, we infuse these causalities into response generation for empathetic responses.Specifically, we design a new model, i.e., the Conditional Variational Graph Auto-Encoder (CV-GAE), for the causality reasoning, and adopt a multi-source attention mechanism in the decoder for the causality infusion.We name the whole framework as CARE 1 , abbreviated for CAusality Reasoning for Empathetic conversation.Experimental results indicate that our method achieves state-of-the-art performance.
Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI

Detecting "toxic" language in internet content is a pressing social and technical challenge.In this work, we focus on PERSPECTIVE from Jigsaw, a state-of-the-art tool that promises to score the "toxicity" of text, with a recent model update that claims impressive results (Lees et al., 2022).We seek to challenge certain normative claims about toxic language by proposing a new benchmark, Selected Adversarial SemanticS, or SASS.We evaluate PER-SPECTIVE on SASS, and compare to low-effort alternatives, like zero-shot and few-shot GPT-3 prompt models, in binary classification settings.We find that PERSPECTIVE exhibits troubling shortcomings across a number of our toxicity categories.SASS provides a new tool for evaluating performance on previously undetected toxic language that avoids common normative pitfalls.Our work leads us to emphasize the importance of questioning assumptions made by tools already in deployment for toxicity detection in order to anticipate and prevent disparate harms.
SCROLLS: Standardized CompaRison Over Long Language Sequences

NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild.We introduce SCROLLS, a suite of tasks that require reasoning over long texts.We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input.SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment.Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS.We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods. 1
Error Analysis of ToTTo Table-to-Text Neural NLG Models

We report error analysis of outputs from four Table-to-Text generation models fine-tuned on ToTTo, an open-domain English language dataset.We carried out a manual error annotation of a subset of outputs (a total of 3,016 sentences) belonging to the topic of Politics generated by these four models.Our error annotation focused on eight categories of errors.The error analysis shows that more than 46% of sentences from each of the four models have been errorfree.It uncovered some of the specific classes of errors; for example, WORD errors (mostly verbs and prepositions) are the dominant errors in all four models and are the most complex ones among other errors.NAME (mostly nouns) and NUMBER errors are slightly higher in two of the GeM benchmark models, whereas DATE_DIMENSION and OTHER categories of errors are more common in our Table-to-Text model.This in-depth error analysis is currently guiding us in improving our Tableto-Text model.Input Table data from ToTTo (includes metadata such as Title, Highlighted cells in yellow and their headers)
Contrastive Demonstration Tuning for Pre-trained Language Models

Pretrained language models can be effectively stimulated by textual prompts or demonstrations, especially in low-data scenarios.Recent works have focused on automatically searching discrete or continuous prompts or optimized verbalizers, yet studies for the demonstration are still limited.Concretely, the demonstration examples are crucial for an excellent final performance of prompt-tuning.In this paper, we propose a novel pluggable, extensible, and efficient approach named contrastive demonstration tuning, which is free of demonstration sampling.Furthermore, the proposed approach can be: (i) Plugged into any previous prompttuning approaches; (ii) Extended to widespread classification tasks with a large number of categories.Experimental results on 16 datasets illustrate that our method integrated with previous approaches LM-BFF and P-tuning can yield better performance 1 .
Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model

In this work, we present a model for documentgrounded response generation in dialog that is decomposed into two components according to Bayes' theorem.One component is a traditional ungrounded response generation model and the other component models the reconstruction of the grounding document based on the dialog context and generated response.We propose different approximate decoding schemes and evaluate our approach on multiple opendomain and task-oriented document-grounded dialog datasets.Our experiments show that the model is more factual in terms of automatic factuality metrics than the baseline model.Furthermore, we outline how introducing scaling factors between the components allows for controlling the tradeoff between factuality and fluency in the model output.Finally, we compare our approach to a recently proposed method to control factuality in grounded dialog, CTRL (Rashkin et al., 2021), and show that both approaches can be combined to achieve additional improvements.
Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation

Radiology report generation systems have the potential to reduce the workload of radiologists by automatically describing the findings in medical images.To broaden the application of the report generation system, the system should generate reports that are not only factually accurate but also chronologically consistent, describing images that are presented in time order, that is, the correct order.We employ a planning-based radiology report generation system that generates the overall structure of reports as "plans" prior to generating reports that are accurate and consistent in order.Additionally, we propose a novel reinforcement learning and inference method, Coordinated Planning (CoPlan), that includes a content planner and a text generator to train and infer in a coordinated manner to alleviate the cascading of errors that are often inherent in planning-based models.We conducted experiments with single-phase diagnostic reports in which the factual accuracy is critical and multi-phase diagnostic reports in which the description order is critical.Our proposed CoPlan improves the content order score by 5.1 pt in time series critical scenarios and the clinical factual accuracy F-score by 9.1 pt in time series irrelevant scenarios, compared those of the baseline models without CoPlan.
When Language Model Meets Private Library

With the rapid development of pre-training techniques, a number of language models have been pre-trained on large-scale code corpora and perform well in code generation.In this paper, we investigate how to equip pre-trained language models with the ability of code generation for private libraries.In practice, it is common for programmers to write code using private libraries.However, this is a challenge for language models since they have never seen private APIs during training.Motivated by the fact that private libraries usually come with elaborate API documentation, we propose a novel framework with two modules: the APIRetriever finds useful APIs, and then the APICoder generates code using these APIs.For APIRetriever, we present a dense retrieval system and also design a friendly interaction to involve uses.For APICoder, we can directly use off-the-shelf language models, or continually pre-train the base model on a code corpus containing API information.Both modules are trained with data from public libraries and can be generalized to private ones.Furthermore, we craft three benchmarks for private libraries, named Torch-DataEval, MonkeyEval, and BeatNumEval.Experimental results demonstrate the impressive performance of our framework. 1.
R-TeaFor: Regularized Teacher-Forcing for Abstractive Summarization

Teacher-forcing is widely used in training sequence generation models to improve sampling efficiency and to stabilize training.However, teacher-forcing is vulnerable to the exposure bias problem.Previous works have attempted to address exposure bias by modifying the training data to simulate model-generated results.Nevertheless, they do not consider the pairwise relationship between the original training data and the modified ones, which provides more information during training.Hence, we propose Regularized Teacher-Forcing (R-TeaFor) to utilize this relationship for better regularization.Empirically, our experiments show that R-TeaFor outperforms previous summarization state-of-the-art models, and the results can be generalized to different pre-trained models.
Abstractive Summarization of Dutch Court Verdicts Using Sequence-to-sequence Models

With the legal sector embracing digitization, the increasing availability of information has led to a need for systems that can automatically summarize legal documents.Most existing research on legal text summarization has so far focused on extractive models, which can result in awkward summaries, as sentences in legal documents can be very long and detailed.In this study, we apply two abstractive summarization models on a Dutch legal domain dataset.The results show that existing models transfer quite well across domains and languages: the ROUGE scores of our experiments are comparable to state-of-the-art studies on English news article texts.Examining one of the models showed the capability of rewriting long legal sentences to much shorter ones, using mostly vocabulary from the source document.Human evaluation shows that for both models hand-made summaries are still perceived as more relevant and readable, and automatic summaries do not always capture elements such as background, considerations and judgement.Still, generated summaries are valuable if only a keyword summary or no summary at all is present.
Localizing Syntactic Composition with Left-Corner Recurrent Neural Network Grammars

In computational neurolinguistics, it has been demonstrated that hierarchical models such as recurrent neural network grammars (RNNGs), which jointly generate word sequences and their syntactic structures via the syntactic composition, better explained human brain activity than sequential models such as long short-term memory networks (LSTMs). However, the vanilla RNNG has employed the top-down parsing strategy, which has been pointed out in the psycholinguistics literature as suboptimal especially for head-final/left-branching languages, and alternatively the left-corner parsing strategy has been proposed as the psychologically plausible parsing strategy. In this article, building on this line of inquiry, we investigate not only whether hierarchical models like RNNGs better explain human brain activity than sequential models like LSTMs, but also which parsing strategy is more neurobiologically plausible, by developing a novel fMRI corpus where participants read newspaper articles in a head-final/left-branching language, namely Japanese, through the naturalistic fMRI experiment. The results revealed that left-corner RNNGs outperformed both LSTMs and top-down RNNGs in the left inferior frontal and temporal-parietal regions, suggesting that there are certain brain regions that localize the syntactic composition with the left-corner parsing strategy.
A Large-Scale Mobile Traffic Dataset For Mobile Application Identification

Abstract With Internet access shifting from desktop-driven to mobile-driven, application-level mobile traffic identification has become a research hotspot. Although considerable progress has been made in this research field, two obstacles are hindering its further development. Firstly, there is a lack of sharable labeled mobile traffic datasets. Although it is easy to capture mobile traffic, labeling traffic at the application level is non-trivial. Besides, researchers usually hold a conservative attitude toward publishing their datasets for privacy concerns. Secondly, most of the datasets used by existing studies are inadequate to evaluate the proposed methods, since they usually have the problems of inaccurate labels, small scale and simple collection configurations. To tackle these two obstacles, a mobile traffic collection is carried out in this paper. The collected traffic has the advantages of large-scale data size, accurate application-level labels and diverse collection configurations. Then, the collected traffic is anonymized carefully to make it public. Several mobile traffic identification methods are compared based on our anonymized dataset, which proves the applicability of our dataset.
Chapter 3. Architectural Patterns for Neuro-Symbolic AI

We propose a set of architectural patterns that describe a large variety of neuro-symbolic systems. As in other areas of computer science (knowledge engineering, software engineering, ontology engineering, process mining and others), such design patterns provide a unified vocabulary to describe a large variety of systems, help to systematise the literature, clarify which combinations of techniques serve which purposes, and encourage re-use of software components. We have validated our set of compositional design patterns against a large body of recent literature, and we apply them to a number of systems described in the different sections of this volume. Sections 1.1-1.7 of this chapter are based on our earlier publications [1] and [2], the new contributions of this chapter are in section 1.5 and 1.6.
Recognising permuted Demidenko matrices

We solve the recognition problem for permuted Demidenko matrices. This problem is relevant in the context of hard combinatorial optimization problems becoming tractable if the input is a Demidenko matrix. For example the TSP is polynomially solvable for Demidenko distance matrices. In the TSP context we look for a renumbering of the cities resulting in a Demidenko distance matrix, thus in a polynomially solvable case. We can find such a renumbering of n cities in O(n4) time, if it exists.