Reviewerly/exHarmony huggingface dataset:
Maknuune: A Large Open Palestinian Arabic Lexicon

We present Maknuune, a large open lexicon for the Palestinian Arabic dialect.Maknuune has over 36K entries from 17K lemmas, and 3.7K roots.All entries include diacritized Arabic orthography, phonological transcription and English glosses.Some entries are enriched with additional information such as broken plurals and templatic feminine forms, associated phrases and collocations, Standard Arabic glosses, and examples or notes on grammar, usage, or location of collected entry.
SNU-Causality Lab @ Causal News Corpus 2022: Detecting Causality by Data Augmentation via Part-of-Speech tagging

Finding causal relations in texts has been a challenge since it requires methods ranging from defining event ontologies to developing proper algorithmic approaches.In this paper, we developed a framework which classifies whether a given sentence contains a causal event.As our approach, we exploited an external corpus that has causal labels to overcome the small size of the original corpus (Causal News Corpus) provided by task organizers.Further, we employed a data augmentation technique utilizing Part-Of-Speech (POS) based on our observation that some parts of speech are more (or less) relevant to causality.Our approach especially improved the recall of detecting causal events in sentences.
Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions

Wikipedia is a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English.However, for many downstream NLP tasks, it is important to understand the degree to which these corpora reflect representative contributions of native speakers.In particular, many entries in a given language may be translated from other languages or produced through other automated mechanisms.Language models built using corpora like Wikipedia can embed history, culture, bias, stereotypes, politics, and more, but it is important to understand whose views are actually being represented.In this paper, we present a case study focusing specifically on differences among the Arabic Wikipedia editions (Modern Standard Arabic, Egyptian, and Moroccan).In particular, we document issues in the Egyptian Arabic Wikipedia with automatic creation/generation and translation of content pages from English without human supervision.These issues could substantially affect the performance and accuracy of Large Language Models (LLMs) trained from these corpora, producing models that lack the cultural richness and meaningful representation of native speakers.Fortunately, the metadata maintained by Wikipedia provides visibility into these issues, but unfortunately, this is not the case for all corpora used to train LLMs.
Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling

Multi-turn dialogue modeling as a challenging branch of natural language understanding (NLU), aims to build representations for machines to understand human dialogues, which provides a solid foundation for multiple downstream tasks.Recent studies of dialogue modeling commonly employ pre-trained language models (PrLMs) to encode the dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues.Therefore, we propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder, which explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks.Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.The official implementation of BiDeN is available at https://github.com/EricLee8/BiDeN.
Towards Learning Arabic Morphophonology

One core challenge facing morphological inflection systems is capturing language-specific morphophonological changes.This is particularly true of languages like Arabic which are morphologically complex.In this paper, we learn explicit morphophonological rules from morphologically annotated Egyptian Arabic and corresponding surface forms.These rules are human-interpretable, capture known morphophonological phenomena in the language, and are generalizable to unseen forms.
PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models

A wide range of NLP tasks benefit from the fine-tuning of pretrained language models (PLMs).However, a number of redundant parameters which contribute less to the downstream task are observed in a directly fine-tuned model.We think the gap between pretraining and downstream tasks hinders the training of these redundant parameters, and results in a suboptimal performance of the overall model.In this paper, we present PATS (Perturbation According To Sensitivity), a noisy training mechanism which considers each parameter's importance in the downstream task to help fine-tune PLMs.The main idea of PATS is to add bigger noise to parameters with lower sensitivity and vice versa, in order to activate more parameters' contributions to downstream tasks without affecting the sensitive ones much.Extensive experiments conducted on different tasks of the GLUE benchmark show PATS can consistently empower the fine-tuning of different sizes of PLMs, and the parameters in the well-performing models always have more concentrated distributions of sensitivities, which experimentally proves the effectiveness of our method.
Weakly and Semi-Supervised Learning for Arabic Text Classification using Monodialectal Language Models

The lack of resources such as annotated datasets and tools for low-resource languages is a significant obstacle to the advancement of Natural Language Processing (NLP) applications targeting users who speak these languages.Although learning techniques such as semisupervised and weakly supervised learning are effective in text classification cases where annotated data is limited, they are still not widely investigated in many languages due to the sparsity of data altogether, both labeled and unlabeled.In this study, we deploy both weakly, and semi-supervised learning approaches for text classification in low-resource languages and address the underlying limitations that can hinder the effectiveness of these techniques.To that end, we propose a suite of languageagnostic techniques for large-scale data collection, automatic data annotation, and language model training in scenarios where resources are scarce.Specifically, we propose a novel data collection pipeline for under-represented languages, or dialects, that is language and task agnostic and of sufficient size for training a language model capable of achieving competitive results on common NLP tasks, as our experiments show.The models will be shared with the research community 1 .
Machine translation impact in E-commerce multilingual search

Previous work suggests that performance of cross-lingual information retrieval correlates highly with the quality of Machine Translation. However, there may be a threshold beyond which improving query translation quality yields little or no benefit to further improve the retrieval performance. This threshold may depend upon multiple factors including the source and target languages, the existing MT system quality and the search pipeline. In order to identify the benefit of improving an MT system for a given search pipeline, we investigate the sensitivity of retrieval quality to the presence of different levels of MT quality using experimental datasets collected from actual traffic. We systematically improve the performance of our MT systems quality on language pairs as measured by MT evaluation metrics including Bleu and Chrf to determine their impact on search precision metrics and extract signals that help to guide the improvement strategies. Using this information we develop techniques to compare query translations for multiple language pairs and identify the most promising language pairs to invest and improve.
Deploying Unified BERT Moderation Model for E-Commerce Reviews

Moderation of user-generated e-commerce content has become crucial due to the large and diverse user base on the platforms.Product reviews and ratings have become an integral part of the shopping experience to build trust among users.Due to the high volume of reviews generated on a vast catalog of products, manual moderation is infeasible, making machine moderation a necessity.In this work, we described our deployed system and models for automated moderation of user-generated content.At the heart of our approach, we outline several rejection reasons for review & rating moderation and explore a unified BERT model to moderate them.We convey the importance of product vertical embeddings for the relevancy of the review for a given product and highlight the advantages of pre-training the BERT models with monolingual data to cope with the domain gap in the absence of huge labelled datasets.We observe a 4.78% F1 increase with less labelled data and a 2.57% increase in F1 score on the review data compared to the publicly available BERT-based models.Our best model In-House-BERT-vertical sends only 5.89% of total reviews to manual moderation and has been deployed in production serving live traffic for millions of users.
You Only Need One Model for Open-domain Question Answering

Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank passages with a separate reranker model and generate an answer using another reader model.Despite performing related tasks, the models have separate parameters and are weaklycoupled during training.We propose casting the retriever and the reranker as internal passage-wise attention mechanisms applied sequentially within the transformer architecture and feeding computed representations to the reader, with the hidden representations progressively refined at each stage.This allows us to use a single question answering model trained end-to-end, which is a more efficient use of model capacity and also leads to better gradient flow.We present a pre-training method to effectively train this architecture and evaluate our model on the Natural Questions and TriviaQA open datasets.For a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores.
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models

Conceptual knowledge is fundamental to human cognition and knowledge bases.However, existing knowledge probing works only focus on evaluating factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge.Since conceptual knowledge often appears as implicit commonsense behind texts, designing probes for conceptual knowledge is hard.Inspired by knowledge representation schemata, we comprehensively evaluate conceptual knowledge of PLMs by designing three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts, respectively.For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark.Extensive experiments on different sizes and types of PLMs show that existing PLMs systematically lack conceptual knowledge and suffer from various spurious correlations.We believe this is a critical bottleneck for realizing human-like cognition in PLMs.COPEN and our codes are publicly released at https: //github.com/THU-KEG/COPEN.
McQueen: a Benchmark for Multimodal Conversational Query Rewrite

The task of query rewrite aims to convert an in-context query to its fully-specified version where ellipsis and coreference are completed and referred-back according to the history context.Although much progress has been made, less efforts have been paid to real scenario conversations that involve drawing information from more than one modalities.In this paper, we propose the task of multimodal conversational query rewrite (McQR), which performs query rewrite under the multimodal visual conversation setting.We collect a largescale dataset named McQueen based on manual annotation, which contains 15k visual conversations and over 80k queries where each one is associated with a fully-specified rewrite version.In addition, for entities appearing in the rewrite, we provide the corresponding image box annotation.We then use the McQueen dataset to benchmark a state-of-the-art method for effectively tackling the McQR task, which is based on a multimodal pre-trained model with pointer generator.Extensive experiments are performed to demonstrate the effectiveness of our model on this task 1 .
A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsing

Foundational Hebrew NLP tasks such as segmentation, tagging and parsing, have relied to date on various versions of the Hebrew Treebank (HTB, Sima'an et al. 2001).However, the data in HTB, a single-source newswire corpus, is now over 30 years old, and does not cover many aspects of contemporary Hebrew on the web.This paper presents a new, freely available UD treebank of Hebrew stratified from a range of topics selected from Hebrew Wikipedia.In addition to introducing the corpus and evaluating the quality of its annotations, we deploy automatic validation tools based on grew (Guillaume, 2021), and conduct the first crossdomain parsing experiments in Hebrew.We obtain new state-of-the-art (SOTA) results on UD NLP tasks, using a combination of the latest language modelling and some incremental improvements to existing transformer based approaches.We also release a new version of the UD HTB matching annotation scheme updates from our new corpus.
Intermediate Entity-based Sparse Interpretable Representation Learning

Interpretable entity representations (IERs) are sparse embeddings that are "human-readable" in that dimensions correspond to fine-grained entity types and values are predicted probabilities that a given entity is of the corresponding type.These methods perform well in zeroshot and low supervision settings.Compared to standard dense neural embeddings, such interpretable representations may permit analysis and debugging.However, while fine-tuning sparse, interpretable representations improves accuracy on downstream tasks, it destroys the semantics of the dimensions which were enforced in pre-training.Can we maintain the interpretable semantics afforded by IERs while improving predictive performance on downstream tasks?Toward this end, we propose Intermediate enTity-based Sparse Interpretable Representation Learning (ItsIRL).ItsIRL realizes improved performance over prior IERs on biomedical tasks, while maintaining "interpretability" generally and their ability to support model debugging specifically.The latter is enabled in part by the ability to perform "counterfactual" fine-grained entity type manipulation, which we explore in this work.Finally, we propose a method to construct entity type based class prototypes for revealing global semantic properties of classes learned by our model.
Navigating Connected Memories with a Task-oriented Dialog System

Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collections.Despite conversation being an intuitive humancomputer interface, current efforts focus mostly on single-shot natural language based media retrieval to aid users query their media and re-live their memories.This severely limits the search functionality as users can neither ask followup queries nor obtain information without first formulating a single-turn query.In this work, we propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multiturn, interactive conversation.Towards this, we collect a new task-oriented dialog dataset COMET, which contains 11.5k user↔assistant dialogs (totalling 103k utterances), grounded in simulated personal memory graphs.We employ a resource-efficient, two-phase data collection pipeline that uses: (1) a novel multimodal dialog simulator that generates synthetic dialog flows grounded in memory graphs, and, (2) manual paraphrasing to obtain natural language utterances.We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines, in order to highlight the multimodal challenges captured by our dataset 1 .
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation

Linguistic analysis of language models is one of the ways to explain and describe their reasoning, weaknesses, and limitations.In the probing part of the model interpretability research, studies concern individual languages as well as individual linguistic structures.The question arises: are the detected regularities linguistically coherent, or on the contrary, do they dissonate at the typological scale?Moreover, the majority of studies address the inherent set of languages and linguistic structures, leaving the actual typological diversity knowledge out of scope.In this paper, we present and apply the GUI-assisted framework allowing us to easily probe a massive number of languages for all the morphosyntactic features present in the Universal Dependencies data.We show that reflecting the anglo-centric trend in NLP over the past years, most of the regularities revealed in the mBERT model are typical for the western-European languages.Our framework can be integrated with the existing probing toolboxes, model cards, and leaderboards, allowing practitioners to use and share their standard probing methods to interpret multilingual models.Thus we propose a toolkit to systematize the multilingual flaws in multilingual models, providing a reproducible experimental setup for 104 languages and 80 morphosyntactic features.GitHub
Sequential Topic Selection Model with Latent Variable for Topic-Grounded Dialogue

Recently, topic-grounded dialogue system has attracted significant attention due to its effectiveness in predicting the next topic to yield better responses via the historical context and given topic sequence.However, almost all existing topic prediction solutions focus on only the current conversation and corresponding topic sequence to predict the next conversation topic, without exploiting other topic-guided conversations which may contain relevant topictransitions to current conversation.To address the problem, in this paper we propose a novel approach, named Sequential Global Topic Attention (SGTA) to exploit topic transition over all conversations in a subtle way for better modeling post-to-response topic-transition and guiding the response generation to the current conversation.Specifically, we introduce a latent space modeled as a Multivariate Skew-Normal distribution with hybrid kernel functions to flexibly integrate the global-level information with sequence-level information, and predict the topic based on the distribution sampling results.We also leverage a topic-aware prior-posterior approach for secondary selection of predicted topics, which is utilized to optimize the response generation task.Extensive experiments demonstrate that our model outperforms competitive baselines on prediction and generation tasks.
Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering

Extractive question answering (QA) models tend to exploit spurious correlations to make predictions when a training set has unintended biases.This tendency results in models not being generalizable to examples where the correlations do not hold.Determining the spurious correlations QA models can exploit is crucial in building generalizable QA models in real-world applications; moreover, a method needs to be developed that prevents these models from learning the spurious correlations even when a training set is biased.In this study, we discovered that the relative position of an answer, which is defined as the relative distance from an answer span to the closest questioncontext overlap word, can be exploited by QA models as superficial cues for making predictions.Specifically, we find that when the relative positions in a training set are biased, the performance on examples with relative positions unseen during training is significantly degraded.To mitigate the performance degradation for unseen relative positions, we propose an ensemble-based debiasing method that does not require prior knowledge about the distribution of relative positions.We demonstrate that the proposed method mitigates the models' reliance on relative positions using the biased and full SQuAD dataset.We hope that this study can help enhance the generalization ability of QA models in real-world applications. 1
Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification

Pre-trained seq2seq models excel at graph semantic parsing with rich annotated data, but generalize worse to out-of-distribution (OOD) and long-tail examples.In comparison, symbolic parsers under-perform on populationlevel metrics, but exhibit unique strength in OOD and tail generalization.In this work, we study compositionality-aware approach to neural-symbolic inference informed by model confidence, performing fine-grained neuralsymbolic reasoning at subgraph level (i.e., nodes and edges) and precisely targeting subgraph components with high uncertainty in the neural parser.As a result, the method combines the distinct strength of the neural and symbolic approaches in capturing different aspects of the graph prediction, leading to well-rounded generalization performance both across domains and in the tail.We empirically investigate the approach in the English Resource Grammar (ERG) parsing problem on a diverse suite of standard in-domain and seven OOD corpora.Our approach leads to 35.26% and 35.60% error reduction in aggregated SMATCH score over neural and symbolic approaches respectively, and 14% absolute accuracy gain in key tail linguistic categories over the neural model, outperforming prior state-of-art methods that do not account for compositionality or uncertainty.
PAIR: Prompt-Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing

Reflections are a core verbal skill used by mental health counselors to express understanding and acknowledgement of the client's experience and concerns.In this paper, we propose a system for the automatic evaluation of counselor reflections.Specifically, our system takes as input one dialog turn containing a client prompt likely leading to a reflection and a counselor response to it, and outputs a numeric score indicating the quality of the reflection made by the counselor.We compile a dataset consisting of reflections portraying different levels of reflective listening skills, and propose Prompt-Aware margIn Ranking (PAIR), a novel framework for reflection scoring that contrasts positive and negative prompt and response pairs using adhoc multi-gap and prompt-aware margin ranking losses.Through empirical evaluations and deployment of our system in a real-life educational environment, we show that our scoring model outperforms several baselines on different metrics, and can be used to provide useful feedback to counseling trainees.