Reviewerly/exHarmony huggingface dataset:
Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization

Models trained via empirical risk minimization (ERM) are revealed to easily rely on spurious correlations, resulting in poor model generalization.Group distributionally robust optimization (group DRO) can alleviate this problem by minimizing the worst-case loss over pre-defined groups.While promising, in practice factors like expensive annotations and privacy preclude the availability of group labels.More crucially, when taking a closer look at the failure modes of out-of-distribution generalization, the typical procedure of reweighting in group DRO loses efficiency.Hinged on the limitations, in this work, we reformulate the group DRO framework by proposing Q-Diversity.Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.Furthermore, a novel mixing strategy across groups is presented to diversify the under-represented groups.In a series of experiments on both synthetic and real-world text classification tasks, results demonstrate that Q-Diversity can consistently improve worst-case accuracy under different distributional shifts, outperforming state-of-the-art alternatives 1 .
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF++ improvement in very low resource language pairs. We perform an extensive analysis of the learned MoE routing to better understand the impact of our regularization methods and how we can improve them.
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models

Pre-trained encoder-only and sequence-tosequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive.We explore recipes to improve pre-training efficiency by initializing one model from the other.( 1) Extracting the encoder from a seq2seq model, we show it under-performs a Masked Language Modeling (MLM) encoder, particularly on sequence labeling tasks.Variations of masking during seq2seq training, reducing the decoder size, and continuing with a small amount of MLM training do not close the gap.(2) Conversely, using an encoder to warm-start seq2seq training, we show that by unfreezing the encoder partway through training, we can match task performance of a from-scratch seq2seq model.Overall, this two-stage approach is an efficient recipe to obtain both a multilingual encoder and a seq2seq model, matching the performance of training each model from scratch while reducing the total compute cost by 27%.
White-Box Multi-Objective Adversarial Attack on Dialogue Generation

Pre-trained transformers are popular in stateof-the-art dialogue generation (DG) systems.Such language models are, however, vulnerable to various adversarial samples as studied in traditional tasks such as text classification, which inspires our curiosity about their robustness in DG systems.One main challenge of attacking DG models is that perturbations on the current sentence can hardly degrade the response accuracy because the unchanged chat histories are also considered for decision-making.Instead of merely pursuing pitfalls of performance metrics such as BLEU, ROUGE, we observe that crafting adversarial samples to force longer generation outputs benefits attack effectiveness-the generated responses are typically irrelevant, lengthy, and repetitive.To this end, we propose a white-box multi-objective attack method called DGSlow.Specifically, DGSlow balances two objectives-generation accuracy and length, via a gradient-based multiobjective optimizer and applies an adaptive searching mechanism to iteratively craft adversarial samples with only a few modifications.Comprehensive experiments 1 on four benchmark datasets demonstrate that DGSlow could significantly degrade state-of-the-art DG models with a higher success rate than traditional accuracy-based methods.Besides, our crafted sentences also exhibit strong transferability in attacking other models.
The Art of Prompting: Event Detection based on Type Specific Prompts

We compare various forms of prompts to represent event types and develop a unified framework to incorporate the event type specific prompts for supervised, few-shot, and zeroshot event detection.The experimental results demonstrate that a well-defined and comprehensive event type prompt can significantly improve event detection performance, especially when the annotated data is scarce (fewshot event detection) or not available (zero-shot event detection).By leveraging the semantics of event types, our unified framework shows up to 22.2% F-score gain over the previous stateof-the-art baselines 1 .
Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation

Research on Korean grammatical error correction (GEC) is limited, compared to other major languages such as English.We attribute this problematic circumstance to the lack of a carefully designed evaluation benchmark for Korean GEC.In this work, we collect three datasets from different sources (Kor-Lang8, Kor-Native, and Kor-Learner) that covers a wide range of Korean grammatical errors.Considering the nature of Korean grammar, We then define 14 error types for Korean and provide KAGAS (Korean Automatic Grammatical error Annotation System), which can automatically annotate error types from parallel corpora.We use KAGAS on our datasets to make an evaluation benchmark for Korean, and present baseline models trained from our datasets.We show that the model trained with our datasets significantly outperforms the currently used statistical Korean GEC system (Hanspell) on a wider range of error types, demonstrating the diversity and usefulness of the datasets.The implementations and datasets are open-sourced.
SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis

Intimacy is an important social aspect of language.Computational modeling of intimacy in language could help many downstream applications like dialogue systems and offensiveness detection.Despite its importance, resources and approaches on modeling textual intimacy remain rare.To address this gap, we introduce MINT, a new Multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic along with SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis.Our task attracted 45 participants from around the world.While the participants are able to achieve overall good performance on languages in the training set, zero-shot prediction of intimacy in unseen languages remains challenging.Here we provide an overview of the task, summaries of the common approaches, and potential future directions on modeling intimacy across languages.
CORE: Cooperative Training of Retriever-Reranker for Effective Dialogue Response Selection

Establishing retrieval-based dialogue systems that can select appropriate responses from the pre-built index has gained increasing attention.Recent common practice is to construct a twostage pipeline with a fast retriever (e.g., biencoder) for first-stage recall followed by a smart response reranker (e.g., cross-encoder) for precise ranking.However, existing studies either optimize the retriever and reranker in independent ways, or distill the knowledge from a pre-trained reranker into the retriever in an asynchronous way, leading to sub-optimal performance of both modules.Thus, an open question remains about how to train them for a better combination of the best of both worlds.To this end, we present a cooperative training of the response retriever and the reranker whose parameters are dynamically optimized by the ground-truth labels as well as list-wise supervision signals from each other.As a result, the two modules can learn from each other and evolve together throughout the training.Experimental results on two benchmarks demonstrate the superiority of our method.
Evaluation Metrics for Depth and Flow of Knowledge in Non-fiction Narrative Texts

In this paper, we describe the problem of automatically evaluating quality of knowledge expressed in a non-fiction narrative text.We focus on a specific type of documents where each document describes a certain technical problem and its solution.The goal is not only to evaluate the quality of knowledge in such a document, but also to automatically suggest possible improvements to the writer so that a better knowledge-rich document is produced.We propose new evaluation metrics to evaluate quality of knowledge contents as well as flow of different types of sentences.The suggestions for improvement are generated based on these metrics.The proposed metrics are completely unsupervised in nature and they are derived from a set of simple corpus statistics.We demonstrate the effectiveness of the proposed metrics as compared to other existing baseline metrics in our experiments.
Modeling Readers’ Appreciation of Literary Narratives Through Sentiment Arcs and Semantic Profiles

Predicting the perception of literary quality and reader appreciation of narrative texts are highly complex challenges in quantitative and computational literary studies due to the fluid definitions of quality and the vast feature space that can be considered when modeling a literary work.This paper investigates the potential of sentiment arcs combined with topical-semantic profiling of literary narratives as indicators for their literary quality.Our experiments focus on a large corpus of 19 th and 20 the century English language literary fiction, using GoodReads' ratings as an imperfect approximation of the diverse range of reader evaluations and preferences.By leveraging a stacked ensemble of regression models, we achieve a promising performance in predicting average readers' scores, indicating the potential of our approach in modeling perceived literary quality.
DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversations

In open-domain dialogue generation tasks, contexts and responses in most datasets are oneto-one mapped, violating an important manyto-many characteristic: a context leads to various responses, and a response answers multiple contexts.Without such patterns, models poorly generalize and prefer responding safely.Many attempts have been made in either multiturn settings from a one-to-many perspective or in a many-to-many perspective but limited to single-turn settings.The major challenge to many-to-many augment multi-turn dialogues is that discretely replacing each turn with semantic similarity breaks fragile context coherence.In this paper, we propose DialoGue Path Sampling (DialoGPS) method in continuous semantic space, the first many-to-many augmentation method for multi-turn dialogues.Specifically, we map a dialogue to our extended Brownian Bridge, a special Gaussian process.We sample latent variables to form coherent dialogue paths in the continuous space.A dialogue path corresponds to a new multi-turn dialogue and is used as augmented training data.We show the effect of DialoGPS with both automatic and human evaluation.
Trinity at SemEval-2023 Task 12: Sentiment Analysis for Low-resource African Languages using Twitter Dataset

This paper presents a summary of our findings obtained on sentiment analysis of 3 African languages among the 17 languages mentioned in the shared task.We carried out a sentiment analysis on Hausa, Yoruba, and Swahili.The models used here for the mentioned task were logistic regression, SVM, RandomForest, and mBERT along with a few data-preprocessing and oversampling techniques.The performance of the models used was evaluated by considering weighted average and macro average F1 scores as metrics.The best set of scores obtained on the languages Hausa, Yoruba and Swahili are (76.53,76.55), (74.83, 73.15) and (57.79, 48.59) respectively.
Prediction and Calibration: Complex Reasoning over Knowledge Graph with Bi-directional Directed Acyclic Graph Neural Network

Answering complex logical queries is a challenging task for knowledge graph (KG) reasoning.Recently, query embedding (QE) has been proposed to encode queries and entities into the same vector space, and obtain answers based on numerical computation.However, such models obtain the node representations of a query only based on its predecessor nodes, which ignore the information contained in successor nodes.In this paper, we proposed a Bi-directional Directed Acyclic Graph neural network (BiDAG) that splits the reasoning process into prediction and calibration.The joint probability of all nodes is considered by applying a graph neural network (GNN) to the query graph in the calibration process.By the prediction in the first layer and the calibration in deep layers of GNN, BiDAG can outperform previous QE based methods on FB15k, FB15k-237, and NELL995.
PAI at SemEval-2023 Task 4: A General Multi-label Classification System with Class-balanced Loss Function and Ensemble Module

The Human Value Detection shared task (Kiesel et al., 2023) aims to classify whether or not the argument draws on a set of 20 value categories, given a textual argument.This is a difficult task as the discrimination of human values behind arguments is often implicit.Moreover, the number of label categories can be up to 20 and the distribution of data is highly imbalanced.To address these issues, we employ a multi-label classification model and utilize a class-balanced loss function.Our system wins 5 first places, 2 second places, and 6 third places out of 20 categories of the Human Value Detection shared task, and our overall average score of 0.54 also places third.The code is publicly available at https://www.github.com/diqiuzhuanzhuan/semeval2023.
KU-DMIS-MSRA at RadSum23: Pre-trained Vision-Language Model for Radiology Report Summarization

In this paper, we introduce CheXOFA, a new pre-trained vision-language model (VLM) for the chest X-ray domain.Our model is initially pre-trained on various multimodal datasets within the general domain before being transferred to the chest X-ray domain.Following a prominent VLM, we unify various domain-specific tasks into a simple sequenceto-sequence schema.It enables the model to effectively learn the required knowledge and skills from limited resources in the domain.Demonstrating superior performance on the benchmark datasets provided by the BioNLP shared task (Delbrouck et al., 2023), our model benefits from its training across multiple tasks and domains.With subtle techniques including ensemble and factual calibration, our system achieves first place on the RadSum23 leaderboard for the hidden test set.
A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering

Question Answering (QA) is a growing area of research, often used to facilitate the extraction of information from within documents.Stateof-the-art QA models are usually pre-trained on domain-general corpora like Wikipedia and thus tend to struggle on out-of-domain documents without fine-tuning.We demonstrate that synthetic domain-specific datasets can be generated easily using domain-general models, while still providing significant improvements to QA performance.We present two new tools for this task: A flexible pipeline for validating the synthetic QA data and training downstream models on it, and an online interface to facilitate human annotation of this generated data.Using this interface, crowdworkers labelled 1117 synthetic QA pairs, which we then used to fine-tune downstream models and improve domain-specific QA performance by 8.75 F1.
Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge

In this paper, we study the problem of knowledge-intensive text-to-SQL, in which domain knowledge is necessary to parse expert questions into SQL queries over domainspecific tables.We formalize this scenario by building a new Chinese benchmark KNOWSQL consisting of domain-specific questions covering various domains.We then address this problem by presenting formulaic knowledge, rather than by annotating additional data examples.More concretely, we construct a formulaic knowledge bank as a domain knowledge base and propose a framework (REGROUP) to leverage this formulaic knowledge during parsing.Experiments using REGROUP demonstrate a significant 28.2% improvement overall on KNOWSQL.* Contribution during the internship at Microsoft Research Asia.1 EBIT is Earnings Before Interest and Tax, and is calculated as Revenue -Cost of Goods Sold -Operating Expenses.What's the balance of trade of BRIC countries?SELECT Ship_Out -Ship_In, Nation FROM Reports WHERE Nation in ('Brazil', 'Russia', 'India', 'China') Ground Parse Retrieval Balance of Trade = Exports -Imports BRIC countries: Country in {Brazil, Russia
Prompt Learning for Domain Adaptation in Task-Oriented Dialogue

Conversation designers continue to face significant obstacles when creating productionquality task-oriented dialogue systems.The complexity and cost involved in schema development and data collection is often a major barrier for such designers, limiting their ability to create natural, user-friendly experiences.We frame the classification of user intent as the generation of a canonical form, a lightweight semantic representation using natural language.We show that canonical forms offer a promising alternative to traditional methods for intent classification.By tuning soft prompts for a frozen large language model, we show that canonical forms generalize very well to new, unseen domains in a zero-or few-shot setting.The method is also sample-efficient, reducing the complexity and effort of developing new task-oriented dialogue domains.
Combining Noisy Semantic Signals with Orthographic Cues: Cognate Induction for the Indic Dialect Continuum

We present a novel method for unsupervised cognate/borrowing identification from monolingual corpora designed for low and extremely low resource scenarios, based on combining noisy semantic signals from joint bilingual spaces with orthographic cues modelling sound change.We apply our method to the North Indian dialect continuum, containing several dozens of dialects and languages spoken by more than 100 million people.Many of these languages are zero-resource and therefore natural language processing for them is nonexistent.We first collect monolingual data for 26 Indic languages, 16 of which were previously zero-resource, and perform exploratory character, lexical and subword cross-lingual alignment experiments for the first time at this scale on this dialect continuum.We create bilingual evaluation lexicons against Hindi for 20 of the languages.We then apply our cognate identification method on the data, and show that our method outperforms both traditional orthography baselines as well as EM-style learnt edit distance matrices.To the best of our knowledge, this is the first work to combine traditional orthographic cues with noisy bilingual embeddings to tackle unsupervised cognate detection in a (truly) low-resource setup, showing that even noisy bilingual embeddings can act as good guides for this task.We release our multilingual dialect corpus, called HinDialect, as well as our scripts for evaluation data collection and cognate induction. 2
Extended Multilingual Protest News Detection - Shared Task 1, CASE 2021 and 2022

We report results of the CASE 2022 Shared Task 1 on Multilingual Protest Event Detection.This task is a continuation of CASE 2021 that consists of four subtasks that are i) document classification, ii) sentence classification, iii) event sentence coreference identification, and iv) event extraction.The CASE 2022 extension consists of expanding the test data with more data in previously available languages, namely, English, Hindi, Portuguese, and Spanish, and adding new test data in Mandarin, Turkish, and Urdu for Sub-task 1, document classification.The training data from CASE 2021 in English, Portuguese and Spanish were utilized.Therefore, predicting document labels in Hindi, Mandarin, Turkish, and Urdu occurs in a zero-shot setting.The CASE 2022 workshop accepts reports on systems developed for predicting test data of CASE 2021 as well.We observe that the best systems submitted by CASE 2022 participants achieve between 79.71 and 84.06 F1-macro for new languages in a zero-shot setting.The winning approaches are mainly ensembling models and merging data in multiple languages.The best two submissions on CASE 2021 data outperform submissions from last year for Subtask 1 and Subtask 2 in all languages.Only the following scenarios were not outperformed by new submissions on CASE 2021: Subtask 3 Portuguese & Subtask 4 English.
Distinguish Sense from Nonsense: Out-of-Scope Detection for Virtual Assistants

Out of Scope (OOS) detection in Conversational AI solutions enables a chatbot to handle a conversation gracefully when it is unable to make sense of the end-user query. Accurately tagging a query as out-of-domain is particularly hard in scenarios when the chatbot is not equipped to handle a topic which has semantic overlap with an existing topic it is trained on. We propose a simple yet effective OOS detection method that outperforms standard OOS detection methods in a real-world deployment of virtual assistants. We discuss the various design and deployment considerations for a cloud platform solution to train virtual assistants and deploy them at scale. Additionally, we propose a collection of datasets that replicates real-world scenarios and show comprehensive results in various settings using both offline and online evaluation metrics.
Understanding Social Media Cross-Modality Discourse in Linguistic Space

The multimedia communications with texts and images are popular on social media.However, limited studies concern how images are structured with texts to form coherent meanings in human cognition.To fill in the gap, we present a novel concept of cross-modality discourse, reflecting how human readers couple image and text understandings.Text descriptions are first derived from images (named as subtitles) in the multimedia contexts.Five labels -entitylevel insertion, projection and concretization and scene-level restatement and extensionare further employed to shape the structure of subtitles and texts and present their joint meanings.As a pilot study, we also build the very first dataset containing 16K multimedia tweets with manually annotated discourse labels.The experimental results show that the multimedia encoder based on multi-head attention with captions is able to obtain the-state-of-the-art results.
Computing and Exploiting Document Structure to Improve Unsupervised Extractive Summarization of Legal Case Decisions

Though many algorithms can be used to automatically summarize legal case decisions, most fail to incorporate domain knowledge about how important sentences in a legal decision relate to a representation of its document structure.For example, analysis of a legal case summarization dataset demonstrates that sentences serving different types of argumentative roles in the decision appear in different sections of the document.In this work, we propose an unsupervised graph-based ranking model that uses a reweighting algorithm to exploit properties of the document structure of legal case decisions.We also explore the impact of using different methods to compute the document structure.Results on the Canadian Legal Case Law dataset show that our proposed method outperforms several strong baselines.