Reviewerly/exHarmony huggingface dataset:
Towards Zero-Shot Multilingual Transfer for Code-Switched Responses

Recent task-oriented dialog systems obtained great successes in building personal assistants for high resource language such as English, but extending these systems to a global audience is challenging due to the need for annotated data or machine translation systems in the target language.An alternative approach is to leverage existing data in a high-resource language to enable cross-lingual transfer in low-resource language models.However, this type of transfer has not been widely explored in natural language response generation.In this research, we investigate the use of state-of-the-art multilingual models such as mBART and T5 to facilitate zero-shot and few-shot transfer of codeswitched responses.We propose a new adapterbased framework that allows for efficient transfer by learning jointly the task-specific, source and target language representations.Our framework is able to successfully transfer language knowledge even when the target language corpus is limited.We present both quantitative and qualitative analyses to evaluate the effectiveness and limitations of our approach 1 .
Exploring Linguistic Style Matching in Online Communities: The Role of Social Context and Conversation Dynamics

Linguistic style matching (LSM) in conversations can be reflective of several aspects of social influence such as power or persuasion.However, how LSM relates to the outcomes of online communication on platforms such as Reddit is an unknown question.In this study, we analyze a large corpus of two-party conversation threads in Reddit where we identify all occurrences of LSM using two types of style: the use of function words and formality.Using this framework, we examine how levels of LSM differ in conversations depending on several social factors within Reddit: post and subreddit features, conversation depth, user tenure, and the controversiality of a comment.Finally, we measure the change of LSM following loss of status after community banning.Our findings reveal the interplay of LSM in Reddit conversations with several community metrics, suggesting the importance of understanding conversation engagement when understanding community dynamics.
DSRM: Boost Textual Adversarial Training with Distribution Shift Risk Minimization

Adversarial training is one of the bestperforming methods in improving the robustness of deep language models.However, robust models come at the cost of high time consumption, as they require multi-step gradient ascents or word substitutions to obtain adversarial samples.In addition, these generated samples are deficient in grammatical quality and semantic consistency, which impairs the effectiveness of adversarial training.To address these problems, we introduce a novel, effective procedure for instead adversarial training with only clean data.Our procedure, distribution shift risk minimization (DSRM), estimates the adversarial loss by perturbing the input data's probability distribution rather than their embeddings.This formulation results in a robust model that minimizes the expected global loss under adversarial attacks.Our approach requires zero adversarial samples for training and reduces time consumption by up to 70% compared to current best-performing adversarial training methods.Experiments demonstrate that DSRM considerably improves BERT's resistance to textual adversarial attacks and achieves state-of-the-art robust accuracy on various benchmarks.
With Prejudice to None: A Few-Shot, Multilingual Transfer Learning Approach to Detect Social Bias in Low Resource Languages

Warning: This paper contains offensive material by way of examples and case studies which is unavoidable due to the nature of the work.In this paper, we describe our work on social bias detection in a low-resource multilingual setting in which the languages are from two very divergent families-Indo-European (English, Hindi, and Italian) and Altaic (Korean).Currently, the majority of the social bias datasets available are in English and this inhibits progress on social bias detection in lowresource languages.To address this problem, we introduce a new dataset for social bias detection in Hindi and investigate multilingual transfer learning using publicly available English, Italian, and Korean datasets.The Hindi dataset contains ∼ 9 social media posts annotated for (i) binary bias labels (bias/neutral), (ii) binary labels for sentiment (positive/negative), (iii) target groups for each bias category, and (iv) rationale for annotated bias labels (a short piece of text).We benchmark our Hindi dataset using different multilingual models, with XLM-R achieving the best performance of 80.8 macro-F1 score.Our results show that the detection of social biases in resource-constrained languages such as Hindi and Korean may be improved with the use of a similar dataset in English.We also show that translating all datasets into English does not work effectively for detecting social bias, since the nuances of source language are lost in translation.
THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval

The NLI4CT task aims to entail hypotheses based on Clinical Trial Reports (CTRs) and retrieve the corresponding evidence supporting the justification.This task poses a significant challenge, as verifying hypotheses in the NLI4CT task requires the integration of multiple pieces of evidence from one or two CTR(s) and the application of diverse levels of reasoning, including textual and numerical.To address these problems, we present a multi-granularity system for CTR-based textual entailment and evidence retrieval in this paper.Specifically, we construct a Multigranularity Inference Network (MGNet) that exploits sentence-level and token-level encoding to handle both textual entailment and evidence retrieval tasks.Moreover, we enhance the numerical inference capability of the system by leveraging a T5-based model, SciFive, which is pre-trained on the medical corpus.Model ensembling and a joint inference method are further utilized in the system to increase the stability and consistency of inference.The system achieves f1-scores of 0.856 and 0.853 on textual entailment and evidence retrieval tasks, resulting in the best performance on both subtasks.The experimental results corroborate the effectiveness of our proposed method.Our code is publicly available at https:// github.com/THUMLP/NLI4CT.
Contrastive Learning of Sociopragmatic Meaning in Social Media

Recent progress in representation and contrastive learning in NLP has not widely considered the class of sociopragmatic meaning (i.e., meaning in interaction within different language communities).To bridge this gap, we propose a novel framework for learning taskagnostic representations transferable to a wide range of sociopragmatic tasks (e.g., emotion, hate speech, humor, sarcasm).Our framework outperforms other contrastive learning frameworks for both in-domain and out-of-domain data, across both the general and few-shot settings.For example, compared to two popular pre-trained language models, our model obtains an improvement of 11.66 average F 1 on 16 datasets when fine-tuned on only 20 training samples per dataset.We also show that our framework improves uniformity and preserves the semantic structure of representations.
Follow the Knowledge: Structural Biases and Artefacts in Knowledge Grounded Dialog Datasets

Crowd-sourcing has been one of the primary ways to curate conversational data, specially for certain scenarios like grounding in knowledge.In this setting, using online platforms like AMT, non-expert participants are hired to converse with each other, following instructions which try to guide the outcome towards the desired format.The resulting data then is used for different parts of dialog modelling like knowledge selection and response selection/generation.In this work, we take a closer look into two of the most popular knowledge grounded dialog (KGD) datasets.Investigating potential biases and artefacts in knowledge selection labels, we observe that in many cases the 'knowledge selection flow' simply follows the order of presented knowledge pieces.In Wizard of Wikipedia (the most popular KGD dataset) we use simple content-agnostic models based on this bias to get significant knowledge selection performance.In Topical-Chat we see a similar correlation between the knowledge selection sequence and the order of entities and their segments, as provided to crowd-source workers.We believe that the observed results, question the significance and origin of the presumed dialog-level attributes like 'knowledge flow' in these crowd-sourced datasets.
Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints

Lexically-constrained NMT (LNMT) aims to incorporate user-provided terminology into translations.Despite its practical advantages, existing work has not evaluated LNMT models under challenging real-world conditions.In this paper, we focus on two important but understudied issues that lie in the current evaluation process of LNMT studies.The model needs to cope with challenging lexical constraints that are "homographs" or "unseen" during training.To this end, we first design a homograph disambiguation module to differentiate the meanings of homographs.Moreover, we propose PLUMCOT, which integrates contextually rich information about unseen lexical constraints from pre-trained language models and strengthens a copy mechanism of the pointer network via direct supervision of a copying score.We also release HOLLY, an evaluation benchmark for assessing the ability of a model to cope with "homographic" and "unseen" lexical constraints.Experiments on HOLLY and the previous test setup show the effectiveness of our method.The effects of PLUMCOT are shown to be remarkable in "unseen" constraints.Our dataset is available at https://github.com/papago-lab/HOLLY-benchmark.
Few-shot Reranking for Multi-hop QA via Language Model Prompting

We study few-shot reranking for multi-hop QA (MQA) with open-domain questions.To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PROMPTRANK, which relies on language model prompting for multi-hop path reranking.PROMPTRANK first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model.PROMPTRANK yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-theart methods trained on thousands of examples -73.6 recall@10 by PROMPTRANK vs. 77.8 by PathRetriever (Asai et al., 2020) and 77.5 by multi-hop dense retrieval (Xiong et al., 2021).
A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering

Recent studies have pointed many welldeveloped Visual Question Answering (VQA) systems suffer from bias problem.Despite the remarkable performance gained on In-Distribution (ID) datasets, the VQA model might capture the superficial correlation from question to answer rather than showing real reasoning abilities.Therefore, when switching to Out-of-Distribution (OOD) dataset, whose test distribution is unknown or even reversed with the training set, significant drops appear.Efforts have been devoted to negative bias brought by language prior but are still limited by two aspects.First, most current debiasing methods achieve promising OOD generalization ability with a sacrifice of the ID performance.Second, they are restricted by exploiting comprehensive biases, since weakening the language bias is mainly focused and few works consider vision bias.In this paper, we investigate a straightforward way to mitigate bias problem for VQA task by subtracting bias score from VQA base score.Then we design two bias learning branches to detect more bias, which is combined with a dynamical constraint loss to alleviate the problem of over-correction and insufficient debiasing.We evaluate our method on the challenging VQA v2.0 and VQA-CP V2.0 datasets and achieve significant improvement.
FSUIE: A Novel Fuzzy Span Mechanism for Universal Information Extraction

Universal Information Extraction (UIE) has been introduced as a unified framework for various Information Extraction (IE) tasks and has achieved widespread success.Despite this, UIE models have limitations.For example, they rely heavily on span boundaries in the data during training, which does not reflect the reality of span annotation challenges.Slight adjustments to positions can also meet requirements.Additionally, UIE models lack attention to the limited span length feature in IE.To address these deficiencies, we propose the Fuzzy Span Universal Information Extraction (FSUIE) framework.Specifically, our contribution consists of two concepts: fuzzy span loss and fuzzy span attention.Our experimental results on a series of main IE tasks show significant improvement compared to the baseline, especially in terms of fast convergence and strong performance with small amounts of data and training epochs.These results demonstrate the effectiveness and generalization of FSUIE in different tasks, settings, and scenarios.
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU).While various visionlanguage pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far.A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines.In this paper, we propose WUKONG-READER, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines.We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines.Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines.Experiments show that WUKONG-READER brings superior performance on various VDU tasks in both English and Chinese.The fine-grained alignment over textlines also empowers WUKONG-READER with promising localization ability.
A Fast Algorithm for Computing Prefix Probabilities

Multiple algorithms are known for efficiently calculating the prefix probability of a string under a probabilistic context-free grammar (PCFG).Good algorithms for the problem have a runtime cubic in the length of the input string.However, some proposed algorithms are suboptimal with respect to the size of the grammar.This paper proposes a novel speed-up of Jelinek and Lafferty's ( 1991) algorithm, which runs in O(N 3 |N | 3 + |N | 4 ), where N is the input length and |N | is the number of non-terminals in the grammar.In contrast, our speed-up runs in O(N 2 |N | 3 + N 3 |N | 2 ).https://github.com/rycolab/prefix-parsing * ⇒ is the closure over applications of derivation rules of the grammar. 1 Our paper gives a more efficient algorithm for the simultaneous computation of the prefix probabilities of all prefixes of a string w under a PCFG.The authors are aware of two existing efficient algorithms to compute prefix probabilities under a PCFG. 2 The first is Jelinek and Lafferty's (1991)1 Specifically, α * ⇒ β means that there exists an n ≥ 0 such that α ⇒ • • • ⇒ n times β, where ⇒ marks a derivation step.
Consistency Regularization Training for Compositional Generalization

Existing neural models have difficulty generalizing to unseen combinations of seen components.To achieve compositional generalization, models are required to consistently interpret (sub)expressions across contexts.Without modifying model architectures, we improve the capability of Transformer on compositional generalization through consistency regularization training, which promotes representation consistency across samples and prediction consistency for a single sample.Experimental results on semantic parsing and machine translation benchmarks empirically demonstrate the effectiveness and generality of our method.In addition, we find that the prediction consistency scores on in-distribution validation sets can be an alternative for evaluating models during training, when commonly-used metrics are not informative.
Reproducibility in NLP: What Have We Learned from the Checklist?

Scientific progress in NLP rests on the reproducibility of researchers' claims.The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include.We provide the first analysis of the Checklist by examining 10,405 anonymous responses to it.First, we find evidence of an increase in reporting of information on efficiency, validation performance, summary statistics, and hyperparameters after the Checklist's introduction.Further, we show acceptance rate grows for submissions with more YES responses.We find that the 44% of submissions that gather new data are 5% less likely to be accepted than those that did not; the average reviewer-rated reproducibility of these submissions is also 2% lower relative to the rest.We find that only 46% of submissions claim to open-source their code, though submissions that do have 8% higher reproducibility score relative to those that do not, the most for any item.We discuss what can be inferred about the state of reproducibility in NLP, and provide a set of recommendations for future conferences, including: a) allowing submitting code and appendices one week after the deadline, and b) measuring dataset reproducibility by a checklist of data collection practices.
DP-BART for Privatized Text Rewriting under Local Differential Privacy

Privatized text rewriting with local differential privacy (LDP) is a recent approach that enables sharing of sensitive textual documents while formally guaranteeing privacy protection to individuals.However, existing systems face several issues, such as formal mathematical flaws, unrealistic privacy guarantees, privatization of only individual words, as well as a lack of transparency and reproducibility.In this paper, we propose a new system 'DP-BART' that largely outperforms existing LDP systems.Our approach uses a novel clipping method, iterative pruning, and further training of internal representations which drastically reduces the amount of noise required for DP guarantees.We run experiments on five textual datasets of varying sizes, rewriting them at different privacy guarantees and evaluating the rewritten texts on downstream text classification tasks.Finally, we thoroughly discuss the privatized text rewriting approach and its limitations, including the problem of the strict text adjacency constraint in the LDP paradigm that leads to the high noise requirement. 1
Revisiting Sentence Union Generation as a Testbed for Text Consolidation

Tasks involving text generation based on multiple input texts, such as multi-document summarization, long-form question answering and contemporary dialogue applications, challenge models for their ability to properly consolidate partly-overlapping multi-text information.However, these tasks entangle the consolidation phase with the often subjective and illdefined content selection requirement, impeding proper assessment of models' consolidation capabilities.In this paper, we suggest revisiting the sentence union generation task as an effective well-defined testbed for assessing text consolidation capabilities, decoupling the consolidation challenge from subjective content selection.To support research on this task, we present refined annotation methodology and tools for crowdsourcing sentence union, create the largest union dataset to date and provide an analysis of its rich coverage of various consolidation aspects.We then propose a comprehensive evaluation protocol for union generation, including both human and automatic evaluation.Finally, as baselines, we evaluate state-of-theart language models on the task, along with a detailed analysis of their capacity to address multi-text consolidation challenges and their limitations. 1
World Models for Math Story Problems

Solving math story problems is a complex task for students and NLP models alike, requiring them to understand the world as described in the story and reason over it to compute an answer.Recent years have seen impressive performance on automatically solving these problems with large pre-trained language models and innovative techniques to prompt them.However, it remains unclear if these models possess accurate representations of mathematical concepts.This leads to lack of interpretability and trustworthiness which impedes their usefulness in various applications.In this paper, we consolidate previous work on categorizing and representing math story problems and develop MATHWORLD, which is a graph-based semantic formalism specific for the domain of math story problems.With MATHWORLD, we can assign world models to math story problems which represent the situations and actions introduced in the text and their mathematical relationships.We combine math story problems from several existing datasets and annotate a corpus of 1, 019 problems and 3, 204 logical forms with MATHWORLD.Using this data, we demonstrate the following use cases of MATHWORLD: (1) prompting language models with synthetically generated questionanswer pairs to probe their reasoning and world modeling abilities, and (2) generating new problems by using the world models as a design space.
Enhancing Grammatical Error Correction Systems with Explanations

Grammatical error correction systems improve written communication by detecting and correcting language mistakes.To help language learners better understand why the GEC system makes a certain correction, the causes of errors (evidence words) and the corresponding error types are two key factors.To enhance GEC systems with explanations, we introduce EXPECT, a large dataset annotated with evidence words and grammatical error types.We propose several baselines and analysis to understand this task.Furthermore, human evaluation verifies our explainable GEC system's explanations can assist second-language learners in determining whether to accept a correction suggestion and in understanding the associated grammar rule.
Revisiting Sample Size Determination in Natural Language Understanding

Knowing exactly how many data points need to be labeled to achieve a certain model performance is a hugely beneficial step towards reducing the overall budgets for annotation.It pertains to both active learning and traditional data annotation, and is particularly beneficial for low resource scenarios.Nevertheless, it remains a largely under-explored area of research in NLP.We therefore explored various techniques for estimating the training sample size necessary to achieve a targeted performance value.We derived a simple yet effective approach to predict the maximum achievable model performance based on small amount of training sampleswhich serves as an early indicator during data annotation for data quality and sample size determination.We performed ablation studies on four language understanding tasks, and showed that the proposed approach allows us to forecast model performance within a small margin of mean absolute error (∽ 0.9%) with only 10% data 1 .