Reviewerly/exHarmony huggingface dataset:
Textual Manifold-based Defense Against Natural Language Adversarial Examples

Recent studies on adversarial images have shown that they tend to leave the underlying low-dimensional data manifold, making them significantly more challenging for current models to make correct predictions.This so-called off-manifold conjecture has inspired a novel line of defenses against adversarial attacks on images.In this study, we find a similar phenomenon occurs in the contextualized embedding space induced by pretrained language models, in which adversarial texts tend to have their embeddings diverge from the manifold of natural ones.Based on this finding, we propose Textual Manifold-based Defense (TMD), a defense mechanism that projects text embeddings onto an approximated embedding manifold before classification.It reduces the complexity of potential adversarial examples, which ultimately enhances the robustness of the protected model.Through extensive experiments, our method consistently and significantly outperforms previous defenses under various attack settings without trading off clean accuracy.To the best of our knowledge, this is the first NLP defense that leverages the manifold structure against adversarial attacks.Our code is available at https://github.com/dangne/tmd.
Does Corpus Quality Really Matter for Low-Resource Languages?

The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl.While prior work has identified major issues on the quality of these datasets (Kreutzer et al., 2021), it is not clear how this impacts downstream performance.Taking representation learning in Basque as a case study, we explore tailored crawling-manually identifying and scraping websites with high-quality content-as an alternative to filtering Common-Crawl.Our new corpus, called EusCrawl, is similar in size to the Basque portion of popular multilingual corpora like CC100 and mC4, yet it has a much higher quality according to native annotators.For instance, 66% of documents are rated as high-quality for EusCrawl, in contrast with < 33% for both mC4 and CC100.Nevertheless, we obtain similar results on downstream NLU tasks regardless of the corpus used for pre-training.Our work suggests that NLU performance in low-resource languages is not primarily constrained by the quality of the data, and other factors like corpus size and domain coverage can play a more important role.LangID
DebiasGAN: Eliminating Position Bias in News Recommendation with Adversarial Learning

Click behaviors are widely used for learning news recommendation models, but they are heavily affected by the biases brought by the news display positions.It is important to remove position biases to train unbiased recommendation model and capture unbiased user interest.In this paper, we propose a news recommendation method named DebiasGAN that can effectively alleviate position biases via adversarial learning.The core idea is modeling the personalized effect of position bias on click behaviors in a candidate-aware way, and learning debiased candidate-aware user embeddings from which the position information cannot be discriminated.More specifically, we use a bias-aware click model to capture the effect of position bias on click behaviors, and use a bias-invariant click model with random candidate positions to estimate the ideally unbiased click scores.We apply adversarial learning to the embeddings learned by the two models to help the bias-invariant click model capture debiased user interest.Experimental results on two real-world datasets show that DebiasGAN effectively improves news recommendation by eliminating position biases.
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding

From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning.For example, if asked to identify the "person who needs healing" in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person.We present a new commonsense task, Humancentric Commonsense Grounding, that tests the models' ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions.We further create a benchmark, Hu-manCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes.We set up a contextobject-aware method as a strong baseline that outperforms previous pre-trained and nonpretrained models.Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works.Data and code will be available at https://github.com/Hxyou/HumanCog.
Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora

The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications.However, standard methods for evaluating these metrics have yet to be established.We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior.We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics.Our measures revealed that recentlydeveloped metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters

Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks by adding and tuning a small number of new parameters.Previously proposed adapter architectures are all feed-forward neural networks.In this paper, we investigate the effectiveness of using tiny-attention-i.e., attention with extremely small per-head dimensionalityas adapters.Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters.Moreover, we view its multiple attention heads as a mixture of experts and propose to average their weights during deployment, which further reduces its inference computation cost.On the GLUE benchmark, our tiny-attention adapter outperforms the other parameter-efficient transfer learning methods as well as full fine-tuning while only updating 0.05% of the parameters.On the FewGLUE benchmark, its performance is comparable to that of
Improving Factual Consistency in Summarization with Compression-Based Post-Editing

State-of-the-art summarization models still struggle to be factually consistent with the input text.A model-agnostic way to address this problem is post-editing the generated summaries.However, existing approaches typically fail to remove entity errors if a suitable input entity replacement is not available or may insert erroneous content.In our work, we focus on removing extrinsic entity errors, or entities not in the source, to improve consistency while retaining the summary's essential information and form.We propose to use sentence-compression data to train the post-editing model to take a summary with extrinsic entity errors marked with special tokens and output a compressed, well-formed summary with those errors removed.We show that this model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and that this model can be applied on top of another post-editor, improving entity precision by up to a total of 38%.We perform an extensive comparison of post-editing approaches that demonstrate trade-offs between factual consistency, informativeness, and grammaticality, and we analyze settings where posteditors show the largest improvements.Source: The oil price collapse sent global markets reeling throughout 2015.... Brent crude oil was up 3% at $37.60 per barrel for the day but down 35% over the year.System Summary Original Wall Street markets closed lower on the last trading day of 2015 as oil prices languished at $28 a barrel for much of the year.Entity Swap No Change Rewrite Wall Street markets closed lower on the last trading day of 2015 as oil prices languished at $37.60 per barrel ... but remain down 35% over the year.Delete Wall Street markets closed lower on 2015 as oil prices languished at a for much of the year.Compress (ours) Wall Street markets closed lower as oil prices languished for much of the year.
Measuring and Improving Semantic Diversity of Dialogue Generation

Response diversity has become an important criterion for evaluating the quality of open-domain dialogue generation models. However, current evaluation metrics for response diversity often fail to capture the semantic diversity of generated responses, as they mainly consider lexical aspects of the generated responses. In this paper, we introduce a new automatic evaluation metric to measure the semantic diversity of generated responses. Through human evaluation, we demonstrate that our proposed metric captures human judgments on response diversity better than existing lexical-level diversity metrics. Furthermore, motivated by analyzing an existing dialogue dataset, we propose a simple yet effective learning method that improves the semantic diversity of generated responses. Our learning method weights training samples based on the semantic distribution of the training set.We show that our learning method improves response diversity and coherency better than other baseline methods through automatic and human evaluation.
BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples

Natural language inference (NLI) is critical for complex decision-making in biomedical domain.One key question, for example, is whether a given biomedical mechanism is supported by experimental evidence.This can be seen as an NLI problem but there are no directly usable datasets to address this.The main challenge is that manually creating informative negative examples for this task is difficult and expensive.We introduce a novel semi-supervised procedure that bootstraps an NLI dataset from existing biomedical dataset that pairs mechanisms with experimental evidence in abstracts.We generate a range of negative examples using nine strategies that manipulate the structure of the underlying mechanisms both with rules, e.g., flip the roles of the entities in the interaction, and, more importantly, as perturbations via logical constraints in a neuro-logical decoding system (Lu et al., 2021b).We use this procedure to create a novel dataset for NLI in the biomedical domain, called BioNLI and benchmark two state-of-the-art biomedical classifiers.The best result we obtain is around mid 70s in F1, suggesting the difficulty of the task.Critically, the performance on the different classes of negative examples varies widely, from 97% F1 on the simple role change negative examples, to barely better than chance on the negative examples generated using neuro-logic decoding. 1
Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering

Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a twostage framework that first retrieves external knowledge given the visual question and then predicts the answer based on the retrieved content.However, the retrieved knowledge is often inadequate.Retrievals are frequently too general and fail to cover specific knowledge needed to answer the question.Also, the naturally available supervision (whether the passage contains the correct answer) is weak and does not guarantee question relevancy.To address these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that provides stronger supervision during training and recognizes questionrelevant entities to help retrieve more specific knowledge.Experiments show that our En-FoRe model achieves superior retrieval performance on OK-VQA, the currently largest outside-knowledge VQA dataset.We also combine the retrieved knowledge with state-of-theart VQA models, and achieve a new state-ofthe-art performance on OK-VQA.Q: What holiday is this?A: Thanksgiving.
Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems

Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data.Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks.However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) systems.In this paper, we showcase the advantages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks.To this end, we utilize light-weight adapters that can be easily integrated with PLMs and serve as a repository for facts learned from different KBs.To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) -a probe designed specifically for TOD models.Experiments 1 on KPRS and the response generation task show improvements of knowledge injection with adapters over strong baselines.
HydraSum: Disentangling Style Features in Text Summarization with Multi-Decoder Models

Summarization systems make numerous "decisions" about summary properties during inference, e.g.degree of copying, specificity and length of outputs, etc.However, these are implicitly encoded within model parameters and specific styles cannot be enforced.To address this, we introduce HYDRASUM, a new summarization architecture that extends the single decoder framework of current models to a mixture-of-experts version with multiple decoders.We show that HYDRASUM's multiple decoders automatically learn contrasting summary styles when trained under the standard training objective without any extra supervision.Through experiments on three summarization datasets (CNN, NEWSROOM and XSUM), we show that HYDRASUM provides a simple mechanism to obtain stylistically-diverse summaries by sampling from either individual decoders or their mixtures, outperforming baseline models.Finally, we demonstrate that a small modification to the gating strategy during training can enforce an even stricter style partitioning, e.g.high-vs low-abstractiveness or high-vs low-specificity, allowing users to sample from a larger area in the generation space and vary summary styles along multiple dimensions. 1
Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problem

Math word problem solver requires both precise relation reasoning about quantities in the text and reliable generation for the diverse equation.Current sequence-to-tree or relation extraction methods regard this only from a fixed view, struggling to simultaneously handle complex semantics and diverse equations.However, human solving naturally involves two consistent reasoning views: top-down and bottom-up, just as math equations also can be expressed in multiple equivalent forms: pre-order and postorder.We propose a multi-view consistent contrastive learning for a more complete semanticsto-equation mapping.The entire process is decoupled into two independent but consistent views: top-down decomposition and bottomup construction, and the two reasoning views are aligned in multi-granularity for consistency, enhancing global generation and precise reasoning.Experiments on multiple datasets across two languages show our approach significantly outperforms the existing baselines, especially on complex problems 1 .We also show after consistent alignment, multi-view can absorb the merits of both views and generate more diverse results consistent with the mathematical laws.
Improving Event Coreference Resolution Using Document-level and Topic-level Information

Event coreference resolution (ECR) aims to cluster event mentions that refer to the same real-world events.Deep learning methods have achieved SOTA results on the ECR task.However, due to the encoding length limitation, previous methods either adopt classical pairwise models based on sentence-level context or split each document into multiple chunks and encode them separately.They failed to capture the interactions and contextual cues among those long-distance event mentions.Besides, highlevel information, such as event topics, is rarely considered to enhance representation learning for ECR.To address the above two issues, we first apply a Longformer-based encoder to obtain the document-level embeddings and an encoder with a trigger-mask mechanism to learn sentence-level embeddings based on local context.In addition, we propose an event topic generator to infer the latent topic-level representations.Finally, using the above event embeddings, we employ a multiple tensor matching method to capture their interactions at the document, sentence, and topic levels.Experimental results on the KBP 2017 dataset show that our model 1 outperforms the SOTA baselines.
SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training

Self-training methods have been explored in recent years and have exhibited great performance in improving semi-supervised learning.This work presents a Simple instance-Adaptive self-Training method (SAT) for semisupervised text classification.SAT first generates two augmented views for each unlabeled data and then trains a meta-learner to automatically identify the relative strength of augmentations based on the similarity between the original view and the augmented views.The weakly-augmented view is fed to the model to produce a pseudo-label and the stronglyaugmented view is used to train the model to predict the same pseudo-label.We conducted extensive experiments and analyses on three text classification datasets and found that with varying sizes of labeled training data, SAT consistently shows competitive performance compared to existing semi-supervised learning methods.Our code can be found at https: //github.com/declare-lab/SAT.git.
DEMETR: Diagnosing Evaluation Metrics for Translation

While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particular candidate translation can be traced back to the presence or absence of certain words.The operations of newer learned metrics (e.g., BLEURT, COMET), which leverage pretrained language models to achieve higher correlations with human quality judgments than BLEU, are opaque in comparison.In this paper, we shed light on the behavior of these learned metrics by creating DEMETR, a diagnostic dataset with 31K English examples (translated from 10 source languages) for evaluating the sensitivity of MT evaluation metrics to 35 different linguistic perturbations spanning semantic, syntactic, and morphological error categories.All perturbations were carefully designed to form minimal pairs with the actual translation (i.e., differ in only one aspect).We find that learned metrics perform substantially better than string-based metrics on DEMETR.Additionally, learned metrics differ in their sensitivity to various phenomena (e.g., BERTSCORE is sensitive to untranslated words but relatively insensitive to gender manipulation, while COMET is much more sensitive to word repetition than to aspectual changes).We publicly release DEMETR to spur more informed future development of machine translation evaluation metrics 1 .
Developing Prefix-Tuning Models for Hierarchical Text Classification

Hierarchical text classification (HTC) is a key problem and task in many industrial applications, which aims to predict labels organized in a hierarchy for given input text.For example, HTC can group the descriptions of online products into a taxonomy or organizing customer reviews into a hierarchy of categories.In real-life applications, while Pre-trained Language Models (PLMs) have dominated many NLP tasks, they face significant challenges too-the conventional fine-tuning process needs to modify and save models with a huge number of parameters.This is becoming more critical for HTC in both global and local modelling-the latter needs to learn multiple classifiers at different levels/nodes in a hierarchy.The concern will be even more serious since PLM sizes are continuing to increase in order to attain more competitive performances.Most recently, prefix tuning has become a very attractive technology by only tuning and saving a tiny set of parameters.Exploring prefix turning for HTC is hence highly desirable and has timely impact.In this paper, we investigate prefix tuning on HTC in two typical setups: local and global HTC.Our experiment shows that the prefix-tuning model only needs less than 1% of parameters and can achieve performance comparable to regular full fine-tuning.We demonstrate that using contrastive learning in learning prefix vectors can further improve HTC performance.
ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation

Adversarial Examples Detection (AED) is a crucial defense technique against adversarial attacks and has drawn increasing attention from the Natural Language Processing (NLP) community.Despite the surge of new AED methods, our studies show that existing methods heavily rely on a shortcut to achieve good performance.In other words, current search-based adversarial attacks in NLP stop once model predictions change, and thus most adversarial examples generated by those attacks are located near model decision boundaries.To surpass this shortcut and fairly evaluate AED methods, we propose to test AED methods with Far Boundary (FB) adversarial examples.Existing methods show worse than random guess performance under this scenario.To overcome this limitation, we propose a new technique, ADDMU, adversary detection with data and model uncertainty, which combines two types of uncertainty estimation for both regular and FB adversarial example detection.Our new method outperforms previous methods by 3.6 and 6.0 AUC points under each scenario.Finally, our analysis shows that the two types of uncertainty provided by ADDMU can be leveraged to characterize adversarial examples and identify the ones that contribute most to model's robustness in adversarial training.
Directions for NLP Practices Applied to Online Hate Speech Detection

Addressing hate speech in online spaces has been conceptualized as a classification task that uses Natural Language Processing (NLP) techniques.Through this conceptualization, the hate speech detection task has relied on common conventions and practices from NLP.For instance, inter-annotator agreement is conceptualized as a way to measure dataset quality and certain metrics and benchmarks are used to assure model generalization.However, hate speech is a deeply complex and situated concept that eludes such static and disembodied practices.In this position paper, we critically reflect on these methodologies for hate speech detection, we argue that many conventions in NLP are poorly suited for the problem and encourage researchers to develop methods that are more appropriate for the task.