Reviewerly/exHarmony huggingface dataset:
Arabic Keyphrase Extraction: Enhancing Deep Learning Models with Pre-trained Contextual Embedding and External Features

Keyphrase extraction is essential to many Information retrieval (IR) and Natural language Processing (NLP) tasks such as summarization and indexing.This study investigates deep learning approaches to Arabic keyphrase extraction.We address the problem as sequence classification and create a Bi-LSTM model to classify each sequence token as either part of the keyphrase or outside of it.We have extracted word embeddings from two pre-trained models, Word2Vec and BERT.Moreover, we have investigated the effect of incorporating linguistic, positional, and statistical features with word embeddings on performance.Our best-performing model has achieved 0.45 F1score on ArabicKPE dataset when combining linguistic and positional features with BERT embedding.
Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search

Online relevance matching is an essential task of e-commerce product search to boost the utility of search engines and ensure a smooth user experience.Previous work adopts either classical relevance matching models or Transformerstyle models to address it.However, they ignore the inherent bipartite graph structures that are ubiquitous in e-commerce product search logs and are too inefficient to deploy online.In this paper, we design an efficient knowledge distillation framework for e-commerce relevance matching to integrate the respective advantages of Transformer-style models and classical relevance matching models.Especially for the core student model of the framework, we propose a novel method using k-order relevance modeling.The experimental results on large-scale real-world data (the size is 6∼174 million) show that the proposed method significantly improves the prediction accuracy in terms of human relevance judgment.We deploy our method to JD.com online search platform.The A/B testing results show that our method significantly improves most business metrics under price sort mode and default sort mode.
CPL: Counterfactual Prompt Learning for Vision and Language Models

Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP.However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts.Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel Counterfactual Prompt Learning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework.Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning.Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP.On image classification, we achieve a 3.55% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09% and 25.08% relative improvements across three few-shot scenarios on unseen test sets respectively. 1
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts

Explicit decomposition modeling, which involves breaking down complex tasks into more straightforward and often more interpretable sub-tasks, has long been a central theme in developing robust and interpretable NLU systems.However, despite the many datasets and resources built as part of this effort, the majority have small-scale annotations and limited scope, which is insufficient to solve general decomposition tasks.In this paper, we look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts, particularly large-scale parallel news.We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible.For example, on semantic parsing, our model, DECOMPT5, improves 20% to 30% on two datasets, Overnight and TORQUE, over the baseline language model.We further use DECOMPT5 to build a novel decompositionbased QA system named DECOMPENTAIL, improving over state-of-the-art models, including GPT-3, on both HotpotQA and StrategyQA by 8% and 4%, respectively.
Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery

Discovering out-of-domain (OOD) intent is important for developing new skills in taskoriented dialogue systems.The key challenges lie in how to transfer prior in-domain (IND) knowledge to OOD clustering, as well as jointly learn OOD representations and cluster assignments.Previous methods suffer from indomain overfitting problem, and there is a natural gap between representation learning and clustering objectives.In this paper, we propose a unified K-nearest neighbor contrastive learning framework to discover OOD intents.Specifically, for IND pre-training stage, we propose a KCL objective to learn inter-class discriminative features, while maintaining intraclass diversity, which alleviates the in-domain overfitting problem.For OOD clustering stage, we propose a KCC method to form compact clusters by mining true hard negative samples, which bridges the gap between clustering and representation learning.Extensive experiments on three benchmark datasets show that our method achieves substantial improvements over the state-of-the-art methods. 1 * The first two authors contribute equally.Weiran Xu
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation

Evaluating automatic text simplification (ATS) systems is a difficult task that is either performed by automatic metrics or user-based evaluations.However, from a linguistic point-ofview, it is not always clear on what bases these evaluations operate.In this paper, we propose annotations of the ASSET corpus that can be used to shed more light on ATS evaluation.In addition to contributing with this resource, we show how it can be used to analyze SARI's behavior and to re-evaluate existing ATS systems.We present our insights as a step to improve ATS evaluation protocols in the future.
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets

Precisely assessing the progress in natural language generation (NLG) tasks is challenging, and human evaluation to establish a preference in a model's output over another is often necessary.However, human evaluation is usually costly, difficult to reproduce, and non-reusable.In this paper, we propose a new and simple automatic evaluation method for NLG called Near-Negative Distinction (NND) that repurposes prior human annotations into NND tests.In an NND test, an NLG model must place a higher likelihood on a high-quality output candidate than on a near-negative candidate with a known error.Model performance is established by the number of NND tests a model passes, as well as the distribution over task-specific errors the model fails on.Through experiments on three NLG tasks (question generation, question answering, and summarization), we show that NND achieves a higher correlation with human judgments than standard NLG evaluation metrics.We then illustrate NND evaluation in four practical scenarios, for example performing fine-grain model analysis, or studying model training dynamics.Our findings suggest that NND can give a second life to human annotations and provide low-cost NLG evaluation.
Finding Dataset Shortcuts with Grammar Induction

Many NLP datasets have been found to contain shortcuts: simple decision rules that achieve surprisingly high accuracy.However, it is difficult to discover shortcuts automatically.Prior work on automatic shortcut detection has focused on enumerating features like unigrams or bigrams, which can find only low-level shortcuts, or relied on post-hoc model interpretability methods like saliency maps, which reveal qualitative patterns without a clear statistical interpretation.In this work, we propose to use probabilistic grammars to characterize and discover shortcuts in NLP datasets.Specifically, we use a contextfree grammar to model patterns in sentence classification datasets and use a synchronous context-free grammar to model datasets involving sentence pairs.The resulting grammars reveal interesting shortcut features in a number of datasets, including both simple and high-level features, and automatically identify groups of test examples on which conventional classifiers fail.Finally, we show that the features we discover can be used to generate diagnostic contrast examples and incorporated into standard robust optimization methods to improve worst-group accuracy. 1
LittleBird: Efficient Faster &amp; Longer Transformer for Question Answering

BERT has shown a lot of sucess in a wide variety of NLP tasks.But it has a limitation dealing with long inputs due to its attention mechanism.Longformer, ETC and BigBird addressed this issue and effectively solved the quadratic dependency problem.However we find that these models are not sufficient, and propose LittleBird, a novel model based on BigBird with improved speed and memory footprint while maintaining accuracy.In particular, we devise a more flexible and efficient position representation method based on Attention with Linear Biases (ALiBi).We also show that replacing the method of global information represented in the BigBird with pack and unpack attention is more effective.The proposed model can work on long inputs even after being pretrained on short inputs, and can be trained efficiently reusing existing pre-trained language model for short inputs.This is a significant benefit for low-resource languages where large amounts of long text data are difficult to obtain.As a result, our experiments show that LittleBird works very well in a variety of languages, achieving high performance in question answering tasks, particularly in KorQuAD2.0,Korean
DeepGen: Diverse Search Ad Generation and Real-Time Customization

We present DeepGen, a system deployed at web scale for automatically creating sponsored search advertisements (ads) for Bing Ads customers.We leverage state-of-the-art natural language generation (NLG) models to generate fluent ads from advertiser's web pages in an abstractive fashion and solve practical issues such as factuality and inference speed.In addition, our system creates a customized ad in real-time in response to the user's search query, therefore highlighting different aspects of the same product based on what the user is looking for.To achieve this, our system generates a diverse choice of smaller pieces of the ad ahead of time and, at query time, selects the most relevant ones to be stitched into a complete ad.We improve generation diversity by training a controllable NLG model to generate multiple ads for the same web page highlighting different selling points.Our system design further improves diversity horizontally by first running an ensemble of generation models trained with different objectives and then using a diversity sampling algorithm to pick a diverse subset of generation results for online selection.Experimental results show the effectiveness of our proposed system design.Our system is currently deployed in production, serving ∼4% of ads globally on Bing.
Extracted BERT Model Leaks More Information than You Think!

The collection and availability of big data, combined with advances in pre-trained models (e.g.BERT), have revolutionized the predictive performance of natural language processing tasks.This allows corporations to provide machine learning as a service (MLaaS) by encapsulating fine-tuned BERT-based models as APIs.Due to significant commercial interest, there has been a surge of attempts to steal remote services via model extraction.Although previous works have made progress in defending against model extraction attacks, there has been little discussion on their performance in preventing privacy leakage.This work bridges this gap by launching an attribute inference attack against the extracted BERT model.Our extensive experiments reveal that model extraction can cause severe privacy leakage even when victim models are facilitated with advanced defensive strategies.
Prototype-Representations for Training Data Filtering in Weakly-Supervised Information Extraction

The availability of high quality training data is still a bottleneck for the practical utilization of information extraction models, despite the breakthroughs in zero and few-shot learning techniques. This is further exacerbated for industry applications, where new tasks, domains, and specific use cases keep arising, which makes it impractical to depend on manually annotated data. Therefore, weak and distant supervision emerged as popular approaches to bootstrap training, utilizing labeling functions to guide the annotation process. Weakly-supervised annotation of training data is fast and efficient, however, it results in many irrelevant and out-of-context matches. This is a challenging problem that can degrade the performance in downstream models, or require a manual data cleaning step that can incur significant overhead. In this paper we present a prototype-based filtering approach, that can be utilized to denoise weakly supervised training data. The system is very simple, unsupervised, scalable, and requires little manual intervention, yet results in significant precision gains. We apply the technique in the task of attribute value extraction in e-commerce websites, and achieve up to 9% gain in precision for the downstream models, with a minimal drop in recall.
ULN: Towards Underspecified Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions.Despite the significant performance improvement, the wide use of fine-grained instructions fails to characterize more practical linguistic variations in reality.To fill in this gap, we introduce a new setting, namely Underspecified vision-and-Language Navigation (ULN), and associated evaluation datasets.ULN evaluates agents using multi-level underspecified instructions instead of purely fine-grained or coarsegrained, which is a more realistic and general setting.As a primary step toward ULN, we propose a VLN framework that consists of a classification module, a navigation agent, and an Exploitation-to-Exploration (E2E) module.Specifically, we propose to learn Granularity Specific Sub-networks (GSS) for the agent to ground multi-level instructions with minimal additional parameters.Then, our E2E module estimates grounding uncertainty and conducts multi-step lookahead exploration to improve the success rate further.Experimental results show that existing VLN models are still brittle to multi-level language underspecification.Our framework is more robust and outperforms the baselines on ULN by "10% relative success rate across all levels. 1
Tractable &amp; Coherent Multi-Document Summarization: Discrete Optimization of Multiple Neural Modeling Streams via Integer Linear Programming

One key challenge in multi-document summarization is the generated summary is often less coherent compared to single document summarization due to the larger heterogeneity of the input source content.In this work, we propose a generic framework to jointly consider coherence and informativeness in multi-document summarization and offers provisions to replace individual components based on the domain of source text.In particular, the framework characterizes coherence through verb transitions and entity mentions and takes advantage of syntactic parse trees and neural modeling for intra-sentential noise pruning.The framework cast the entire problem as an integer linear programming optimization problem with neural and non-neural models as linear components.We evaluate our method in the news and legal domains.The proposed approach consistently performs better than competitive baselines for both objective metrics and human evaluation.
TABS: Efficient Textual Adversarial Attack for Pre-trained NL Code Model Using Semantic Beam Search

As pre-trained models have shown successful performance in program language processing as well as natural language processing, adversarial attacks on these models also attract attention.However, previous works on blackbox adversarial attacks generated adversarial examples in a very inefficient way with simple greedy search.They also failed to find out better adversarial examples because it was hard to reduce the search space without performance loss.In this paper, we propose TABS, an efficient beam search black-box adversarial attack method.We adopt beam search to find out better adversarial examples, and contextual semantic filtering to effectively reduce the search space.Contextual semantic filtering reduces the number of candidate adversarial words considering the surrounding context and the semantic similarity.Our proposed method shows good performance in terms of attack success rate, the number of queries, and semantic similarity in attacking models for two tasks: NL code search classification and retrieval tasks.
GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks.It demonstrates that flat minima tend to imply better generalization abilities.However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs.In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective.We propose that the shift of the training and test distributions can be equivalently seen as a virtual parameter corruption or perturbation, which can explain why flat minima that are robust against parameter corruptions or perturbations have better generalization performances.On its basis, we propose a Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM) algorithm to help to learn algorithms find flat minima that generalize better.Results in various language benchmarks validate the effectiveness of the proposed GA-SAM algorithm on natural language tasks.
Open World Classification with Adaptive Negative Samples

Open world classification is a task in natural language processing with key practical relevance and impact.Since the open or unknown category data only manifests in the inference phase, finding a model with a suitable decision boundary accommodating for the identification of known classes and discrimination of the open category is challenging.The performance of existing models is limited by the lack of effective open category data during the training stage or the lack of a good mechanism to learn appropriate decision boundaries.We propose an approach based on adaptive negative samples (ANS) designed to generate effective synthetic open category samples in the training stage and without requiring any prior knowledge or external datasets.Empirically, we find a significant advantage in using auxiliary one-versus-rest binary classifiers, which effectively utilize the generated negative samples and avoid the complex threshold-seeking stage in previous works.Extensive experiments on three benchmark datasets show that ANS achieves significant improvements over stateof-the-art methods.
Post-Hoc Interpretation of Transformer Hyperparameters with Explainable Boosting Machines

Hyperparameter tuning is important for achieving high accuracy in deep learning models, yet little interpretability work has focused on hyperparameters.We propose to use the Explainable Boosting Machine (EBM), a glassbox method, as a post-hoc analysis tool for understanding how hyperparameters influence model accuracy.We present a case study on Transformer models in machine translation to illustrate the kinds of insights that may be gleaned, and perform extensive analysis to test the robustness of EBM under different data conditions.
STRUDEL: Structured Dialogue Summarization for Dialogue Comprehension

Abstractive dialogue summarization has long been viewed as an important standalone task in natural language processing, but no previous work has explored the possibility of whether abstractive dialogue summarization can also be used as a means to boost an NLP system's performance on other important dialogue comprehension tasks.In this paper, we propose a novel type of dialogue summarization task -STRUctured DiaLoguE Summarization (STRUDEL ) -that can help pre-trained language models to better understand dialogues and improve their performance on important dialogue comprehension tasks.In contrast to the holistic approach taken by the traditional free-form abstractive summarization task for dialogues, STRUDEL aims to decompose and imitate the hierarchical, systematic and structured mental process that we human beings usually go through when understanding and analyzing dialogues, and thus has the advantage of being more focused, specific and instructive for dialogue comprehension models to learn from.We further introduce a new STRUDEL dialogue comprehension modeling framework that integrates STRUDEL into a dialogue reasoning module over transformer encoder language models to improve their dialogue comprehension ability.In our empirical experiments on two important downstream dialogue comprehension tasks -dialogue question answering and dialogue response prediction -we demonstrate that our STRUDEL dialogue comprehension models can significantly improve the dialogue comprehension performance of transformer encoder language models.
Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping

Fine-tuning over large pretrained language models (PLMs) has established many stateof-the-art results.Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications.Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively fine-tuning layers in top-down manner is a promising solution.In this paper, we first point out that this method does not always work out due to different convergence speeds of different layers/modules.Inspired by this observation, we propose a simple componentwise gradient norm clipping method to adjust the convergence speed for different components.Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed and training stability.The codebase can be found at https://github.com/yangalan123/FineTuningStability.