Reviewerly/exHarmony huggingface dataset:
ASKSpell: Adaptive Surface Knowledge Enhances Tokens’ Semantic Representations for Chinese Spelling Check

Chinese Spelling Check (CSC) is a challenging task to detect and correct spelling errors in Chinese texts. Existing state-of-the-art methods try to use pre-trained language model and incorporate phonological or shape information as external knowledge. However, they neglect the importance of key tokens (i.e., misspellings, certain nouns or verbs, etc.) that might have great impacts on the semantics of other tokens and the sentence. They also neither fully utilize the surface knowledge of characters (i.e., their inherent features such as phonetic and shape). In this paper, a novel CSC framework for capturing adaptive surface knowledge (ASKSpell) is proposed, which consists of a PLM-based encoder for building token representations, a knowledge enhancing extractor for extracting surface knowledge, and a knowledge adjusting controller for focusing on key tokens. The extractor utilizes K-layer transformer blocks to extract surface knowledge from input embeddings, which can alleviate the impacts of errors. And the controller employs multi-head attention units to focus on key tokens by regulating each token’s surface knowledge which is fed to the backbone. Experiments conducted on three widely used benchmarks demonstrate that our method outperforms current existing models and achieves a new state-of-the-art.
KARN: Knowledge Augmented Reasoning Network for Question Answering

Question answering with information from pre-trained language models (LMs) and knowledge graphs (KGs) presents the following problems. First of all, some information in the knowledge graph is irrelevant to the question and answer. Then, the influence of edge information on node representation is ignored. To address these issues, we propose a Knowledge Augmented Reasoning Network (KARN), which utilizes external knowledge and contextual information to remove redundant entities, while enhancing the feature representation of nodes by optimizing the weight of edges. Specifically, we first use context and external knowledge to score the relevance of nodes and tasks, and then for nodes with low scores, we delete nodes and their related edges. Furthermore, we consider the node’s global information and external knowledge in the edge weight calculation, making the calculation of edge weight scores more comprehensive. We conduct experiments on three datasets in the fields of commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE), and the experimental results verify the effectiveness of our method.
Discourse Relation-Aware Multi-turn Dialogue Response Generation

Multi-turn dialogue response generation aims to generate a response with consideration of the context. It is not equal to multiple single-turn dialogues due to the context dependence of response. Many existing models achieve great success for response generation, but they still struggle to model the contextual semantics of dialogue history. Sequence models have difficulties to explore the interactive relations between contextual utterances, which affects the coherence of generated responses. To solve the issue, we propose a discourse relation-aware model, which encodes the contextual utterances with a directed acyclic graph neural network (DAGNN) with constraints on dialogue-specific discourse relations to better model the intrinsic structure. Besides, we introduce an auxiliary discourse relation recognition task to enhance the model’s ability of representing the context. Extensive experimental results show that our proposed model outperforms baselines.
COSYWA: Enhancing Semantic Integrity in Watermarking Natural Language Generation

With the increasing use of natural language generation (NLG) models, there is a growing need to differentiate between machine-generated text and natural language text. One promising approach is watermarking, which can help identify machine-generated text and protect against risks such as spam emails and academic dishonesty. However, existing watermarking methods can significantly affect the semantic meaning of the text, creating a need for more effective techniques that maintain semantic integrity. In this paper, we propose a novel watermarking method called COntextual SYnonym WAtermarking (COSYWA) that embeds watermarks in text using a Masked Language Model (MLM) without significantly impairing its semantics. Specifically, we use post-processing to embed watermarks in the output of an NLG model. We generate a context-based synonym set using an MLM model to embed watermark information and use statistical hypothesis testing to detect the existence of watermarking. Our experimental results show that COSYWA significantly enhances the text’s capacity to maintain its original meaning while effectively embedding a watermark, making it a promising approach for protecting against misinformation in NLG.
MCVIE: An Effective Batch-Mode Active Learning for Multi-label Text Classification

Data labeling for multi-label text is a challenging task in natural language processing, and active learning has emerged as a promising approach to reduce annotation effort while improving model performance. The primary challenge in multi-label active learning is to develop query strategies that can effectively select the most valuable unlabeled instances for annotation. Batch-mode active learning approaches, which select a batch of informative and diverse instances in each iteration, have been considered useful for improving annotation efficiency. However, challenges such as incomplete information ranking and high computational costs still hinder the progress of batch-mode methods. In this paper, we propose MCVIE, a novel batch-mode active learning method for multi-label text. MCVIE employs a two-stage active learning query strategy. Firstly, we combine two measures of prediction uncertainty and category vector inconsistency to calculate the basic information score for each example-label pair. Then, we use the Euclidean distance of text feature vectors to iteratively select diverse and informative example-label pairs for annotation. Experimental results on three benchmark datasets demonstrate that MCVIE outperforms other competitive methods.
An Adaptive Learning Method for Solving the Extreme Learning Rate Problem of Transformer

Transformer, a neural sequence model entirely based on attention, has achieved great success in natural language processing and become the de facto default model for multiple NLP tasks. Albeit its prevalence, the attention-based structure poses unmet challenges that the widely-used adaptive optimization methods, e.g., Adam, have serious difficulty in learning and often fail to converge if applied alone. In this work, we illustrate the problem that the adaptive optimization methods produce extremely-large learning rates that break the balance of stability. We further propose AdaMod, which smooths out extremely-large learning rates with adaptive and momental upper bounds on a per-parameter basis, instead of the uniform scaling in the warmup scheme. We empirically demonstrate AdaMod can improve the learning stability and bring significant improvements to the performance of Transformers and CNNs. Moreover, empirical results verify its effectiveness and robustness across different applications.
Bilevel Scheduled Sampling for Dialogue Generation

Exposure bias poses a common challenge in numerous natural language processing tasks, particularly in the dialog generation. In response to this issue, researchers have devised various techniques, among which scheduled sampling has proven to be an effective method for mitigating exposure bias. However, the existing state-of-the-art scheduled sampling methods solely consider the current sampling words’ quality for threshold truncation sampling, which overlooks the importance of sentence-level information and the method of threshold truncation warrants further discussion. In this paper, we propose a bilevel scheduled sampling model that takes the sentence-level information into account and incorporates it with word-level quality. To enhance sampling diversity and improve the model’s adaptability, we propose a smooth function that maps the combined result of sentence-level and word-level information to an appropriate range, and employ probabilistic sampling based on the mapped values instead of threshold truncation. Experiments conducted on the DailyDialog and PersonaChat datasets demonstrate the effectiveness of our proposed methods, which significantly alleviate the exposure bias problem and outperform state-of-the-art scheduled sampling methods.
Task-Consistent Meta Learning for Low-Resource Speech Recognition

We propose a new meta learning based framework that enhances previous approaches for low-resource speech recognition. Meta-learning has proven to be a powerful paradigm for transferring knowledge from prior tasks to facilitate the learning of a novel task. However, when faced with complex task environments and diverse task learning directions, averaging all task gradients is ineffective at capturing meta-knowledge. To address this challenge, we propose a task-consistent multilingual meta-learning (TCMML) method that adopts the gradient agreement algorithm to direct the model parameters in a direction where tasks have more consistency. If a task’s gradient matches the average gradient, its weight in meta-optimization is increased, and vice versa. Experiments on two datasets demonstrate that our proposed system can achieve comparable or even superior performance to state-of-the-art baselines on low-resource languages, and can easily combine with various meta learning methods.
PromptCL: Improving Event Representation via Prompt Template and Contrastive Learning

The representation of events in text plays a significant role in various NLP tasks. Recent research demonstrates that contrastive learning has the ability to improve event comprehension capabilities of Pre-trained Language Models (PLMs) and enhance the performance of event representation learning. However, the efficacy of event representation learning based on contrastive learning and PLMs is limited by the short length of event texts. The length of event texts differs significantly from the text length used in the pre-training of PLMs. As a result, there is inconsistency in the distribution of text length between pre-training and event representation learning, which may undermine the learning process of event representation based on PLMs. In this study, we present PromptCL, a novel framework for event representation learning that effectively elicits the capabilities of PLMs to comprehensively capture the semantics of short event texts. PromptCL utilizes a Prompt template borrowed from prompt learning to expand the input text during Contrastive Learning. This helps in enhancing the event representation learning by providing a structured outline of the event components. Moreover, we propose Subject-Predicate-Object (SPO) word order and Event-oriented Masked Language Modeling (EventMLM) to train PLMs to understand the relationships between event components. Our experimental results demonstrate that PromptCL outperforms state-of-the-art baselines on event related tasks. Additionally, we conduct a thorough analysis and demonstrate that using a prompt results in improved generalization capabilities for event representations (Our code will be available at https://github.com/YuboFeng2023/PromptCL ).
Exploiting Multiple Features for Hash Codes Learning with Semantic-Alignment-Promoting Variational Auto-encoder

Semantic hashing is an effective technique to empower information retrieval. Currently, considerable efforts have been dedicated to generating high-quality hash codes by modeling document features using generative models and other approaches. However, most of these methods rely solely on a single type of feature, such as TFIDF features, BERT embeddings, etc. As different types of features have distinct but complementary information of documents, e.g. TFIDF mainly contains the keywords information and BERT focuses on the semantics, hash codes generated solely from either may not capture the full essence of the documents. To overcome this challenge, we propose a semantic-alignment-promoting variational auto-encoder to generate hash codes from multiple document features. Specifically, a VAE-based generative model is first developed to model the multiple features. Then, we propose a semantic-alignment-promoting inference network to estimate the parameters of the variational posterior from multiple features. Additionally, the quality of hash codes is further improved by promoting the semantic alignment between the hash codes of connected documents in a constructed connection graph. The results of extensive experiments on three public datasets demonstrate that our proposed model significantly outperforms current state-of-the-art models.
Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task

With the increasing capabilities of large language models (LLMs), these high-performance models have achieved state-of-the-art results on a wide range of natural language processing (NLP) tasks. However, the models’ performance on commonly-used benchmark datasets often fails to accurately reflect their reliability and robustness when applied to real-world noisy data. To address these challenges, we propose a unified robustness evaluation framework based on the slot-filling task to systematically evaluate the dialogue understanding capability of LLMs in diverse input perturbation scenarios. Specifically, we construct a input perturbation evaluation dataset, Noise-LLM, which contains five types of single perturbation and four types of mixed perturbation data. Furthermore, we utilize a multi-level data augmentation method (character, word, and sentence levels) to construct a candidate data pool, and carefully design two ways of automatic task demonstration construction strategies (instance-level and entity-level) with various prompt templates. Our aim is to assess how well various robustness methods of LLMs perform in real-world noisy scenarios. The experiments have demonstrated that the current open-source LLMs generally achieve limited perturbation robustness performance. Based on these experimental observations, we make some forward-looking suggestions to fuel the research in this direction (The code is available at https://github.com/ZhaoJin-xu/A-Unified-Robustness-Evaluation-Fram ework-for-Noisy-Slot-Filling-Task ).
An Explicit-Memory Few-Shot Joint Learning Model

There are two difficulties in existing spoken language understanding models. The first problem is that it is difficult to extract the implicit relationship information between the intention and the slot in the utterance for the inference process, and the inference effect is not ideal; the second problem is that the training data is scarce, and the existing models cannot learn from the small amount of training data. Get more useful information. To address these two challenges, this paper proposes an Explicit-Memory Few-shot join learning model. In order to solve the first problem, a multi-layer model structure from coarse to fine is adopted to train the hidden semantic relationship and hidden state information between intentions and slots in the utterance; in order to solve the second problem, using the Siamese BERT metric learning method to jointly train the model. We use the Snips and ATIS datasets to train the model, and the test results show better results. In the case of a small amount of data, the model can also obtain stronger inference ability.
What Events Do Pre-trained Language Models Learn from Text? Probing Event-Based Commonsense Knowledge by Confidence Sorting

Recently, there are a lot of works trying to probe knowledge in pre-trained language models (PLMs). Most probing works use data in knowledge bases to create a “fill-in-the-blank” task form and probe entity knowledge in auto-encoding PLMs (e.g. BERT). Though these works have got success, their methods can not be applied to some complicated knowledge like event-based commonsense knowledge and other PLMs such as auto-regressive models (like GPT). In this paper, we develop a new knowledge probe based on confidence sorting and detect event-based commonsense knowledge with it. To make the probe suitable for different types of PLMs, we integrate different knowledge scoring methods with a new method called probability difference log-likelihood score (PDL) among them. Finally, we conduct extensive experiments on several representative PLMs, explore their commonsense abilities and analyze the factors that influence their performances.
RMGCN: Masked Graph Convolutional Networks for Relation-Aware Entity Alignment with Dangling Cases

Entity alignment (EA) is to match entities referring to identical real-world facts among different knowledge graphs (KGs). For simplicity, most previous work ignores the existence of dangling entities in the source KG. However, entity alignment task with dangling entities has emerged as a novel demand in real scenarios. Some work explores new dangling-aware loss functions and transfers existing methods on dangling settings, whose performance is still not satisfactory. Thus, in this work, we propose Relation-aware Masked Graph Convolutional Networks (RMGCN). In the learning stage, it can not only take advantage of the masking mechanism to alleviate the negative impact from nearby dangling neighbors, but also take both graph structure and relations of KGs into consideration. In the inference stage, it performs a two-step alignment which firstly filters out dangling entities, and then align the remaining entities. We adopt a novel distribution-based dangling entity detection method in first step to decrease the error propagation from dangling detection to the following EA task. The experimental results show that our model outperforms all state-of-the-art models with at least 7.5% on F1 score.
Creative Destruction: Can Language Models Interpret Oxymorons?

Oxymoron is a figurative language which combines seemingly contradictory words in a short phrase. It is used to create an impression, enhance a concept or entertain the readers. In this work, we propose a novel task named oxymoron interpretation, which requires destructing the contradiction and creating new senses to make the phrase comprehensible and harmonious with the context. We construct a dataset, Oxymoron with Context-Based Interpretation(OCBI) by gathering data from the internet and manually annotating it. Furthermore, some prevalent models and methods, including definition generation, GPT-2, T5, ChatGPT, LLaMA and Alpaca, are tested to explicitly generate interpretation. According to automatic and manual evaluation results, models except ChatGPT still struggle to generate plain and context-dependent interpretation, which brings us some insights of the capability gap between models. Our work reveals that oxymoron interpretation is a challenging task requiring common sense and contextual association ability. It can be used to assess how well language models comprehend figurative language. The OCBI dataset is publicly available ( https://github.com/pku0xff/oxymoron_interpretation ).
Global Prompt Cell: A Portable Control Module for Effective Prompt Tuning

As a novel approach to tuning pre-trained models, prompt tuning involves freezing the parameters in downstream tasks while inserting trainable embeddings into inputs in the first layer. However, previous methods have mainly focused on the initialization of prompt embeddings. The strategy of training and utilizing prompt embeddings in a reasonable way has become a limiting factor in the effectiveness of prompt tuning. To address this issue, we introduce the Global Prompt Cell (GPC), a portable control module for prompt tuning that selectively preserves prompt information across all encoder layers. Our experimental results demonstrate a 5.8% improvement on SuperGLUE datasets compared to vanilla prompt tuning.
SymCoNLL: A Symmetry-Based Approach for Document Coreference Resolution

Coreference resolution aims to identify the referential relationships between all entities in a text. However, traditional rule-based or hand-crafted feature-based methods encounter difficulties with long sentences and complex structures. Although many neural end-to-end coreference resolution models have been proposed, there are still challenges in resolving coreferences among multiple sentences due to weak contextual connections and ambiguity caused by distant mentions. To address these problems, we propose a symmetry-based method called SymCoNLL that studies coreference resolution from two aspects: mention identification and coreference prediction. Specifically, at the local level, our method focuses on mention identification based on semantic, syntactic, and mention type features, and improves coreference prediction by enhancing the similarity between mentions using semantic and mention type features. At the global level, it strengthens the internal connections of mentions in the same cluster using symmetry. We validate the effectiveness of our method in mention identification and coreference prediction through experiments on OntoNotes 5.0 dataset. The results show that our method significantly outperforms the baseline methods.
Probing Bilingual Guidance for Cross-Lingual Summarization

Bilingual guidance as an augmentation method is often applied in Seq2Seq cross-lingual summarization (CLS) to improve its performance. In this paper, we empirically explore bilingual guidance from the encoder/decoder side, as well as the perspective of representation alignment. We examine the respective benefits of their influence on CLS, and how this benefit varies with regard to guidance quality. Toward this, we devise a probing model to incorporate bilingual guidance, and construct a probing dataset containing bilingual document-summary pairs. We also comprehensively examine the representation alignment strategy for utilizing bilingual guidance in CLS. Empirical results robustly substantiate the impact of bilingual guidance on both the encoder and decoder aspects, and affirm the efficacy of representation alignment in the context of CLS. We also have some inspiring discoveries that encoder side guidance is more sensitive to guidance quality, and that representation alignment greatly benefits low-resource settings. Our code is publicly available at https://github.com/dwzhu-pku/BiGuid .
ECOD: A Multi-modal Dataset for Intelligent Adjudication of E-Commerce Order Disputes

With the widespread popularity of e-commerce websites, online order reviews and scores help customers to choose quality products and stores. At the same time, the order dispute problem has gradually attracted attention. Specifically, bad reviews with low scores can have negative impacts on the store, where some unfair and biased reviews mislead other consumers. In order to maintain their reputation, stores usually submit responses against bad reviews. In this paper, we creatively define an intelligent adjudication task of e-commerce order disputes, which aims to judge disputes fairly based on customer reviews and store responses. Moreover, we construct a multi-modal dataset about E-Commerce Order Disputes (ECOD). It contains 6,366 pairs of multi-modal reviews and responses. And each dispute has an adjudication label annotated by business experts. We evaluate the ECOD dataset with baseline models, and analyze the difficulties and challenges in detail. We believe that the proposed dataset will not only facilitate future research in the field of dispute adjudication and multi-modal understanding, but also advance intelligent management for e-commerce websites.