Reviewerly/exHarmony huggingface dataset:
A Relational Classification Network Integrating Multi-scale Semantic Features

Among many natural language processing tasks, the task of relation classification is an important basic work. Relational classification is required to provide high-quality corpus in fields such as machine translation, structured data generation, knowledge graphs, and semantic question answering. Existing relational classification models include models based on traditional machine learning, models based on deep learning, and models based on attention mechanisms. The above-mentioned models all have the disadvantage of only using a single feature for relationship classification, and do not effectively combine entity features and context features. Based on the above problems, this study proposes a relational classification network that integrates multi-scale semantic features. Through the combination of entity features and context semantic features, the model can better learn the relationship features between entities. In addition, in order to verify the validity of the model and the effectiveness of each module, this study conducted experiments on the SemEval-2010 Task 8 and KBP37 data sets. Experimental results demonstrate that the model performance is higher than most existing models.
QAE: A Hard-Label Textual Attack Considering the Comprehensive Quality of Adversarial Examples

Adversarial examples will induce errors in the target model under the condition that humans can hardly observe the changes between them and the original examples. But low-quality adversarial examples are easily identified and alerted to by humans, affecting the effectiveness of the attack. The comprehensive quality of the adversarial example is measured not only in terms of the degree of explicit perturbation and semantic similarity but also in terms of implicit textual grammatical errors and textual fluency. Existing hard-label attacks often generate candidates without paying attention to context, as a result, the generated adversarial examples are not semantic and syntactic, and the quality is poor. In this paper, we propose QAE, a hard-label attack based on a pre-trained masked language model and optimal example selection rules, to comprehensively improve the Quality of the Adversarial Examples. QAE uses a pre-trained masked language model to generate candidates that better match the semantic and syntactic rules of the context and constructs an adversarial example by random substitution. Then, it optimizes the adversarial example using a genetic algorithm that combines optimal example selection rules. Extensive experiments and human evaluation show that QAE can generate high-quality adversarial examples with better semantic fluency and fewer grammar errors while maintaining a similar attack success rate to the existing hard-label attacks.
NTAM: A New Transition-Based Attention Model for Nested Named Entity Recognition

Traditional Named Entity Recognition (NER) research only deals with flat entities and ignores nested entities. The transition-based method maps a sentence to a designated forest to recognize nested entities by predicting an action sequence through a state transition system which includes transition actions and a state of structures. However, the subsequent transition actions are affected by the previous transition actions resulting in error propagation, and the method ignores the correlation between the structures. To tackle these issues, we propose a new transition-based attention model (NTAM) to recognize nested entities. First, the structures and transition actions of the state transition system are redefined to eliminate error propagation. The prediction of an action sequence is converted to the prediction of a series of states, which predict whether the words between the structures can form entities. Second, we introduce an attention mechanism that strengthens the association between the structures. Experiments on two public nested NER datasets outperform previous state-of-the-art models.
Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base

Pre-trained language models (PLMs) like BERT have made significant progress in various downstream NLP tasks. However, by asking models to do cloze-style tests, recent work finds that PLMs are short in acquiring knowledge from unstructured text. To understand the internal behaviour of PLMs in retrieving knowledge, we first define knowledge-baring (K-B) tokens and knowledge-free (K-F) tokens for unstructured text and ask professional annotators to label some samples manually. Then, we find that PLMs are more likely to give wrong predictions on K-B tokens and attend less attention to those tokens inside the self-attention module. Based on these observations, we develop two solutions to help the model learn more knowledge from unstructured text in a fully self-supervised manner. Experiments on knowledge-intensive tasks show the effectiveness of the proposed methods. To our best knowledge, we are the first to explore fully self-supervised learning of knowledge in continual pre-training.
Medical Report Generation Based on Segment-Enhanced Contrastive Representation Learning

Automated radiology report generation has the potential to improve radiology reporting and alleviate the workload of radiologists. However, the medical report generation task poses unique challenges due to the limited availability of medical data and the presence of data bias. To maximize the utility of available data and reduce data bias, we propose MSCL (Medical image Segmentation with Contrastive Learning), a framework that utilizes the Segment Anything Model (SAM) to segment organs, abnormalities, bones, etc., and can pay more attention to the meaningful ROIs in the image to get better visual representations. Then we introduce a supervised contrastive loss that assigns more weight to reports that are semantically similar to the target while training. The design of this loss function aims to mitigate the impact of data bias and encourage the model to capture the essential features of a medical image and generate high-quality reports. Experimental results demonstrate the effectiveness of our proposed model, where we achieve state-of-the-art performance on the IU X-Ray public dataset.
Towards Malay Abbreviation Disambiguation: Corpus and Unsupervised Model

Abbreviation disambiguation constitutes a highly crucial natural language processing task in all languages, including Malay. Its objective involves the identification of the most suitable definition, from a candidate set of definitions, that corresponds to a given abbreviation based on contextual information. The current state of research on Malay abbreviation disambiguation is hindered by the absence of an extensive database of abbreviations, thus posing difficulties in supporting model training. Simultaneously, the challenge lies in developing a Malay abbreviation disambiguation model that can achieve a satisfactory level of restoration performance even in the absence of annotated samples, thereby facilitating enhanced comprehension of literature among individuals. Consequently, the lack of a large-scale abbreviation database and the construction of an effective disambiguation model without annotated samples present ongoing challenges in the field of Malay abbreviation disambiguation. To address the above issues, we construct a dataset of Malay abbreviations and propose an unsupervised method based on a pre-trained model to solve the problem of abbreviation disambiguation. This method sorts out the perplexity score of each definition according to the definition corresponding to the abbreviation in the same sentence. Subsequently, the definition associated with the lowest perplexity score is selected as the most suitable choice. On the constructed Malay dataset, our method exhibits a mere 3% decrease in accuracy compared to the current state-of-the-art (SOTA) supervised approach, thereby showcasing a remarkable advantage within the domain of unsupervised methods. Notably, in the SDU@AAAI-22-Shared Task 2: Acronym Disambiguation, our experimental results demonstrate effectiveness across all four test sets. Particularly, the performance is exceptionally notable in the context of legal English, achieving an accuracy rate of 77.28%. The source code and dataset of this paper is publicly available at https://github.com/bhysss/TMAD-CUM .
Promoting Open-Domain Dialogue Generation Through Learning Pattern Information Between Contexts and Responses

Recently, utilizing deep neural networks to build the open-domain dialogue models has become a hot topic. However, the responses generated by these models suffer from many problems such as responses not being contextualized and tend to generate generic responses that lack information content, damaging the user’s experience seriously. Therefore, many studies try introducing more information into the dialogue models to make the generated responses more vivid and informative. Unlike them, this paper improves the quality of generated responses by learning the implicit pattern information between contexts and responses in the training samples. In this paper, we first build an open-domain dialogue model based on the pre-trained language model (i.e., GPT-2). And then, an improved scheduled sampling method is proposed for pre-trained models, by which the responses can be used to guide the response generation in the training phase while avoiding the exposure bias problem. More importantly, we design a response-aware mechanism for mining the implicit pattern information between contexts and responses so that the generated replies are more diverse and approximate to human replies. Finally, we evaluate the proposed model (RAD) on the Persona-Chat and DailyDialog datasets; and the experimental results show that our model outperforms the baselines on most automatic and manual metrics.
Span-Based Pair-Wise Aspect and Opinion Term Joint Extraction with Contrastive Learning

Pair-wise aspect and opinion term extraction (PAOTE) is a subtask of aspect-based sentiment analysis (ABSA), aiming at extracting aspect terms (AT) and corresponding opinion terms (OT) in the review sentences in the form of aspect-opinion pairs (AO), which provides global profiles about products and services for users. Existing methods cannot identify the boundary information of AT and OT precisely. In addition, pairing errors are easily caused if there are multiple AT and OT in a sentence. In our work, to address above limitations, we design a span-based joint extraction framework with contrastive learning (SJCL) to enhance both term extraction and pairing in PAOTE. For term extraction, we utilize span-based convolutional neural network (CNN) to merge contextual syntactic and semantic features to identify the boundaries of aspect and opinion terms precisely and combine contrastive learning (CL) to enhance the distinctiveness of different types of terms. For term pairing, we prune the original dependency trees according to part-of-speech (POS) information to remove insignificant noise information and leverage graph convolutional network (GCN) to learn pairing features, then negative sampling and contrastive learning are used to avoid mismatched aspect-opinion pairs. Experimental results on four public datasets certify the state-of-the-art performance of our model.
CD-BLI: Confidence-Based Dual Refinement for Unsupervised Bilingual Lexicon Induction

Unsupervised bilingual lexicon induction is a crucial and challenging task in multilingual NLP, which aims to induce word translation by aligning monolingual word embeddings. Existing works treat word pairs equally and lack consideration of word pair credibility, which further leads to the fact that these methods are limited to global operations on static word embeddings and fail when using pre-trained language models. To address this problem, we propose confidence-based dual refinement for unsupervised bilingual lexicon induction, where embeddings are refined by dual aspects (static word embeddings and pre-trained models) based on the confidence of word pairs, i.e., the credibility of word pairs in correct alignment. For static word embeddings, instead of global operations, we calculate personalized mappings for different words based on confidence. For pre-trained language models, we fine-tune the model with positive and negative samples generated according to confidence. Finally, we combine the output of both aspects as the final result. Extensive experimental results on public datasets, including both rich-resource and low-resource languages, demonstrate the superiority of our proposal.
A Text-Image Pair Is Not Enough: Language-Vision Relation Inference with Auxiliary Modality Translation

The semantic relations between language and vision modalities become more and more vital since they can effectively facilitate downstream multi-modal tasks. Although several approaches have been proposed to handle language-vision relation inference (LVRI), they normally rely on the limited information of the posted text-image pair. In this paper, to extend the information width of the original input, we introduce a concept of modality translation and propose the auxiliary modality translation framework (AMT) for LVRI. Specifically, the original input and the text pair (original and generated) are passed into two separate multi-layer bidirectional transformer structures respectively. The different linguistic and visual hybrid features are extracted and subsequently feed into a feature fusion module followed by a classifier. Systematic experiments and extensive analysis demonstrate the effectiveness of our approach with auxiliary modality translation.
Learning Well-Separated and Representative Prototypes for Few-Shot Event Detection

Event detection can be solved with two subtasks: identification and classification of trigger words. Depending on whether these two subtasks are handled simultaneously, event detection models are divided into the pipeline-based paradigm and the joint-based paradigm. For both paradigms, prototypical networks are an extremely important method to solve a few-shot event detection task. Prototypical networks classify through the distances between instances and prototypes. Therefore, both the distance between prototypes and prototype representations are the key to prototypical networks. Compared with the pipeline-based paradigm, the joint-based paradigm handles two subtasks at the same time to effectively avoid the error cascade. To make better use of prototypical networks in the joint-based paradigm, we propose PN-SR, a Prototypical Network with well-Separated and Representative prototypes. Specifically, we use representation learning methods to generate prototypes. At the same time, we add a loss factor to further enlarge the distances between prototypes of different classes and reduce semantic representation distances of instances with the same class. Experiments show our method outperforms many baselines by at least 26.15% F1 points under the same few-shot settings and achieves SOTA.
Emotionally-Bridged Cross-Lingual Meta-Learning for Chinese Sexism Detection

Sexism detection remains as an extremely low-resource task for most of the languages including Chinese. To address this issue, we propose a zero-shot cross-lingual method to detect sexist speech in Chinese and perform qualitative and quantitative analyses on the data we employed. The proposed method aims to explicitly model the knowledge transfer process from rich-resource language to low-resource language using metric-based meta-learning. To overcome the semantic disparity between various languages caused by language-specific biases, a common label space of emotions expressed across languages is used to integrate universal emotion features into the meta-learning framework. Experiment results show that the proposed method improves over the state-of-the-art zero-shot cross-lingual classification methods.
UKT: A Unified Knowledgeable Tuning Framework for Chinese Information Extraction

Large Language Models (LLMs) have significantly improved the performance of various NLP tasks. Yet, for Chinese Information Extraction (IE), LLMs can perform poorly due to the lack of fine-grained linguistic and semantic knowledge. In this paper, we propose Unified Knowledgeable Tuning (UKT), a lightweight yet effective framework that is applicable to several recently proposed Chinese IE models based on Transformer. In UKT, both linguistic and semantic knowledge is incorporated into word representations. We further propose the relational knowledge validation technique in UKT to force model to learn the injected knowledge to increase its generalization ability. We evaluate our UKT on five public datasets related to two major Chinese IE tasks. Experiments confirm the effectiveness and universality of our approach, which achieves consistent improvement over state-of-the-art models.
EP-Transformer: Efficient Context Propagation for Long Document

Nowadays, transformers are widely used in NLP applications. However, since the model complexity increases quadratically with sequence length, it is intractable to use transformers on very long documents. In this paper, we propose EP-Transformer, a hierarchical Transformer concentrating on efficient context propagation for the whole long documents. Specifically, two components are designed: similarity-sensitive fusion block and unsupervised contrast learning strategy, the former is dedicated to the multi-level aggregation of representation between segments, and the latter aims to generate more unbiased global features. EP-Transformer not only enables effectively capturing local and global contextual information of long sequence, but also reduces the computational complexity. In order to verify the effectiveness of EP-Transformer on long sequences, we evaluate it on three commonly used classification datasets (IMDB, Hyperpartisan news, and Arxiv) and two QA datasets (WikiHop and TriviaQA). Experimental results demonstrate that EP-Transformer significantly outperforms other baseline models.
Prompt-Free Few-Shot Learning with ELECTRA for Acceptability Judgment

Few-shot learning remains a great challenge for the task of acceptability judgment that identifies whether a sentence is acceptable or unacceptable. In this paper, we propose a prompt-free learning approach, namely PF-ELECTRA, to few-shot acceptability judgment. First, we leverage a pre-trained token replaced detection model, ELECTRA, as our basic few-shot learner to deal with the challenge of data distribution difference. Second, we design a prompt-free few-shot learning strategy that uses both the maximal unacceptability score for a single token and the overall unacceptability score for the whole sentence to judge the acceptability. Empirical studies validate the effectiveness of PF-ELECTRA on challenging few-shot acceptability judgment. To the best of our knowledge, it is the first work that improves the performance of few-shot acceptability judgment based on standard fine-tuning.
Collective Entity Linking with Joint Subgraphs

The problem of collective methods based on graph presents challenges: given mentions and their candidate entities, methods need to build graphs to model correlation of linking decisions between different mentions. In this paper, we propose three ideas: (i) build subgraphs made up of partial mentions instead of those in the entire document to improve computation efficiency, (ii) perform joint disambiguation over context and knowledge base (KB), and (iii) identify closely related knowledge from KB. With regard to above innovations, we propose EL-Graph, which addresses the challenges of collective methods: (i) attention mechanism, where we select low attention scores of partial mentions of a document to form a subgraph to improve the computation efficiency, (ii) joint disambiguation, where we connect mention context and KB to form a joint graph, gather and spread the message to update their node representations simultaneously through graph neural networks, and (iii) relevance scoring, where we compute similarity to estimate importance of KB nodes relative to the given context. We evaluate our model on publicly available dataset and show the effectiveness of our model.
A Frustratingly Easy Improvement for Position Embeddings via Random Padding

Position embeddings, encoding the positional relationships among tokens in text sequences, make great contributions to modeling local context features in Transformer-based pre-trained language models. However, in Extractive Question Answering, position embeddings trained with instances of varied context lengths may not perform well as we expect. Since the embeddings of rear positions are updated fewer times than the front position embeddings, the rear ones may not be properly trained. In this paper, we propose a simple but effective strategy, Random Padding, without any modifications to architectures of existing pre-trained language models. We adjust the token order of input sequences when fine-tuning, to balance the number of updating times of every position embedding. Experiments show that Random Padding can significantly improve model performance on the instances whose answers are located at rear positions, especially when models are trained on short contexts but evaluated on long contexts. Our code and data will be released for future research.
Enriching Semantic Features for Medical Report Generation

Medical reports play an important role in diagnosing a patient’s illness. However, writing medical reports is time-consuming and labor-intensive, and writing high-quality medical reports often requires extensive clinical experience, which can be a challenge for less experienced doctors. Based on this, the medical report generation tasks emerged. The model architecture used is basically Encoder-Decoder architecture. In this architecture, the extraction quality of visual features is particularly critical. Based on this, previous research has focused on how to better extract visual features, but most researchers have neglected the acquisition of textual features. In this paper, we focus on how to obtain better textual features and how to perform multi-modal feature fusion of visual and textual features more effectively. To this end, we propose two modules, the BM25 Multi-Source Text feature interaction module (BM25-MST) and the Multi-Modal Feature memory fusion module (Mul-MF),respectively. The aim of this BM25-MST module is to generate a multi-source text feature containing rich semantic information, and then use the Mul-MF module to perform full fusion of visual features and text features.
Joint Cross-Domain and Cross-Lingual Adaptation for Chinese Opinion Element Extraction

Opinion element extraction, through which key elements of opinions are identified, is critical for understanding opinions. We can achieve state-of-the-art performance by leveraging supervised learning backended by pretrained language models for the task. Yet, it relies heavily on a manual-annotated training corpus. This corpus could be biased towards specific domains, as well as scarce for non-English languages like Chinese. In this work, we focus on the Chinese language, investigating the joint exploration of cross-domain and cross-lingual adaptation. To facilitate our research, we annotate a training corpus of the COVID-19 genre for cross-domain exploration. For the cross-lingual exploration, we utilize the English newsware MPQA corpus directly. Finally, we design a uniform architecture to incorporate cross-domain and cross-lingual transfer jointly. Experimental results demonstrate that cross-domain and cross-lingual transfers can both deliver impressive performance, and their combination can increase the performance even further.