Reviewerly/exHarmony huggingface dataset:
Multi-angle Prediction Based on Prompt Learning for Text Classification

The assessment of Chinese essays with respect to text coherence using deep learning has been relatively understudied due to the lack of large-scale, high-quality discourse coherence evaluation data resources. Existing research predominantly focuses on characters, words, and sentences, neglecting automatic evaluation of Chinese essays based on articles’ coherence. This paper aims to research automatic evaluation of Chinese essays based on articles’ coherence by leveraging some data from LEssay, a Chinese essay coherence evaluation dataset jointly constructed by the CubeNLP laboratory of East China Normal University and Microsoft. The coherence of Chinese essays is primarily evaluated based on two big aspects: 1. The smoothness of logic (the appropriateness of using related words, and the appropriateness of logical relationship between contexts) 2. The reasonableness of sentence breaks (how well punctuation is used and how well the sentence is structured). Therefore, in this paper, we adopt prompt learning and cleverly design a multi-angle prediction prompt template that can realize the assessment of the coherence of Chinese essay from four angles. During the inference stage, the prediction of the coherence of Chinese essays is obtained through the multi-angle prediction template and voting mechanism. Notably, the proposed method demonstrates excellent results in the NLPCC2023 SharedTask7 Track1.
A Unified Framework for Optimizing Video Corpus Retrieval and Temporal Answer Grounding: Fine-Grained Modality Alignment and Local-Global Optimization

Present advancements in digital content have resulted in an enhanced interest in video understanding. The Temporal Answer Grounding in Video Corpus (TAGVC) aims to pinpoint the visual response within an extensive array of untrimmed instructional videos using language-based questions. This research explores TAGVC, a notably complex task involving an intricate combination of skills including video retrieval and comprehension, visual answer localization, and collaboration between vision and language, posing challenges greater than the initial Temporal Answer Grounding in a Single Video (TAGSV). This paper outlines a novel approach to tackling such challenges, proposing a Fine-grained Modality Alignment and Local-Global Optimization Framework(FMALG) for TAGVC. By combining the strengths of visual and textual predictions, this system offers a resilient solution. The fine-grained modality alignment is used to understand each video segment’s context succinctly. In addition, the local-global optimization technique is implemented to learn the global retrieval capabilities and visualize answer localization. The subtitle quality is also improved using OpenAI’s ChatGPT. The efficacy of the proposed methods is evidenced through extensive experiments, where we achieved first place on track 3 and second place on track 2.
Improving Conversational Aspect-Based Sentiment Quadruple Analysis with Overall Modeling

In this paper, we describe the experimental schemes of Team HLT-base for NLPCC-2023-Shared-Task-4 Conversational Aspect-based Sentiment Quadruple Analysis (ConASQ). Different from the aspect-based sentiment quadruple analysis task, the ConASQ task requires modeling the relationship between different utterances in context. Previous works commonly apply the attention mechanism (e.g., self-attention, transformer) to model the interaction of different utterances after extracting the feature of each utterance. However, this approach may not capture the interaction of different utterances effectively with a single self-attention layer or a transformer layer. To address this issue, we propose a simple and efficient method in this paper. Specially, we concatenate all utterances as a single sentence and feed this sentence into the pre-trained model, which can better construct the representation of utterances from scratch. Then, we utilize different mask matrices to model the features of dialogue threads, speakers, and replies. Finally, we apply the gird-tagging method to quadruple extraction. Extensive experimental results show that our proposed framework outperforms other competitive methods and achieves 2nd performance in the ConASQ competition.
Multi-task Biomedical Overlapping and Nested Information Extraction Model Based on Unified Framework

Biomedical information extraction technology can mine crucial biomedical knowledge from biomedical literature automatically, which plays an important role in constructing knowledge graphs and databases. Biomedical named entity recognition and event detection are crucial tasks in biomedical information extraction. Previous research mainly focused on building models for single tasks, with low generalization ability. Moreover, most prior work neglected the detection of overlapping events and the recognition of nested entities. Although some models can solve these problems, they suffer from low efficiency and poor performance. Therefore, we propose a unified framework multi-task MBONIEUF model for biomedical entity recognition and event detection. Our model converts sequence labeling problems into machine reading comprehension problems, first using ChatGPT to generate semantically rich questions based on the biomedical corpus labels, and then encoding the concatenated generated questions and original sentences using PubMedBERT. Furthermore, BiGRU, Biaffine attention module, and IDCNN module are designed as the feature extraction layer, used to capture the complex interactions between token pairs and the features between events and entities. In the decoding layer, a stacked classification pointer matrix and a multi-head nested query matrix are learned, and a corresponding detection algorithm is designed to decode the entities and events in the two matrices. We propose a model that can handle both biomedical named entity recognition and event detection tasks. Compared with traditional methods, our model not only improves the model’s task generalization ability but also performs better in solving the overlapping and nested problems in biomedical information extraction. On the MLEE, BioNLP’09, and GENIA datasets, the proposed model achieves good performances with F1 scores of 81.62%, 75.76%, and 77.75% respectively.
Unsupervised Clustering for Negative Sampling to Optimize Open-Domain Question Answering Retrieval

Open-domain question answering(ODQA), as a rising question answering task, has attracted attention of many researchers due to its large number of information sources from various fields and can be applied in search engines and intelligent robots. ODQA relies heavily on the information retrieval task. Previous research has mostly focused on the accuracy of open-domain retrieval. However, in practical applications, the convergence speed of ODQA retrieval task training is also important because it affects the generalization ability of ODQA retrieval tasks on new datasets. This paper proposes an unsupervised clustering negative sampling method to improve the convergence speed and retrieval performance of the model by changing the distribution of negative samples in contrastive learning. Experiments show that, the method improves the convergence speed of the model and achieves 5.3% and 2.2% higher performance on two classic open-domain question answering datasets compared to the random negative sampling baseline model. At the same time, the gap statistics method is introduced to find the most suitable number of clusters for open-domain question answering retrieval tasks, reducing the difficulty of using the method.
Enhancing In-Context Learning with Answer Feedback for Multi-span Question Answering

Whereas the emergence of large language models (LLMs) like ChatGPT has exhibited impressive general performance, it still has a large gap with fully-supervised models on specific tasks such as multi-span question answering. Previous researches found that in-context learning is an effective approach to exploiting LLM, by using a few task-related labeled data as demonstration examples to construct a few-shot prompt for answering new questions. A popular implementation is to concatenate a few questions and their correct answers through simple templates, informing LLM of the desired output. In this paper, we propose a novel way of employing labeled data such that it also informs LLM of some undesired output, by extending demonstration examples with feedback about answers predicted by an off-the-shelf model, e.g., correct, incorrect, or incomplete. Experiments on three multi-span question answering datasets as well as a keyphrase extraction dataset show that our new prompting strategy consistently improves LLM’s in-context learning performance.
Unraveling viscoelastic properties of Philippine Mahogany and manufactured acoustic guitar for improving acoustic performance of materials

The acoustic conversion efficiency is one of the parameters used to assess the acoustic performance of a material, which is dependent on the values of tan delta and complex modulus. Studies so far have been conducted looking at tan delta and complex modulus [1], but the specific components of these, which are the storage and loss modulus, and how it affects the tan delta and complex modulus, and therefore acoustic conversion efficiency has yet been studied. Moreover, the effect of heat treatment on the storage and loss modulus, which is more affected by the process have yet been explored. This study aims to characterize the dynamic mechanical properties of Philippine Mahogany, to gain some insights on tuning the acoustic performance of materials for musical instruments, as well as to gain insights on the effect of heat treatment on these properties, composition and to compare it to that of a manufactured guitar. Dynamic Mechanical Analysis (DMA) and Thermogravimetric analysis (TGA) were used to accomplish these objectives. The DMA results showed that storage modulus, loss modulus, complex modulus, and tan delta decreased upon heat treatment, while the sample from guitar showed highest storage modulus, loss modulus, and complex modulus, and lowest tan delta. The TGA results showed minute differences in composition between treated and untreated samples. However, this provided explanation on the difference in the mechanical properties of the samples. Noteworthy, even with tan delta <0.1 for both samples, storage modulus wasn’t dominating the tan delta for both. This study showed the importance of looking at the storage and loss modulus components in improving the acoustic performance of materials as well as the effect of heat treatment on Philippine Mahogany.
An ensemble learning approach for anomaly detection in credit card data with imbalanced and overlapped classes

Electronic payment methods have become increasingly popular for business transactions, both online and in-person, across the globe. Anomalies like online fraud and default payments, which can result in substantial financial losses, have become more common as the usage of credit cards in online purchases has increased. To address this issue, researchers have explored various machine learning models and their ensemble techniques for detecting anomalies in credit card transaction data. However, detecting anomalies in this data can be challenging due to overlapping class samples and an imbalanced class distribution. Therefore, the detection rate of anomalies from minority class samples is relatively low, and general learning algorithms can be biased towards the majority class samples. In this paper, we propose a model called Credit Card Anomaly Detection (CCAD) that leverages the base learners paradigm and meta-learning ensemble techniques to improve the detection rate of credit card anomalies. We utilize four outlier detection algorithms as base learners and XGBoost algorithm as meta learner in the proposed stacked ensemble approach to detect anomaly in credit card transactions. We apply stratified sampling technique and k-fold cross-validation process to address the issues of data imbalance and overfitting. In addition, the discordance rate is calculated to enhance the accuracy of ensemble learning performances. The proposed model is trained and tested using two datasets: CCF (Credit Card Fraud) and CCDP (Credit Card Default Payment). Experimental results demonstrate that our approach outperforms existing approaches, particularly in detecting anomalies from the minority class instances of these datasets.
DSLA 2023: Facing the dangers of data illiteracy

Abstract The Data Science and Literacy Act (DSLA) of 2023 aims to boost data literacy and data science education across the USA. Gregg M. Gascon, Katherine K. Wallman, Sridhar Ravula, Joseph Cappelleri, Julia Lee, Kristian Lum, Chaitra H. Nagaraja, Sudeshna Paul, Dooti Roy and Edward Wu explain why this opportunity demands our attention
rT5: A Retrieval-Augmented Pre-trained Model for Ancient Chinese Entity Description Generation

Ancient Chinese, the natural language of ancient China, serves as the key to understanding and propagating Chinese rich history and civilization. However, to facilitate comprehension and education, human experts previously need to write modern language descriptions for special entities, such as persons and locations, out of ancient Chinese texts. This process requires specialized knowledge and can be time-consuming. To address these challenges, we propose a new task called Ancient Chinese Entity Description Generation (ACEDG), which aims to automatically generate modern language descriptions for ancient entities. To address ACEDG, we propose two expert-annotated datasets, XunZi and MengZi, each containing ancient Chinese texts, and some of them have been annotated with entities and their descriptions by human experts. To leverage both labeled and unlabeled texts, we propose a retrieval-augmented pre-trained model called rT5. Specifically, a pseudo-parallel corpus is constructed using retrieval techniques to augment the pre-training stage. Subsequently, the pre-trained model is fine-tuned on our high-quality human-annotated entity-description corpus. Our experimental results, evaluated using various metrics, demonstrate the effectiveness of our method. By combining retrieval techniques and pre-training, our approach significantly advances the state-of-the-art performance in the ACEDG task compared with strong pre-trained models.
Punctuation Matters! Stealthy Backdoor Attack for Language Models

Recent studies have pointed out that natural language processing (NLP) models are vulnerable to backdoor attacks. A backdoored model produces normal outputs on the clean samples while performing improperly on the texts with triggers that the adversary injects. However, previous studies on textual backdoor attack pay little attention to stealthiness. Moreover, some attack methods even cause grammatical issues or change the semantic meaning of the original texts. Therefore, they can easily be detected by humans or defense systems. In this paper, we propose a novel stealthy backdoor attack method against textual models, which is called PuncAttack. It leverages combinations of punctuation marks as the trigger and chooses proper locations strategically to replace them. Through extensive experiments, we demonstrate that the proposed method can effectively compromise multiple models in various tasks. Meanwhile, we conduct automatic evaluation and human inspection, which indicate the proposed method possesses good performance of stealthiness without bringing grammatical issues and altering the meaning of sentences.
Towards Making the Most of LLM for Translation Quality Estimation

Machine Translation Quality Estimation (QE) aims to evaluate the quality of machine translation without relying on references. Recently, Large-scale Language Model (LLM) has made major breakthroughs, and has shown excellent zero-shot ability on various natural language processing tasks. However, its application on QE is non-trivial and has not yet been explored. In this work, we aim to exploit the translation estimation ability of LLM, and propose an unsupervised QE framework via exploring the useful information that can be extracted from the LLM. We firstly formulate QE in a machine translation template, and derive the sequence-level probabilities as the translation estimation result. Moreover, we exploit the uncertainty of LLM as another QE evidence, by randomize the LLM with different demonstrations and prompts, and obtain the variance. We evaluate our method on WMT’22 QE data, and achieve high correlation with human judgments of quality, rivalling state-of-the-art supervised QE models. We also provide in-detailed analysis on the ability of LLM on QE task.
Positive-Guided Knowledge Distillation for Document-Level Relation Extraction with Noisy Labeled Data

Since one entity may have multiple mentions and relations between entities may stretch across multiple sentences in a document, the annotation of document-level relation extraction datasets becomes a challenging task. Many studies have identified that these datasets contain a large number of noisy labels, hindering performance improvement for the document-level relation extraction task. The previous and most straightforward method is denoising noisy labeled data from a data annotation perspective. However, this time-consuming approach is not suitable for large-scale datasets. In this paper, we propose a novel Positive-Guided Knowledge Distillation (PGKD) model to address the noisy labeled data problem for document-level relation extraction. We design a new teacher-student architecture. The teacher model trained with only positive samples can partially supervise the student model. The positive-guided knowledge distillation algorithm transfers the clean positive-class patterns from the teacher model to the student model. In this way, the student model trained with all samples can efficiently prevent the interference of false negative samples. Extensive experiments on Mix-DocRED demonstrate that PGKD achieves state-of-the-art effectiveness for document-level relation extraction with noisy labeled data. Moreover, PGKD also surpasses other baselines even on the well-annotated Re-DocRED.
Biomedical Entity Normalization Using Encoder Regularization and Dynamic Ranking Mechanism

Biomedical entity normalization is a fundamental method for lots of downstream applications. Due to the rich additional information for biomedical entities in medical dictionaries, such as synonyms or definitions, transformer-based models are applied to dig semantic representations in normalization recently. Despite the high performance of the transformer-based model, the over-fitting problem remains challenging and unsolved. Besides, bi-encoder structure and cross-encoder structure are popularly applied in many biomedical entity normalization works, the issue to measure the distance of such encoder structures is very challenging. Moreover, the triples margin ranking loss mechanism is widely used in reranking stage of entity normalization. In this paper, we proposed an encoder-level regularization to restrain the over-fitting problem caused by the deep representation of transformer. Moreover, we use a dynamic margin ranking mechanism instead of fixed margin selection in reranking stage during training. In this way, we experiment our model on three biomedical entity normalization datasets, and the empirical results outperform previous state-of-the-art models.
Bounding and Filling: A Fast and Flexible Framework for Image Captioning

Most image captioning models following an autoregressive manner suffer from significant inference latency. Several models adopted a non-autoregressive manner to speed up the process. However, the vanilla non-autoregressive manner results in subpar performance, since it generates all words simultaneously, which fails to capture the relationships between words in a description. The semi-autoregressive manner employs a partially parallel method to preserve performance, but it sacrifices inference speed. In this paper, we introduce a fast and flexible framework for image captioning called BoFiCap based on bounding and filling techniques. The BoFiCap model leverages the inherent characteristics of image captioning tasks to pre-define bounding boxes for image regions and their relationships. Subsequently, the BoFiCap model fills corresponding words in each box using two-generation manners. Leveraging the box hints, our filling process allows each word to better perceive other words. Additionally, our model offers flexible image description generation: 1) by employing different generation manners based on speed or performance requirements, 2) producing varied sentences based on user-specified boxes. Experimental evaluations on the MS-COCO benchmark dataset demonstrate that our framework in a non-autoregressive manner achieves the state-of-the-art on task-specific metric CIDEr (125.6) while speeding up 9.22 $$\times $$ than the baseline model with an autoregressive manner; in a semi-autoregressive manner, our method reaches 128.4 on CIDEr while a 3.69 $$\times $$ speedup.
Mixture-of-Experts for Biomedical Question Answering

Biomedical Question Answering (BQA) has attracted increasing attention in recent years. It is a challenging task because biomedical questions are professional and usually vary widely. Existing question answering methods answer all questions with a homogeneous model, leading to various types of questions competing for the shared parameters, which will confuse the model decision for each single type of question. In this paper, in order to alleviate the parameter competition problem, we propose a Mixture-of-Experts (MoE) based question answering method called MoEBQA that decouples the computation for different types of questions by sparse routing. To be specific, we split a pretrained Transformer model into the bottom and the top blocks. The bottom blocks are shared by all the examples, aiming to capture the general features. The top blocks are extended to an MoE version that consists of a series of independent experts, where each example is assigned to a few experts according to its underlying question type. MoEBQA automatically learns the routing strategy in an end-to-end manner so that each expert tends to deal with a subset of questions it is expert in. We evaluate MoEBQA on three BQA datasets constructed based on real examinations. The results show that within a tolerable computational overhead, our MoE extension significantly improves the accuracy of existing question answering models by 9.4% on average. In addition, we elaborately analyze our MoE modules to reveal how MoEBQA works and find that it can automatically group the questions into human-readable clusters. The code is available at https://github.com/Hunter-DDM/moebqa .
Improve the Diversity and Novelty for Open-Ended Neural Text Generation via Inverse Probability Weighting

Stochastic sampling methods for open-ended neural text generation severely affect the diversity and novelty of generated texts. Traditional sampling methods only prune the low-likelihood part of the predicted distribution to trade-off between the fluency and diversity of generated texts. They do not manipulate the high-likelihood part, which leads to the likelihood trap that induces low diversity and boredom. They also do not directly leverage that human does not always favor high-likelihood texts. Inspired by these, we propose a novel sampling method that rescales the high-likelihood part of the distribution with inverse probability weighting. It increases the diversity and novelty of generated texts by rescaling and penalizing the high-likelihood part of the predicted distribution, and preserves the fluency by using multi-filtering truncation on the low-likelihood part. Experimental results show that compared with traditional methods our algorithm can significantly increase the diversity and novelty of generated texts without sacrificing fluency.