Reviewerly/exHarmony huggingface dataset:
DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models

Pretrained language models have learned a vast amount of human knowledge from large-scale corpora, but their powerful memorization capability also brings the risk of data leakage. Some risks may only be discovered after the model training is completed, such as the model memorizing a specific phone number and frequently outputting it. In such cases, model developers need to eliminate specific data influences from the model to mitigate legal and ethical penalties. To effectively mitigate these risks, people often have to spend a significant amount of time and computational costs to retrain new models instead of finding ways to cure the ‘sick’ models. Therefore, we propose a method to locate and erase risky neurons in order to eliminate the impact of privacy data in the model. We use a new method based on integrated gradients to locate neurons associated with privacy texts, and then erase these neurons by setting their activation values to zero.Furthermore, we propose a risky neuron aggregation method to eliminate the influence of privacy data in the model in batches. Experimental results show that our method can effectively and quickly eliminate the impact of privacy data without affecting the model’s performance. Additionally, we demonstrate the relationship between model memorization and neurons through experiments, further illustrating the robustness of our method.
Vector-Quantized Prompt Learning for Paraphrase Generation

Deep generative modeling of natural languages has achieved many successes, such as producing fluent sentences and translating from one language into another. However, the development of generative modeling techniques for paraphrase generation still lags behind largely due to the challenges in addressing the complex conflicts between expression diversity and semantic preservation. This paper proposes to generate diverse and high-quality paraphrases by exploiting the pre-trained models with instance-dependent prompts. To learn generalizable prompts, we assume that the number of abstract transforming patterns of paraphrase generation (governed by prompts) is finite and usually not large. Therefore, we present vector-quantized prompts as the cues to control the generation of pre-trained models. Extensive experiments demonstrate that the proposed method achieves new state-of-art results on three benchmark datasets, including Quora, Wikianswers, and MSCOCO. We will release all the code upon acceptance.
MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition

Distantly supervised named entity recognition (DS-NER) aims to locate entity mentions and classify their types with only knowledge bases or gazetteers and unlabeled corpus. However, distant annotations are noisy and degrade the performance of NER models. In this paper, we propose a noise-robust prototype network named MProto for the DS-NER task. Different from previous prototype-based NER methods, MProto represents each entity type with multiple prototypes to characterize the intra-class variance among entity representations. To optimize the classifier, each token should be assigned an appropriate ground-truth prototype and we consider such token-prototype assignment as an optimal transport (OT) problem. Furthermore, to mitigate the noise from incomplete labeling, we propose a novel denoised optimal transport (DOT) algorithm. Specifically, we utilize the assignment result between *Other* class tokens and all prototypes to distinguish unlabeled entity tokens from true negatives. Experiments on several DS-NER benchmarks demonstrate that our MProto achieves state-of-the-art performance. The source code is now available on Github.
Self-Detoxifying Language Models via Toxification Reversal

Language model detoxification aims to minimize the risk of generating offensive or harmful content in pretrained language models (PLMs) for safer deployment. Existing methods can be roughly categorized as finetuning-based and decoding-based. However, the former is often resource-intensive, while the latter relies on additional components and potentially compromises the generation fluency. In this paper, we propose a more lightweight approach that enables the PLM itself to achieve “self-detoxification”. Our method is built upon the observation that prepending a negative steering prompt can effectively induce PLMs to generate toxic content. At the same time, we are inspired by the recent research in the interpretability field, which formulates the evolving contextualized representations within the PLM as an information stream facilitated by the attention layers. Drawing on this idea, we devise a method to identify the toxification direction from the normal generation process to the one prompted with the negative prefix, and then steer the generation to the reversed direction by manipulating the information movement within the attention layers. Experimental results show that our approach, without any fine-tuning or extra components, can achieve comparable performance with state-of-the-art methods.
CompleQA: Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering

How much success in Knowledge Graph Completion (KGC) would translate into the performance enhancement in downstream tasks is an important question that has not been studied in depth. In this paper, we introduce a novel benchmark, namely CompleQA, to comprehensively assess the influence of representative KGC methods on Knowledge Graph Question Answering (KGQA), one of the most important downstream applications. This benchmark includes a knowledge graph with 3 million triplets across 5 distinct domains, coupled with over 5000 question-answering pairs and a completion dataset that is well-aligned with these questions. Our evaluation of four well-known KGC methods in combination with two state-of-the-art KGQA systems shows that effective KGC can significantly mitigate the impact of knowledge graph incompleteness on question-answering performance. Surprisingly, we also find that the best-performing KGC method(s) does not necessarily lead to the best QA results, underscoring the need to consider downstream applications when doing KGC.
Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance

Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PLMs) have achieved substantial advancements in the field of natural language processing. However, in real-world scenarios, data labels are often noisy due to the complex annotation process, making it essential to develop strategies for fine-tuning PLMs with such noisy labels. To this end, we introduce an innovative approach for fine-tuning PLMs using noisy labels, which incorporates the guidance of Large Language Models (LLMs) like ChatGPT. This guidance assists in accurately distinguishing between clean and noisy samples and provides supplementary information beyond the noisy labels, thereby boosting the learning process during fine-tuning PLMs. Extensive experiments on synthetic and real-world noisy datasets further demonstrate the superior advantages of our framework over the state-of-the-art baselines.
Adaptive Gating in Mixture-of-Experts based Language Models

Large language models have demonstrated exceptional language understanding capabilities in many NLP tasks. Sparsely activated mixture-of-experts (MoE) has emerged as a promising solution for scaling models while maintaining a constant number of computational operations. Existing MoE models adopt a fixed gating network where each token is computed by the same number of experts. This contradicts our intuition that the tokens in each sequence vary in terms of their linguistic complexity and, consequently, require different computational costs. Little is discussed in prior research on the trade-off between computation per token and model performance. This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. Adaptive gating preserves sparsity while improving training efficiency. We further draw upon curriculum learning to better align the order of training samples and maximize the training time savings. Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality. Moreover, we conduct a comprehensive analysis of the gating decisions and present our insights on which tokens are inherently difficult to process, depending on the specific language task.
Gradually Excavating External Knowledge for Implicit Complex Question Answering

Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire extrinsic information, then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA in the ~10B LLM class.
Contrastive Distant Supervision for Debiased and Denoised Machine Reading Comprehension

Distant Supervision (DS) is a promising learning approach for MRC by leveraging easily-obtained question-answer pairs. Unfortunately, the heuristically annotated dataset will inevitably lead to mislabeled instances, resulting in answer bias and context noise problems. To learn debiased and denoised MRC models, this paper proposes the Contrastive Distant Supervision algorithm – CDS, which can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. Specifically, to eliminate answer bias, CDS samples counterfactual negative instances, which ensures that MRC models must take both answer information and question-context interaction into consideration. To denoise distantly annotated contexts, CDS samples confusing negative instances to increase the margin between correct and mislabeled instances. We further propose a confidence-aware contrastive loss to model and leverage the uncertainty of all DS instances during learning. Experimental results show that CDS is effective and can even outperform supervised MRC models without manual annotations.
Findings of the WMT 2023 Biomedical Translation Shared Task: Evaluation of ChatGPT 3.5 as a Comparison System

Mariana Neves, Antonio Jimeno Yepes, Aurélie Névéol, Rachel Bawden, Giorgio Maria Di Nunzio, Roland Roller, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann, Cristian Grozea. Proceedings of the Eighth Conference on Machine Translation. 2023.
Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators

Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives – Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLM-generated knowledge with human annotations will be released to facilitate future research.
Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs

Current model testing work has mostly focused on creating test cases. Identifying what to test is a step that is largely ignored and poorly supported. We propose Weaver, an interactive tool that supports requirements elicitation for guiding model testing. Weaver uses large language models to generate knowledge bases and recommends concepts from them interactively, allowing testers to elicit requirements for further testing. Weaver provides rich external knowledge to testers and encourages testers to systematically explore diverse concepts beyond their own biases. In a user study, we show that both NLP experts and non-experts identified more, as well as more diverse concepts worth testing when using Weaver. Collectively, they found more than 200 failing test cases for stance detection with zero-shot ChatGPT. Our case studies further show that Weaver can help practitioners test models in real-world settings, where developers define more nuanced application scenarios (e.g., code understanding and transcript summarization) using LLMs.
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, Luoyi Fu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training

In this work we propose a pragmatic method that reduces the annotation cost for structured label spaces using active learning. Our approach leverages partial annotation, which reduces labeling costs for structured outputs by selecting only the most informative sub-structures for annotation. We also utilize self-training to incorporate the current model’s automatic predictions as pseudo-labels for un-annotated sub-structures. A key challenge in effectively combining partial annotation with self-training to reduce annotation cost is determining which sub-structures to select to label. To address this challenge, we adopt an error estimator to adaptively decide the partial selection ratio according to the current model’s capability. In evaluations spanning four structured prediction tasks, we show that our combination of partial annotation and self-training using an adaptive selection ratio reduces annotation cost over strong full annotation baselines under a fair comparison scheme that takes reading time into consideration.
Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification

Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, Vittorio Castelli. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Antarlekhaka: A Comprehensive Tool for Multi-task Natural Language Annotation

One of the primary obstacles in the advancement of Natural Language Processing (NLP) technologies for low-resource languages is the lack of annotated datasets for training and testing machine learning models. In this paper, we present Antarlekhaka, a tool for manual annotation of a comprehensive set of tasks relevant to NLP. The tool is Unicode-compatible, language-agnostic, Web-deployable and supports distributed annotation by multiple simultaneous annotators. The system sports user-friendly interfaces for 8 categories of annotation tasks. These, in turn, enable the annotation of a considerably larger set of NLP tasks. The task categories include two linguistic tasks not handled by any other tool, namely, sentence boundary detection and deciding canonical word order, which are important tasks for text that is in the form of poetry. We propose the idea of sequential annotation based on small text units, where an annotator performs several tasks related to a single text unit before proceeding to the next unit. The research applications of the proposed mode of multi-task annotation are also discussed. Antarlekhaka outperforms other annotation tools in objective evaluation. It has been also used for two real-life annotation tasks on two different languages, namely, Sanskrit and Bengali. The tool is available at https://github.com/Antarlekhaka/code
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, user-facing artificial intelligence systems will require similar skills in pragmatics: relying on various types of context — from shared linguistic goals and conventions, to the visual and embodied world — to use language effectively. We survey existing grounded settings and pragmatic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. We present recommendations for future grounded task design to naturally elicit pragmatic phenomena, and suggest directions that focus on a broader range of communicative contexts and affordances.
Innovative Reform and Practice in College English Teaching in the Context of the “Internet+”

In the context of the “Internet+” era, actively exploring innovative reforms and practices in college English teaching is crucial. Teachers should grasp the impact of the “Internet+” background, critically reflect on issues in teaching activities from multiple perspectives, and attempt to overcome existing challenges. This involves constructing a completely new teaching guidance system to stimulate systematic exploration of course knowledge by students. Starting with the “Internet+” background, this paper analyzes the innovative challenges in college English teaching, explores the strategies and methods of teaching reform, and provides in-depth interpretations of innovative teaching models, including intelligent teaching modes, blended learning, project-based learning, and action-oriented teaching. The aim is to comprehensively enhance the effectiveness of college English teaching guidance, fostering the English communicative competence of university students.
Pelatihan Penggunaan Aplikasi Geogebra sebagai Media Pembelajaran Inovatif pada Calon Guru Sekolah Dasar

Innovative learning media is a form of developing learning media designed to provide a more interesting and interactive learning experience for students. Due to the importance of using innovative learning media, prospective elementary school teacher students must be equipped with various technology mastery skills to develop their abilities so that they can present a meaningful teaching and learning process. Meaningful learning certainly requires interactive media to support the implementation of classroom learning. One program that can be used is the GeoGebra program. The GeoGebra program is a computer or smartphone program that can visualize abstract mathematical material. The GeoGebra application shows that this program can be used as a bridge in delivering mathematics material so as to make it easier for educators in the teaching and learning process. The purpose of this training is to increase the knowledge and abilities of prospective elementary school teachers in designing and using GeoGebra in learning mathematics. In this service, an example of the material practised during the training is. The results of this service can be said to be successful, as seen from the enthusiasm of the participants in participating in the training. This is also marked from the results of the posttest questionnaire that 100% of participants are ready to use the GeoGebra application in developing learning media.
Get Ready for Corporate Digital Currencies: Facebook failed, but another tech giant might soon pull it of

BUYING AND SELLING cryptocurrencies is a big business. Bitcoin, for exam-ple, processed US$3 trillion worth of transactions in 2021, more than double what American Express did. But most of those transactions were just for speculation. The fraction that involved buying actual stuff (goods and ser-vices) is so small that it's hard to measure. What development might enable cryptocurrencies to displace the U.S. dollar as the dominant medium for exchange in the United States? It might look a lot like what Facebook (now called Meta) proposed with its Libra stable coin (which morphed into Diem). Although Diem suffered a fatal setback in 2021 when U.S. Treasury Secretary Janet Yellen refused to support it, that doesn't mean that a related model couldn't succeed. Indeed, Yellen's refusal to support Diem suggests that she saw a private digital currency as a potentially serious competitor to the U.S. dollar-and hence to the U.S. Treasury.$$Here I outline the rationale for such a private digital currency and explain how one might soon take off in the United States.
Dense Swin Transformer for Classification of Thyroid Nodules

As an early sign of thyroid cancer, thyroid nodules are the most common nodular lesions. As a non-invasive imaging method, ultrasound is widely used in the diagnosis of benign and malignant thyroid nodules. As there is no obvious difference in appearance between the two types of thyroid nodules, and the contrast with the surrounding muscle tissue is too low, it is difficult to distinguish the benign and malignant nodules. Therefore, a dense nodal Swin-Transformer(DST) method for the diagnosis of thyroid nodules is proposed in this paper. Image segmentation is carried out through patch, and feature maps of different sizes are constructed in four stages, which consider different information of each layer of features. In each stage block, a dense connection mechanism is used to make full use of multi-layer features and effectively improve the diagnostic performance. The experimental results of multi-center ultrasound data collected from 17 hospitals show that the accuracy of the proposed method is 87.27%, the sensitivity is 88.63%, and the specific effect is 85.16%, which verifies that the proposed algorithm has the potential to assist clinical practice.
A Web Application for Editing the Color Theme of a Complex ERP Platform

This paper presents a web-based tool designed and implemented for facilitating the editing of the CSS values that control the color theme of a complex web ERP platform. The regular way of modifying the color specifications that control the aspect of web applications usually involves a lot of effort divided into CSS attributes editing, web page refreshing and impacted zones analyzing. This is a time-consuming process that is more demanding the more complex the targeted application is. Since the performed operations are repetitive, an automation of the CSS color editing process comes as a natural consequence. Having an intuitive interface and presenting the color changing effects immediately in a demo page, the implemented color editor takes as input a CSS file, parses its content, allows the user to modify the corresponding values and exports a ready-to-use CSS specification that reflects the current aspect of the demo page. A direct replacement of the initial CSS file with the one produced by the application changes immediately the entire aspect of the targeted web application, in our case being a complex platform that is used for developing specific ERP solutions.