Reviewerly/exHarmony huggingface dataset:
A Keyword Based Approach to Understanding the Overpenalization of Marginalized Groups by English Marginal Abuse Models on Twitter

Harmful content detection models tend to have higher false positive rates for content from marginalized groups. In the context of marginal abuse modeling on Twitter, such disproportionate penalization poses the risk of reduced visibility, where marginalized communities lose the opportunity to voice their opinion on the platform. Current approaches to algorithmic harm mitigation, and bias detection for NLP models are often very ad hoc and subject to human bias. We make two main contributions in this paper. First, we design a novel methodology, which provides a principled approach to detecting and measuring the severity of potential harms associated with a text-based model. Second, we apply our methodology to audit Twitter’s English marginal abuse model, which is used for removing amplification eligibility of marginally abusive content. Without utilizing demographic labels or dialect classifiers, we are still able to detect and measure the severity of issues related to the over-penalization of the speech of marginalized communities, such as the use of reclaimed speech, counterspeech, and identity related terms. In order to mitigate the associated harms, we experiment with adding additional true negative examples and find that doing so provides improvements to our fairness metrics without large degradations in model performance.
Understanding Emotion Valence is a Joint Deep Learning Task

The valence analysis of speakers' utterances or written posts helps to understand the activation and variations of the emotional state throughout the conversation.More recently, the concept of Emotion Carriers (EC) has been introduced to explain the emotion felt by the speaker and its manifestations.In this work, we investigate the natural inter-dependency of valence and ECs via a multi-task learning approach.We experiment with Pre-trained Language Models (PLM) for single-task, two-step, and joint settings for the valence and EC prediction tasks.We compare and evaluate the performance of generative (GPT-2) and discriminative (BERT) architectures in each setting.We observed that providing the ground truth label of one task improves the prediction performance of the models in the other task.We further observed that the discriminative model achieves the best tradeoff of valence and EC prediction tasks in the joint prediction setting.As a result, we attain a single model that performs both tasks, thus, saving computation resources at training and inference times.
Enhancing textual counterfactual explanation intelligibility through Counterfactual Feature Importance

Textual counterfactual examples explain a prediction by modifying the tokens of an initial instance in order to flip the outcome of a classifier.Even under sparsity constraint, counterfactual generation can lead to numerous changes from the initial text, making the explanation hard to understand.We propose Counterfactual Feature Importance, a method to make non-sparse counterfactual explanations more intelligible.Counterfactual Feature Importance assesses token change importance between an instance to explain and its counterfactual example.We develop two ways of computing Counterfactual Feature Importance, respectively based on classifier gradient computation and counterfactual generator loss evolution during counterfactual search.Then we design a global version of Counterfactual Feature Importance, providing rich information about semantic fields globally impacting classifier predictions.Counterfactual Feature Importance enables to focus on impacting parts of counterfactual explanations, making counterfactual explanations involving numerous changes more understandable.
Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement

Large language models have achieved impressive few-shot performance on a wide variety of tasks.However, in many settings, users require confidence estimates for model predictions.While traditional classifiers produce scores for each label, language models instead produce scores for the generation which may not be well calibrated.We compare generations across diverse prompts and show that these can be used to create confidence scores.By utilizing more prompts we can get more precise confidence estimates and use response diversity as a proxy for confidence.We evaluate this approach across ten multiple-choice questionanswering datasets using three models: T0, FLAN-T5, and GPT-3.In addition to analyzing multiple human written prompts, we automatically generate more prompts using a language model in order to produce finer-grained confidence estimates.Our method produces more calibrated confidence estimates compared to the log probability of the answer to a single prompt.These improvements could benefit users who rely on prediction confidence for integration into a larger system or in decisionmaking processes.
GPTs Don’t Keep Secrets: Searching for Backdoor Watermark Triggers in Autoregressive Language Models

This work analyzes backdoor watermarks in an autoregressive transformer fine-tuned to perform a generative sequence-to-sequence task, specifically summarization.We propose and demonstrate an attack to identify trigger words or phrases by analyzing open ended generations from autoregressive models that have backdoor watermarks inserted.It is shown in our work that triggers based on random common words are easier to identify than those based on single, rare tokens.The attack proposed is easy to implement and only requires access to the model weights.Code used to create the backdoor watermarked models and analyze their outputs is shared at https: //github.com/evan-person/findingBackdoorWatermarks.
Emotions in Spoken Language - Do we need acoustics?

Work on emotion detection is often focused on textual data from i.e.Social Media.If multimodal data (i.e.speech) is analysed, the focus again is often placed on the transcription.This paper takes a closer look at how crucial acoustic information actually is for the recognition of emotions from multimodal data.To this end we use the IEMOCAP data, which is one of the larger data sets that provides transcriptions, audio recordings and manual emotion categorization.We build models for emotion classification using text-only, acoustics-only and combining both modalities in order to examine the influence of the various modalities on the final categorization.Our results indicate that using text-only models outperform acoustics-only models.But combining text-only and acousticonly models improves the results.Additionally, we perform a qualitative analysis and find that a range of misclassifications are due to factors not related to the model, but to the data such as, recording quality, a challenging classification task and misclassifications that are unsurprising for humans.
Text-To-KG Alignment: Comparing Current Methods on Classification Tasks

In contrast to large text corpora, knowledge graphs (KG) provide dense and structured representations of factual information.This makes them attractive for systems that supplement or ground the knowledge found in pre-trained language models with an external knowledge source.This has especially been the case for classification tasks, where recent work has focused on creating pipeline models that retrieve information from KGs like ConceptNet as additional context.Many of these models consist of multiple components, and although they differ in the number and nature of these parts, they all have in common that for some given text query, they attempt to identify and retrieve a relevant subgraph from the KG.Due to the noise and idiosyncrasies often found in KGs, it is not known how current methods compare to a scenario where the aligned subgraph is completely relevant to the query.In this work, we try to bridge this knowledge gap by reviewing current approaches to text-to-KG alignment and evaluating them on two datasets where manually created graphs are available, providing insights into the effectiveness of current methods.We release our code for reproducibility. 1
Transformer-based Prediction of Emotional Reactions to Online Social Network Posts

Emotional reactions to Online Social Network posts have recently gained importance in the study of the online ecosystem.Prior to post publication, the number of received reactions can be predicted based on either the textual content of the post or the related metadata.However, existing approaches suffer from both the lack of semantic-aware language understanding models and the limited explainability of the prediction models.To overcome these issues, we present a new transformer-based method to predict the number of emotional reactions of different types to social posts.It leverages the attention mechanism to capture arbitrary semantic textual relations neglected by prior works.Furthermore, it also provides end-users with textual explanations of the predictions.The results achieved on a large collection of Facebook posts confirm the applicability of the presented methodology.
Privacy- and Utility-Preserving NLP with Anonymized data: A case study of Pseudonymization

This work investigates the effectiveness of different pseudonymization techniques, ranging from rule-based substitutions to using pretrained Large Language Models (LLMs), on a variety of datasets and models used for two widely used NLP tasks: text classification and summarization.Our work provides crucial insights into the gaps between original and anonymized data (focusing on the pseudonymization technique) and model quality and fosters future research into higherquality anonymization techniques to better balance the trade-offs between data protection and utility preservation.We make our code, pseudonymized datasets, and downstream models publicly available.
Zero-Shot Dialogue Relation Extraction by Relating Explainable Triggers and Relation Names

Developing dialogue relation extraction (DRE) systems often requires a large amount of labeled data, which can be costly and timeconsuming to annotate.In order to improve scalability and support diverse, unseen relation extraction, this paper proposes a method for leveraging the ability to capture triggers and relate them to previously unseen relation names.Specifically, we introduce a model that enables zero-shot dialogue relation extraction by utilizing trigger-capturing capabilities.Our experiments on a benchmark DialogRE dataset demonstrate that the proposed model achieves significant improvements for both seen and unseen relations.Notably, this is the first attempt at zero-shot dialogue relation extraction using trigger-capturing capabilities, and our results suggest that this approach is effective for inferring previously unseen relation types.Overall, our findings highlight the potential for this method to enhance the scalability and practicality of DRE systems. 1
On the Surprising Effectiveness of Name Matching Alone in Autoregressive Entity Linking

Fifteen years of work on entity linking has established the importance of different information sources in making linking decisions: mention and entity name similarity, contextual relevance, and features of the knowledge base.Modern state-of-the-art systems build on these features, including through neural representations (Wu et al., 2020).In contrast to this trend, the autoregressive language model GENRE (De Cao et al., 2021) generates normalized entity names for mentions and beats many other entity linking systems, despite making no use of knowledge base (KB) information.How is this possible?We analyze the behavior of GENRE on several entity linking datasets and demonstrate that its performance stems from memorization of name patterns.In contrast, it fails in cases that might benefit from using the KB.We experiment with a modification to the model to enable it to utilize KB information, highlighting challenges to incorporating traditional entity linking information sources into autoregressive models.
Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions?

While many real-life tasks require reasoning over multi-step sequential instructions, collecting fine-grained annotations for each intermediate step can be prohibitively expensive.In this work, we study how general pretrained sequence-to-sequence transformers perform under varying types of annotation for sequential instruction understanding.We conduct experiments using T5 (Raffel et al., 2020) on a commonly-used multi-step instruction understanding dataset SCONE (Long et al., 2016) that includes three sub-tasks.First, we show that with only gold supervision for the final step of a multi-step instruction sequence, depending on the sequential properties of different tasks, transformers may exhibit extremely bad performance on intermediate steps, in stark contrast with their performance on the final step.Next, we explore two directions to relieve this problem.We show that with the same limited annotation budget, using supervision uniformly distributed across different steps (instead of only final-step supervision), we can greatly improve the performance on intermediate steps with a drop in final-step performance.Further, we explore a contrastive learning approach to provide training signals on intermediate steps with zero intermediate gold supervision.This, however, achieves mixed results.It significantly improves the model's bad intermediate-step performance on one subtask, but also shows decreased performance on another subtask.
Conversational Recommendation as Retrieval: A Simple, Strong Baseline

Conversational recommendation systems (CRS) aim to recommend suitable items to users through natural language conversation.However, most CRS approaches do not effectively utilize the signal provided by these conversations.They rely heavily on explicit external knowledge e.g., knowledge graphs to augment the models' understanding of the items and attributes, which is quite hard to scale.To alleviate this, we propose an alternative information retrieval (IR)-styled approach to the CRS item recommendation task, where we represent conversations as queries and items as documents to be retrieved.We expand the document representation used for retrieval with conversations from the training set.With a simple BM25-based retriever, we show that our task formulation compares favorably with much more complex baselines using complex external knowledge on a popular CRS benchmark.We demonstrate further improvements using user-centric modeling and data augmentation to counter the cold start problem for CRSs.
Combining Active Learning and Task Adaptation with BERT for Cost-Effective Annotation of Social Media Datasets

Social media provide a rich source of data that can be mined and used for a wide variety of research purposes.However, annotating this data can be expensive, yet necessary for state-ofthe-art pre-trained language models to achieve high prediction performance.Therefore, we combine pool-based active learning based on prediction uncertainty (an established method for reducing annotation costs) with unsupervised task adaptation through Masked Language Modeling (MLM).The results on three different datasets (two social media corpora, one benchmark dataset) show that task adaptation significantly improves results and that with only a fraction of the available training data, this approach reaches similar F1-scores as those achieved by an upper-bound baseline model fine-tuned on all training data.We hereby contribute to the scarce corpus of research on active learning with pre-trained language models and propose a cost-efficient annotation sampling and fine-tuning approach that can be applied to a wide variety of tasks and datasets.
Application of a Technology-Based Data Collection System for the Writing Class of the IAIN Cirebon Student Da'wah Institute

Effective data management plays a crucial role in optimizing the performance of educational institutions. This study aims to explore the implementation of a technology-based data management system for writing classes in the Student Preaching Institution of the State Islamic Institute (IAIN) Cirebon. The objective is to improve the efficiency and effectiveness of data collection, storage, and analysis, ultimately enhancing the overall quality of the writing classes. This research utilizes a mixed-methods approach, incorporating qualitative and quantitative data collection techniques. Firstly, a comprehensive review of the existing data management practices is conducted to identify the limitations and challenges faced by the institution. Secondly, a technology-based data management system is developed, integrating modern tools and techniques to streamline the data collection process. The system is designed to facilitate easy data entry, retrieval, and analysis, ensuring accurate and reliable information for decision-making purposes. The implementation of the technology-based data management system is evaluated through a series of case studies and interviews with relevant stakeholders. The findings reveal several significant benefits, including increased efficiency in data management, improved accuracy in reporting, and enhanced collaboration among the stakeholders involved in the writing classes. Moreover, the system enables the institution to track the progress of individual students, identify their strengths and weaknesses, and provide targeted interventions to support their learning and development. The results of this study contribute to the growing body of literature on the application of technology in educational institutions. The findings highlight the importance of leveraging technology to optimize data management processes, particularly in the context of writing classes in the Student Preaching Institution of IAIN Cirebon. The insights gained from this research can serve as a valuable resource for other educational institutions seeking to enhance their data management practices and improve the overall quality of their programs.
Towards a Secure Technology-Driven Architecture for Smart Health Insurance Systems: An Empirical Study

Health insurance has become a crucial component of people’s lives as the occurrence of health problems rises. Unaffordable healthcare problems for individuals with little income might be a problem. In the case of a medical emergency, health insurance assists individuals in affording the costs of healthcare services and protects them financially against the possibility of debt. Security, privacy, and fraud risks may impact the numerous benefits of health insurance. In recent years, health insurance fraud has been a contentious topic due to the substantial losses it causes for individuals, commercial enterprises, and governments. Therefore, there is a need to develop mechanisms for identifying health insurance fraud incidents. Furthermore, a large quantity of highly sensitive electronic health insurance data are generated on a daily basis, which attracts fraudulent users. Motivated by these facts, we propose a smart healthcare insurance framework for fraud detection and prevention (SHINFDP) that leverages the capabilities of cutting-edge technologies including blockchain, 5G, cloud, and machine learning (ML) to enhance the health insurance process. The proposed framework is evaluated using mathematical modeling and an industrial focus group. In addition, a case study was demonstrated to illustrate the SHINFDP’s applicability in enhancing the security and effectiveness of health insurance. The findings indicate that the SHINFDP aids in the detection of healthcare fraud at early stages. Furthermore, the results of the focus group show that SHINFDP is adaptable and simple to comprehend. The case study further strengthens the findings and also describes the implications of the proposed solution in a real setting.
Deep Learning in Big Data, Image, and Signal Processing in the Modern Digital Age

Data, such as images and signals, are constantly generated from various industries, including the internet [...]
Optimization with Neural Network Feasibility Surrogates: Formulations and Application to Security-Constrained Optimal Power Flow

In many areas of constrained optimization, representing all possible constraints that give rise to an accurate feasible region can be difficult and computationally prohibitive for online use. Satisfying feasibility constraints becomes more challenging in high-dimensional, non-convex regimes which are common in engineering applications. A prominent example that is explored in the manuscript is the security-constrained optimal power flow (SCOPF) problem, which minimizes power generation costs, while enforcing system feasibility under contingency failures in the transmission network. In its full form, this problem has been modeled as a nonlinear two-stage stochastic programming problem. In this work, we propose a hybrid structure that incorporates and takes advantage of both a high-fidelity physical model and fast machine learning surrogates. Neural network (NN) models have been shown to classify highly non-linear functions and can be trained offline but require large training sets. In this work, we present how model-guided sampling can efficiently create datasets that are highly informative to a NN classifier for non-convex functions. We show how the resultant NN surrogates can be integrated into a non-linear program as smooth, continuous functions to simultaneously optimize the objective function and enforce feasibility using existing non-linear solvers. Overall, this allows us to optimize instances of the SCOPF problem with an order of magnitude CPU improvement over existing methods.
Compact Structure-Preserving Signatures with Almost Tight Security

Abstract In structure-preserving cryptography, every building block shares the same bilinear groups. These groups must be generated for a specific, a priori fixed security level, and thus, it is vital that the security reduction in all involved building blocks is as tight as possible. In this work, we present the first generic construction of structure-preserving signature schemes whose reduction cost is independent of the number of signing queries. Its chosen-message security is almost tightly reduced to the chosen-plaintext security of a structure-preserving public-key encryption scheme and the security of Groth–Sahai proof system. Technically, we adapt the adaptive partitioning technique by Hofheinz (Eurocrypt 2017) to the setting of structure-preserving signature schemes. To achieve a structure-preserving scheme, our new variant of the adaptive partitioning technique relies only on generic group operations in the scheme itself. Interestingly, however, we will use non-generic operations during our security analysis. Instantiated over asymmetric bilinear groups, the security of our concrete scheme is reduced to the external Diffie–Hellman assumption with linear reduction cost in the security parameter, independently of the number of signing queries. The signatures in our schemes consist of a larger number of group elements than those in other non-tight schemes, but can be verified faster, assuming their security reduction loss is compensated by increasing the security parameter to the next standard level.