Reviewerly/exHarmony huggingface dataset:
Discourse-Aware Causal Emotion Entailment

Causal Emotion Entailment (CEE) aims to identify the corresponding causal utterances for a target emotional utterance in conversations. Most previous research has focused on the use of sequential encoding to model conversational contexts, without fully considering the interaction effects between different utterances. In this paper, we explore the significance of discourse parsing in addressing these interactions, and propose a new model called discourse-aware model (DAM) to tackle the CEE task. Concretely, we use a multi-task learning framework to jointly model CEE and discourse parsing to fuse rich discourse information. In addition, we use a graph neural network to further enhance our CEE model by explicitly encoding discourse and other discourse-related structure features. The results on the benchmark corpus show that the DAM outperforms the state-of-the-art systems in the literature. This suggests that the discourse structure may contain a potential link between emotional utterances and their corresponding cause expressions. We will release the codes of this paper to facilitate future research ( https://github.com/Sakurakdx/DAM ).
Dual-Enhancement Model of Entity Pronouns and Evidence Sentence for Document-Level Relation Extraction

Document-level relation extraction (DocRE) aims to identify all relations between entities in different sentences within a document. Most works are committed to achieving more accurate relation prediction by optimizing model structure. However, the usage of entity pronoun information and extracting evidence sentences are limited by incomplete manual annotation data. In this paper, we propose a Dual-enhancement model of entity pronouns and EvideNce senTences (DeepENT), which efficiently leverages pronoun information and effectively extracts evidence sentences to improve DocRE. First, we design an Entity Pronouns Enhancement Module, which achieves co-reference resolution and automatic data fusion to enhance the completeness of entity information. Then, we define two types of evidence sentences and design heuristic rules to extract them, used in obtaining sentence-aware context embedding. In this way, we can logically utilize complete and accurate evidence sentence information. Experimental results reveal that our approach performs excellently on the Re-DocRED benchmark, especially in predicting inter-sentence expression relations.
Sample Selection Based on Uncertainty for Combating Label Noise

Automatic segmentation of medical images plays a crucial role in scientific research and healthcare. Obtaining large-scale training datasets with high-quality manual annotations poses challenges in many clinical applications. Utilizing noisy datasets has become increasingly important, but label noise significantly affects the performance of deep learning models. Sample selection is an effective method for handling label noise. In this study, we propose a medical image segmentation framework based on entropy estimation uncertainty for sample selection to address datasets with noisy labels. Specifically, after sample selection, parallel training of two networks and cross-model information exchange are employed for collaborative optimization learning. Based on the exchanged information, sample selection is performed using entropy estimation uncertainty, following a carefully designed schedule for gradual label filtering and correction of noisy labels. The framework is flexible in terms of the precise deep neural network (DNN) models used. Method analysis and empirical evaluation demonstrate that our approach exhibits superior performance on open datasets with noisy annotations. The sample selection method outperforms small loss criterion approaches, and the segmentation results surpass those of traditional fully supervised models. Our framework provides a valuable solution for effectively handling noisy label datasets in medical image segmentation tasks.
Membership Inference Attacks Against Medical Databases

Membership inference is a powerful attack to privacy databases especially for medical data. Existing attack model utilizes the shadow model to inference the private members in the privacy data-sets and information, which can damage the benefits of the data owners and may cause serious data leakage. However, existing defence are concentrated on the encryption methods, which ignore the inference can also cause the unacceptable loss in the real applications. In this work, we propose a novel inference attack model, which utilizes a shadow model to simulate the division system in the medical database and subsequently infer the members in the medical databases. Moreover, the established shadow inference model can classify the labels of medical data and obtain the privacy members in the medical databases. In contrast with traditional inference attacks, we apply the attack in the medical databases rather than recommendation system or machine learning classifiers. From our extensive simulation and comparison with traditional inference attacks, we can observe the proposed model can achieve the attacks in the medical data with reasonable attack accuracy and acceptable computation costs.
NITRO: A Gadget to Transform Standard Rollators into Smart Rollators for Monitoring User Conditions

The goal of using a rollator is to provide physical stability, support, and increase independence while walking or performing daily activities. The decision to use a rollator is often made in consultation with healthcare professionals, who assess the individual’s mobility limitations and recommend the appropriate assistive device. Rollators can be made “smart” via the inclusion of sensors and actuators, so that they can monitor user condition. This allows caregivers or family members to remotely monitor the user’s condition, track their activities, and receive alerts or notifications if any issues arise. That is particularly important in care home facilities, where the ratio of users per caregiver is high. This work presents the design and initial test of NITRO, a gadget that turNs a rollator Into a smarT ROllator to monitor the user’s gait and routines in care home facilities. Two main requirements have been imposed by the caregivers: first, the gadget should be attached (and detached) to any standard rollator frame (four-wheels, three-wheels, two rear wheels), so users can continue using their rollators: and seconds, it can monitor at any time and anywhere while they use their rollators.
Multi-granularity Contrastive Siamese Networks for Abstractive Text Summarization

Abstractive text summarization is an important task in natural language generation, which aims to compress input documents and generate concise and informative summaries. Sequence-to-Sequence (Seq2 Seq) models have achieved good results in abstractive text summarization in recent years. However, such models are often sensitive to noise information in the training data and exhibit fragility in practical applications. To enhance the denoising ability of the models, we propose a Multi-Granularity Contrastive Siamese Networks for Abstractive Text Summarization. Specifically, we first perform word-level and sentence-level data augmentation on the input text and integrate the noise information of the two granularities into the input text to generate augmented text pairs with diverse noise information. Then, we jointly train the Seq2Seq model using contrastive learning to maximize the consistency between the representations of the augmented text pairs through a Siamese network. We conduct empirical experiments on the CNN/Daily Mail and XSum datasets. Compared to many existing benchmarks, the results validate the effectiveness of our model.
Mutually Guided Dendritic Neural Models

With the explosive growth of Internet data, it has become quite easy to collect unlabeled training data in many practical machine learning and its applications, but it is relatively difficult to obtain a large amount of labeled training data. Meanwhile, based on the idea of semi-supervised learning, we propose the mutually guided dendritic neural models (MGDNM) framework, which can realize the expansion of labeled datasets. MGDNM utilizes two base classifiers for data exchange, so as to achieving complementary advantages. To simulate the problem of insufficient labeled data, we used 20% of the dataset as the training dataset. On this basis, we conducted experiments on three datasets (Iris, Breast Cancer and Glass). By calculating the accuracy and confusion matrices, the comparison shows that the classification effect of MGDNM is significantly higher than dendritic neural model (DNM), Support Vector Machine (SVM), Gaussian Naive Bayesian (GaussianNB) and Back Propagation Neural Network (BP). It shows that MGDNM framework is effective and feasible.
Domain Generalization via Implicit Domain Augmentation

Deep convolutional neural networks often suffer significant performance degradation when deployed to an unknown domain. To tackle this problem, domain generalization (DG) aims to generalize the model learned from source domains to an unseen target domain. Prior work mostly focused on obtaining robust cross-domain feature representations, but neglecting the generalization ability of the classifier. In this paper, we propose a novel approach named Implicit Domain Augmentation (IDA) for classifier regularization. Our motivation is to prompt the classifier to see more diverse domains and thus become more knowledgeable. Specifically, the styles of samples will be transferred and re-equipped to original features. To obtain the direction of meaningful style transfer, we use the multivariate normal distribution to model the feature statistics. Then new styles are sampled from the distribution to simulate potential unknown domains. To efficiently implement IDA, we achieve domain augmentation implicitly by minimizing an upper bound of expected cross-entropy loss on the augmented feature set instead of generating new samples explicitly. As a plug-and-play technique, IDA can be easily applied to other DG methods and boost the performance, introducing negligible computational overhead. Experiments on several tasks demonstrate the effectiveness of our method.
Research and Practice on the Teaching Reform of Probability Theory and Mathematical Statistics Based on Knowledge Visualization

The main research object of the course of probability theory and mathematical statistics is relatively abstract random phenomena, which requires students to have strong thinking level and learning ability. Due to the special nature of the course of probability theory and mathematical statistics, it is difficult for students to have an intuitive and profound understanding of the corresponding knowledge points in the course.Practice has proved that the visualization teaching method using Matlab, Office PowerPoint and Office Excel software as tools in the course of Probability Theory and Mathematical Statistics can convert "abstract" into "intuitive" to a large extent.The high requirement for students' "thinking level" is transformed into a low requirement for "observation ability", and the learning efficiency of the course "Probability Theory and Mathematical Statistics" is greatly improved.
Hybrid CNN-Interpreter: Interprete Local and Global Contexts for CNN-Based Models

Convolutional neural network (CNN) models have seen advanced improvements in performance in various domains, but lack of interpretability is a major barrier to assurance and regulation during operation for acceptance and deployment of AI-assisted applications. There have been many works on input interpretability focusing on analyzing the input-output relations, but the internal logic of models has not been clarified in the current mainstream interpretability methods. In this study, we propose a novel hybrid CNN-interpreter through: (1) An original forward propagation mechanism to examine the layer-specific prediction results for local interpretability. (2) A new global interpretability that indicates the feature correlation and filter importance effects. By combining the local and global interpretabilities, hybrid CNN-interpreter enables us to have a solid understanding and monitoring of model context during the whole learning process with detailed and consistent representations. Finally, the proposed interpretabilities have been demonstrated to adapt to various CNN-based model structures.
Learning Disentangled Meaning and Style Representations for Positive Text Reframing

The positive text reframing (PTR) task which generates a text giving a positive perspective with preserving the sense of the input text, has attracted considerable attention as one of the NLP applications. Due to the significant representation capability of the pre-trained language model (PLM), a beneficial baseline can be easily obtained by just fine-tuning the PLM. However, how to interpret a diversity of contexts to give a positive perspective is still an open problem. Especially, it is more serious when the size of the training data is limited. In this paper, we present a PTR framework, that learns representations where the meaning and style of text are structurally disentangled. The method utilizes pseudo-positive reframing datasets which are generated with two augmentation strategies. A simple but effective multi-task learning-based model is learned to fuse the generation capabilities from these datasets. Experimental results on Positive Psychology Frames (PPF) dataset, show that our approach outperforms the baselines, BART by five and T5 by six evaluation metrics. Our source codes and data are available online.
Neural Generation Meets Real People: Building a Social, Informative Open-Domain Dialogue Agent

Ethan A. Chi, Ashwin Paranjape, Abigail See, Caleb Chiam, Trenton Chang, Kathleen Kenealy, Swee Kiat Lim, Amelia Hardy, Chetanya Rastogi, Haojun Li, Alexander Iyabor, Yutong He, Hari Sowrirajan, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, Jillian Tang, Avanika Narayan, Giovanni Campagna, Christopher Manning. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.
MERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems

Automatic Evaluation (AE) and Response Selection (RS) models assign quality scores to various candidate responses and rank them in conversational setups. Prior response ranking research compares various models’ performance on synthetically generated test sets. In this work, we investigate the performance of model-based reference-free AE and RS models on our constructed response ranking datasets that mirror real-case scenarios of ranking candidates during inference time. Metrics’ unsatisfying performance can be interpreted as their low generalizability over more pragmatic conversational domains such as human-chatbot dialogs. To alleviate this issue we propose a novel RS model called MERCY that simulates human behavior in selecting the best candidate by taking into account distinct candidates concurrently and learns to rank them. In addition, MERCY leverages natural language feedback as another component to help the ranking task by explaining why each candidate response is relevant/irrelevant to the dialog context. These feedbacks are generated by prompting large language models in a few-shot setup. Our experiments show the better performance of MERCY over baselines for the response ranking task in our curated realistic datasets.
Enhancing Task Bot Engagement with Synthesized Open-Domain Dialog

The construction of dialog systems for various types of conversations, such as task-oriented dialog (TOD) and open-domain dialog (ODD), has been an active area of research. In order to more closely mimic human-like conversations that often involve the fusion of different dialog modes, it is important to develop systems that can effectively handle both TOD and ODD and access different knowledge sources. In this work, we present a new automatic framework to enrich TODs with synthesized ODDs. We also introduce the PivotBot model, which is capable of handling both TOD and ODD modes and can access different knowledge sources to generate informative responses. Evaluation results indicate the superior ability of the proposed model to switch smoothly between TOD and ODD tasks.
Grounded Complex Task Segmentation for Conversational Assistants

Following complex instructions in conversational assistants can be quite daunting due to the shorter attention and memory spans when compared to reading the same instructions. Hence, when conversational assistants walk users through the steps of complex tasks, there is a need to structure the task into manageable pieces of information of the right length and complexity. In this paper, we tackle the recipes domain and convert reading structured instructions into conversational structured ones. We annotated the structure of instructions according to a conversational scenario, which provided insights into what is expected in this setting. To computationally model the conversational step’s characteristics, we tested various Transformer-based architectures, showing that a token-based approach delivers the best results. A further user study showed that users tend to favor steps of manageable complexity and length, and that the proposed methodology can improve the original web-based instructional text. Specifically, 86% of the evaluated tasks were improved from a conversational suitability point of view.
Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System

Jianguo Zhang, Stephen Roller, Kun Qian, Zhiwei Liu, Rui Meng, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong. Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue. 2023.
Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning

Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In reality, token-level annotations (slot labels) are time-consuming and difficult to acquire. In this work, we study the Slot Induction (SI) task whose objective is to induce slot boundaries without explicit knowledge of token-level slot annotations. We propose leveraging Unsupervised Pre-trained Language Model (PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised semantic knowledge extracted from PLM, and (2) additional sentence-level intent label signals available from TOD. Our approach is shown to be effective in SI task and capable of bridging the gaps with token-level supervised models on two NLU benchmark datasets. When generalized to emerging intents, our SI objectives also provide enhanced slot label representations, leading to improved performance on the Slot Filling tasks.
DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems

Dialogue act annotations are important to improve response generation quality in task-oriented dialogue systems. However, it can be challenging to use dialogue acts to control response generation in a generalizable way because different datasets and tasks may have incompatible annotations. While alternative methods that utilize latent action spaces or reinforcement learning do not require explicit annotations, they may lack interpretability or face difficulties defining task-specific rewards. In this work, we present a novel end-to-end latent dialogue act model (DiactTOD) that represents dialogue acts in a latent space. DiactTOD, when pre-trained on a large corpus, is able to predict and control dialogue acts to generate controllable responses using these latent representations in a zero-shot fashion. Our approach demonstrates state-of-the-art performance across a wide range of experimental settings on the MultiWOZ dataset, including zero-shot, few-shot, and full data fine-tuning with both end-to-end and policy optimization configurations.
Impact of Synthetic Images on Morphing Attack Detection Using a Siamese Network

This paper evaluated the impact of synthetic images on Morphing Attack Detection (MAD) using a Siamese network with a semi-hard-loss function. Intra and cross-dataset evaluations were performed to measure synthetic image generalisation capabilities using a cross-dataset for evaluation. Three different pre-trained networks were used as feature extractors from traditional MobileNetV2, MobileNetV3 and EfficientNetB0. Our results show that MAD trained on EfficientNetB0 from FERET, FRGCv2, and FRLL can reach a lower error rate in comparison with SOTA. Conversely, worse performances were reached when the system was trained only with synthetic images. A mixed approach (synthetic + digital) database may help to improve MAD and reduce the error rate. This fact shows that we still need to keep going with our effort to include synthetic images in the training process.
Face Image Quality Estimation on Presentation Attack Detection

Non-referential Face Image Quality Assessment (FIQA) methods have gained popularity as a pre-filtering step in Face Recognition (FR) systems. In most of them, the quality score is usually designed with face comparison in mind. However, a small amount of work has been done on measuring their impact and usefulness on Presentation Attack Detection (PAD). In this paper, we study the effect of quality assessment methods on filtering bona fide and attack samples, their impact on PAD systems, and how the performance of such systems is improved when training on a filtered (by quality) dataset. On a Vision Transformer PAD algorithm, a reduction of 20% of the training dataset by remoing lower-quality samples allowed us to improve the Bona fide Presentation Classification Error Rate (BPCER) by 3% in a cross-dataset test.
Title Pending 10541

In 2022 we planned speech data collection with speakers of Syrian and Jordanian dialects to inform an updated Syrian Arabic dialectology in response to sustained displacement of millions of Syrians. The pandemic imposed remote data collection, but an internet-based approach also facilitated recruitment with this highly distributed speech community. Their vulnerable situation brings barriers, however, since most prospective participants have limited internet data and rarely use email. We collected self-recorded short audio files in which participants read scripted materials and described pictures. Three platforms were tested: Gorilla, Phonic and Awesome Voice Recorder (AVR, smartphone app). Gorilla/Phonic offer stimulus presentation advantages, so were piloted thoroughly, but the audio quality obtained was not suitable for phonetic analysis, ruling out their use in the main study. AVR yields full spectrum wav files but requires participants to submit files by email, so we recruited local fieldworkers to support participants with recording and file submission. We asked fieldworkers and participants about their experience of working with us, through surveys and interviews. The results confirm fieldworker involvement was crucial to the success of the project which generated high quality audio data, suitable for phonetic analysis, from 134 speakers within three months (Almbark, Hellmuth, &amp;amp; Brown, forthcoming).