Reviewerly/exHarmony huggingface dataset:
A Process for Evaluating Explanations for Transparent and Trustworthy AI Prediction Models

This study proposes a process to generate and validate algorithmic explanations for the reasoning of an AI prediction model, implemented using a Bayesian network (BN). The intention of the generated explanations is to increase the transparency and trustworthiness of a decision-support system that uses a BN prediction model. To achieve this, explanations should be presented in an easy-to-understand, clear, and concise natural language narrative. We have developed an algorithm for explaining the reasoning of a prediction made using a BN. For the narrative part of the explanation, we use a template which presents the ‘content’ part of the explanation; this content is a word-less information structure that applies to all BNs. The template, on the other hand, needs to be designed specifically for each BN model. In this paper, we use a BN for the risk of trauma-induced coagulopathy, a critical bleeding problem. We outline a process for using experts’ explanations as the basis for designing the explanation template. We do not believe that an algorithmic explanation needs to be indistinguishable from expert explanations; instead we aim to imitate the narrative structure of explanations given by experts, although we find that there is considerable variation in these. We then consider how the generated explanations can be evaluated, since a direct comparison (in the style of a Turing test) would likely fail. We describe a study using questionnaires and interviews to evaluate the effect of an algorithmic explanation on the transparency and also on the trustworthiness of the predictions made by the system. The preliminary results of our study suggest that the presence of an explanation makes the AI model more transparent but not necessarily more trustworthy.
Direct Metric Optimization for Imbalanced Classification

Imbalanced classification (IC) involves training models in datasets with uneven label distribution. It has been widely stuided [1] , [2] , with various applications such as object detection [3] , [4] , image retrieval [5] , [6] , and image segmentation [7] , [8] , [9] . In clinical research, data imbalance is also a prominent issue, where negative control usually significantly outweighs positive cases due to the innate low prevalence rate of certain diseases. Especially for tasks related to rare diseases (e.g., amyotrophic lateral sclerosis, where the prevalence rate can be as low as 0.001% [10] ), the imbalance problem is severe and almost inevitable.
Trustworthy Computing for Biomedical Challenges

This tutorial focuses on the role of trustworthy machine learning techniques in transforming healthcare through the analysis of vast and intricate medical data. It covers the core techniques of accuracy, robustness, fairness, and interpretability in computational medicine, including their challenges and potential for future development. The tutorial aims to provide a comprehensive overview of trustworthy machine learning techniques that can be adopted in healthcare and is suitable for audiences with a background in biomedicine and machine learning. By emphasizing the importance of trustworthy machine learning in healthcare, the tutorial aims to offer insights into how these techniques can lead to more precise diagnoses and treatment regimes.
Population-Level Visual Analytics of Smartphone Sensed Health and Wellness Using Community Phenotypes

The development of mobile health (mHealth) assessment machine learning models requires data gathering studies in which smartphone sensor data is gathered continuously from users’ phones as they live their lives "In-the-wild". Periodically, participants annotate their sensor data with health, wellness and context labels, which serve as ground truth for machine learning models that can predict a user’s health from their smartphone data. However, as the scale of such studies increases, it becomes difficult to analyze such data and build machine learning models that can work across increasingly diverse, heterogeneous participants. Additionally, non-visual analytics approaches have limited interpretability. This paper innovatively takes a visual analytics approach instead. We propose Visualizing COMmunity Phenotypes (VICOMP), an interactive visual analytics framework for exploring complex population-level smartphone-sensed data. Our approach is based on the concept of Community Phenotypes, which effectively visualizes the groups (or phenotypes) that study participant profiles belong to based on how similar they are (communities). Visual representations of community phenotypes within a large population facilitates sensemaking of group patterns. Using VICOMP, analysts can construct multiple community phenotypes using configurable clustering algorithms of sensed and reported information, and explore and reason about them. VICOMP enables analysts to discover homogenous phenotypical sub-groups within a larger heterogeneous population, for whom group-specific machine learning models are more accurate than one-size-fits-all population-level models. VICOMP depicts community phenotypes using accessible visual metaphors such as superimposed bars to visualize community wellness reports, dimension reduction, projections and heatmaps to represent the distribution of smartphone-sensed features. Connected views facilitate contextualization of health and wellness measures across communities that manifest in smartphone data. The utility of VICOMP is demonstrated by visualizing a real, large scale, dataset collected in-the-wild, containing smartphone sensor data and participant-provided health and wellness ground truth labels. In addition, VICOMP is validated using evaluations with smartphone-sensed context recognition and health experts.
“On-the-Fly” Radiometric Calibration of Active 3D Imaging Setups using Superellipse Fitting

We present a novel approach for the radiometric calibration of phase-measuring 3D sensors. Our approach can be applied to a broad variety of scenes and calculates the calibration parameters “on-the-fly” from the actual object measurement images, meaning that no additional calibration session is required.
Identifying Dynamic User Roles in Online Health Communities

Online Health Communities (OHCs) have become an effective platform for gaining and sharing health-related information during the past two decades. Users on OHCs may act in different roles. Identifying user roles can help improve user engagement and user retention. However, there have been few works to identify user roles in OHCs using user embeddings to consider the temporal and textual information at the same time. This study is to propose an approach to generate dynamic user embeddings considering the semantic features of users’ posts and the temporal evolution of their interests using a coupled recurrent neural network model. Meanwhile, we use the unsupervised t-SNE to visualize the high-dimensional user embeddings, and LDA to generate top-N words for different clusters, in interpreting distinguished roles in OHCs.
CareD: Caregiver’s Experience with Cognitive Decline in Reddit Posts

With advancements in analysis of cognitive decline in electronic health records, the research community witnesses a recent surge in social media posting by caregivers and/or loved ones of people with cognitive decline. The major challenges in this area are availability of large and diverse datasets, ethics of data collection and sharing, diagnostic specificity and clinical acceptability. To this end, we construct a new dataset, Caregivers experiences with cognitive Decline (CareD), of 1005 posts with more than 194K words and 9541 sentences, highlighting discussions on people with dementia and Alzheimer's disease on Reddit. We discuss the changing trends of discussions on cognitive decline in social media and open challenges for natural language processing and social computing. We first identify the Reddit posts reflecting substantial information as candidate posts. We further formulate the annotation guidelines, handle perplexities to investigate the existence of experiences, self-reported articles and potential caregiver in candidate posts, resulting in the discovery of latent symptoms, firsthand information, and prospective source of longitudinal information about the patient, respectively.
(Non-)Physiological Data for Wearables in VR Sport Applications and Exergames

The rise of wearable health devices, along with the opportunity to measure physiological data such as heart rate, calories, and steps, is driven by technological innovations and their applications across various domains, including medicine, healthcare, sports, and the entertainment industry. Consequently, the benefits of combining physiological data with VR have been extensively explored. Researchers have investigated the utility of these combinations for various applications, such as measuring emotions or using physiological data to adjust game difficulty levels. Unfortunately, there is no research on the required physiological parameters and non-physiological information in VR. Visualized data could help users improve their VR exercise experience and ensure efficient data use in virtual space. It enables users to adjust exercise intensity, protect against overexertion, or help set specific goals. We adopt a user-centric approach and propose the necessary data in VR for two distinct user groups (athletic and health-conscious). Based on the results, we identify commonalities and differences between these two groups concerning the application of physiological and non-physiological information in VR displays. From this contribution, specific requirements for (non-)physiological data displays can be determined, and generic solutions suitable for both groups can be derived.
Extracting Periodontitis Diagnosis in Clinical Notes with RoBERTa and Regular Expression

This study aimed to utilize text processing and natural language processing (NLP) models to mine clinical notes for the diagnosis of periodontitis and to evaluate the performance of a named entity recognition (NER) model on different regular expression (RE) methods. Two complexity levels of RE methods were used to extract and generate the training data. The SpaCy package and RoBERTa transformer models were used to build the NER model and evaluate its performance with the manual-labeled gold standards. The comparison of the RE methods with the gold standard showed that as the complexity increased in the RE algorithms, the F1 score increased from 0.3-0.4 to around 0.9. The NER models demonstrated excellent predictions, with the simple RE method showing 0.84-0.92 in the evaluation metrics, and the advanced and combined RE method demonstrating 0.95-0.99 in the evaluation. This study provided an example of the benefit of combining NER methods and NLP models in extracting target information from free-text to structured data and fulfilling the need for missing diagnoses from unstructured notes.
Identifying Major Depressive Disorder From Clinical Notes Using Neural Language Models with Distant Supervision

Major Depressive Disorder (MDD) is the leading cause of disability in the world and its prevalence continues to increase. Phenotyping and relevance classification of prevalent but underdiagnosed conditions from clinical notes using Natural Language Processing (NLP) methods has been an effective way to learn valuable clinical insights as well as for performance on multiple clinical domain-specific NLP tasks. Recent NLP advancements in using self-supervision techniques with transformer architectures, such as Bidirectional Encoder Representations from Transformers (BERT), pioneered a new standard of contextual representation for fine-tuning models toward domain-specific tasks. However, supervised fine-tuning of neural models typically requires too many labeled samples to feasibly annotate entirely manually, constricting most research in clinical applications to shared, general purpose datasets. In this work, we propose using distant supervision to generate adequate data for training transformer-based language models for MDD-relatedness sentence classification. We evaluate the performance of two transformer instances - BERT and Bio-clinical BERT - for classifying sentences into one of four classes representing different MDD concept contexts, then compare against three traditional machine learning baseline models. The Bio-Clinical BERT model, pre-trained on biomedical abstracts then further on clinical domain corpora, achieved the highest F1-score for all class labels. Our results show that the neural language models, especially those with relevant domain-based pretraining, are superior at discerning semantic contexts and that distant supervision is an effective tactic for improving the accuracy of the pretrained transformer models.
Intent Recognition on Low-Resource Language Messages in a Health Marketplace Chatbot

In this paper we explore intent recognition models of user messages sent to a chatbot that primarily uses system-initiated navigation. The chatbot (askNivi) discusses sexual and reproductive health topics for educational purposes and to facilitate healthcare access. It is deployed in four languages, three of which are considered low-resource languages (Hausa, Hindi, and Swahili; the fourth language is English). Although the primary navigation mode is system-initiated, many users attempt to take initiative by expressing various intents with natural language messages. This paper describes a multi-lingual corpus of those initiative attempts manually annotated with intent labels as well as results of modeling experiments on the classification of these intents using context-dependent supervised machine learning.
PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data

Detailed phenotype information is fundamental to accurate diagnosis and risk estimation of diseases. As a rich source of phenotype information, electronic health records (EHRs) promise to empower diagnostic variant interpretation. However, how to accurately and efficiently extract phenotypes from the heterogeneous EHR data remains a challenge. In this work, we present PheME, an Ensemble framework using Multi-modality data of structured EHRs and unstructured clinical notes for accurate Phenotype prediction. Firstly, we employ multiple deep neural networks to learn reliable representations from the sparse structured EHR data and redundant clinical notes. A multi-modal model then aligns multi-modal features onto the same latent space to predict phenotypes. Secondly, we leverage ensemble learning to combine outputs from single-modal models and multi-modal models to improve phenotype predictions. We chose seven diseases to evaluate the phenotyping performance of the proposed framework. Experimental results show that using multi-modal data significantly improves phenotype prediction in all diseases, the proposed ensemble learning framework can further boost the performance.
Causal Structure Learning from Imperfect Longitudinal Data in Healthcare

Causal Structure Discovery (CSD) is the task of learning the set of underlying causal relationships from observational data. Due to their computational scalability and flexibility, a recently developed class of CSD methods, NOTEARS, based on a formulation allowing for continuous optimization, is gaining popularity. However, this formulation can lead to incorrect edge orientations when little/no likelihood advantage is conferred upon any edge orientation, e.g., a → b → c versus c → b → a. In longitudinal data, like electronic health records (EHRs), temporal relationships are observable among many pairs of variables. Such temporal relationship is imperfect but still suggest an orientation since causal effects cannot travel backwards in time. Following this idea, we propose methods to incorporate precedence constraints into continuous optimization-based CSD methods. Experiments on both a synthetic and two real-world datasets validate the effectiveness of the proposed precedence constraints.
Machine Learning and Visualizations for Time-Series Healthcare Data

Time-series healthcare data is difficult to analyze and visualize due to: (1) variations in data over time, (2) different modes of physiological signals (genetic data, behavioral data, etc.), and (3) difficulty in analyzing high-dimensional samples in cases where the data is diverse in nature. My research focuses on analyzing such data using machine learning and visualizations. The research questions I wish to address through my dissertation are: (1) How can multidimensional healthcare data be analyzed using machine learning to identify key moments of significance? and (2) What are the methods of visualizing these key moments in an interpretable manner to aid healthcare professionals and caregivers?
Patient Dashboards of Electronic Health Record Data to Support Clinical Care: A Systematic Review

Visualisations in Electronic Health Records (EHRs) are crucial for clinical care. Since clinicians need to quickly diagnose and treat their patients, having appropriate ways to visualise patients’ characteristics and issues documented in the EHR, can be instrumental. However, the existing literature has not yet summarised the characteristics and lessons learned from the studies on patient dashboards for clinical care. Our review analysed patient dashboards, that visualised EHR data to support clinical care, and which were evaluated with end-users. We read papers from Human-Computer Interaction, Information Visualisation, and Medical Informatics, focusing on the user interfaces and the end-user evaluation results. From a set of 3545 articles, we selected 30 studies, which were analysed using Thematic Analysis. Results provide an understanding of the patient dashboard designs, the visualisation techniques employed, the data represented, as well as the lessons learned from this body work; which should contribute to future designs.
Auditing ICU Readmission Rates in an Clinical Database: An Analysis of Risk Factors and Clinical Outcomes

This study presents a machine learning (ML) pipeline for clinical data classification in the context of a 30-day readmission problem, along with a fairness audit on subgroups based on sensitive attributes. A range of ML models are used for classification and the fairness audit is conducted on the model predictions. The fairness audit uncovers disparities in equal opportunity, predictive parity, false positive rate parity, and false negative rate parity criteria on the MIMIC III dataset based on attributes such as gender, ethnicity, language, and insurance group. The results identify disparities in the model's performance across different groups and highlight the need for better fairness and bias mitigation strategies. The study suggests the need for collaborative efforts among researchers, policymakers, and practitioners to address bias and fairness in artificial intelligence (AI) systems.
Foundational Data Science Training for Health Equity Researchers at Minority Serving Institutions: A SoRDS Event

Contemporary research, particularly when addressing the most significant transdisciplinary research challenges, cannot effectively be done without a range of skills relating to data management, data analysis, and cyberinfrastructure (CI). These data and CI skills are common to all disciplines that conduct data-centric research. Research Data Science acts as a vital component of the scientific process. In a grassroots attempt to address this gap, the CODATA-RDA Schools of Research Data Science (SoRDS) was founded in 2016 to provide instruction on foundational data science and open research concepts to early career researchers in low and middle-income countries. This partnership between international collaborators has since 2016 provided this training to over 1000 early career researchers in 24 events in 10 countries worldwide. The most recent event was held at Georgia Institute of Technology and focused on health equity and included researchers from minority-serving institutions in the southeast United States. This paper covers the background of the SoRDS project along with organization and curriculum details. It also covers the transition of the events from a non-domain-centric curriculum to spotlighting biological and social health equity data and what we learned to make future health-related events more engaging and valuable to the attendees. It also looks toward future events that will serve international students studying health informatics and other data-centric disciplines.
Prompting for Few-shot Adverse Drug Reaction Recognition from Online Reviews

The growing popularity of social network sites (SNS) contributes to the abundance of healthcare-related user-generated content (UGC), which can be used for identifying adverse drug reactions (ADR). Most previous works on automatic ADR detection, however, require a large amount of labeled data to reach acceptable recognition performance, deemed labor-intensive and inefficient when building an ADR detection model for UGC. To address this issue, this study proposes to apply LightNER [1], a few-shot Named Entity Recognition (NER) approach that exploits the power of a pre-trained language model (i.e., BART) and introduces a small set of prompt parameters to enable few-shot learning. Our approach first warms up the prompt parameters for learning knowledge for NER from rich-resource datasets. Then, the approach only requires a few labeled UGC to fine-tune the parameters for satisfactory ADR recognition performance. In our experiment, we collect a review dataset for a migraine drug called MigraineReviews and adopt three rich-resource datasets, including one general-purpose dataset and two datasets for clinical notes. The experimental results demonstrate that BART with prompt parameters helps transfer knowledge across datasets even if they are of different domains, thus reducing the need for a large amount of labeled ADR data. Our approach performs competitively with a limited size of labeled data (e.g., 5-shots).
Developing Multi-Task Learning Methods to Aid in Electronic Healthcare Prediction

Multi-Task Learning (MTL) is a machine learning paradigm that allows for a neural network model to simultaneously train and predict on multiple related tasks/targets using a shared core network and task-specific branching layers. The related nature of the tasks can be exploited by MTL to regularize the model and its use has been shown to increase performance in all tasks. Imbalanced complexity in tasks leads to MTL models being dominated by a few tasks, which cause conflicting loss function parameters and poor training, resulting in generalized but unspecific models. Medical healthcare records (MHRs) taken on events such as pregnancy often track multiple related targets such maternal morbidity factors (i.e., Sudden Maternal Morbidity, Hemorrhage, Preeclampsia) and long-term health outcomes (i.e., cardiovascular disease, obesity, hypertension, diabetes mellitus, post-partum depression, substance use), which serve as an excellent testbed for the MTL models. This study seeks to develop solutions to task imbalance and interference for MTL systems and demonstrate their efficacy compared to traditional single-task models on a prenatal MHR dataset.
Shipwrecked: Expedition Data Challenges and the AQUAVIEW Solution

AQUAVIEW is a metadata management system that consolidates access to data from multiple expeditions and external datasets within a single platform. AQUAVIEW addresses the problem of non-co-located data by offering a unified portal for users to easily access and retrieve data from multiple sources. Similar in some respects to a traditional ETL pipeline, AQUAVIEW's focus on metadata rather than underlying database structure makes it relatively unique among approaches within the NOAA data ecosystem. This paper documents the initial development and deployment of AQUAVIEW and examines the strengths and challenges of the approach.