cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Algorithmic fairness in computational medicine
Authors: Jie Xu; Yunyu Xiao; Hui Wang; Yue Ning; Elizabeth Shenkman; Jiang Bian; Fei Wang
Affiliations: Cornell University; Cornell University; Stevens Institute of Technology; Stevens Institute of Technology; University of Florida Health; University of Florida Health; Cornell University
Year: 2022
Domain: Health & Clinical AI

Abstract: Machine learning models are increasingly adopted for facilitating clinical decision-making. However, recent research has shown that machine learning techniques may result in potential biases when making decisions for people in different subgroups, which can lead to detrimental effects on the health and well-being of specific demographic groups such as vulnerable ethnic minorities. This problem, termed algorithmic bias, has been extensively studied in theoretical machine learning recently. However, the impact of algorithmic bias on medicine and methods to mitigate this bias remain topics of active discussion. This paper presents a comprehensive review of algorithmic fairness in the context of computational medicine, which aims at improving medicine with computational approaches. Specifically, we overview the different types of algorithmic bias, fairness quantification metrics, and bias mitigation methods, and summarize popular software libraries and tools for bias evaluation and mitigation, with the goal of providing reference and insights to researchers and practitioners in computational medicine.
Title: A framework for the automatic description of healthcare processes in natural language: Application in an aortic stenosis integrated care process
Authors: Yago Fontenla-Seco; Manuel Lama; Violeta González‐Salvado; Carlos Peña; Alberto Bugarín
Affiliations: Center for Research in Molecular Medicine and Chronic Diseases; Universidade de Santiago de Compostela; Servicio Gallego de Salud; Servicio Gallego de Salud; Universidade de Santiago de Compostela
Year: 2022
Domain: Health & Clinical AI

Abstract: In this paper, we propose a framework for the automatic generation of natural language descriptions of healthcare processes using quantitative and qualitative data and medical expert knowledge. Inspired by the demand of novel ways of conveying process mining analysis results of healthcare processes (Rojas et al., 2016), our framework is based on the most widely used Data-To-Text (D2T) pipeline (Reiter, 2007) and on the usage of process mining techniques. Backed by a general model that handles process data, this framework is able to quantify attributes in time during a process life-span, recall temporal relations and waiting times between events and its possible causes and compare case (patient) attributes between groups, among other features. Through integrating fuzzy quantification techniques, our framework is able to represent relevant quantitative process information with some degree of uncertainty present on it and describe it in natural language involving uncertain terms. A real application over the Aortic Stenosis Integrated Care Process of the University Hospital of Santiago de Compostela is presented, showcasing the potential of our framework for providing natural language descriptions of healthcare processes addressed to medical experts. Following the standards of D2T systems, manual human validation was conducted for the generated natural language descriptions by fifteen medical experts in Cardiology. Validation results are very positive, since a global average of 4.07/5.00 was achieved for questions related to understandability, usefulness and impact of the natural language descriptions on the medical experts work. More precisely, results indicate i) that the modality which conveyed the information most efficiently was natural language ii) a very clear preference of texts over the usual graphic representation of process information as the way for conveying information to experts (4.28/5.00), and iii) natural language descriptions provide relevant and useful information about the process, allowing for its improvement.
Title: Are Graph Representation Learning Methods Robust to Graph Sparsity and Asymmetric Node Information?
Authors: Pierre Sevestre; Marine Neyret
Affiliations: Société Générale; Institut Louis Bachelier
Year: 2022
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: The growing popularity of Graph Representation Learning (GRL) methods has resulted in the development of a large number of models applied to a miscellany of domains. Behind this diversity of domains, there is a strong heterogeneity of graphs, making it difficult to estimate the expected performance of a model on a new graph, especially when the graph has distinctive characteristics that have not been encountered in the benchmark yet. To address this, we have developed an experimental pipeline, to assess the impact of a given property on the models performances. In this paper, we use this pipeline to study the effect of two specificities encountered on banks transactional graphs resulting from the partial view a bank has on all the individuals and transactions carried out on the market. These specific features are graph sparsity and asymmetric node information. This study demonstrates the robustness of GRL methods to these distinctive characteristics. We believe that this work can ease the evaluation of GRL methods to specific characteristics and foster the development of such methods on transactional graphs.
Title: EDITS: Modeling and Mitigating Data Bias for Graph Neural Networks
Authors: Yushun Dong; Ninghao Liu; Brian Jalaian; Jundong Li
Affiliations: University of Virginia; University of Georgia; DEVCOM Army Research Laboratory; University of Virginia
Year: 2022
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Graph Neural Networks (GNNs) have shown superior performance in analyzing attributed networks in various web-based applications such as social recommendation and web search. Nevertheless, in high-stake decision-making scenarios such as online fraud detection, there is an increasing societal concern that GNNs could make discriminatory decisions towards certain demographic groups. Despite recent explorations on fair GNNs, these works are tailored for a specific GNN model. However, myriads of GNN variants have been proposed for different applications, and it is costly to fine-tune existing debiasing algorithms for each specific GNN architecture. Different from existing works that debias GNN models, we aim to debias the input attributed network to achieve fairer GNNs through feeding GNNs with less biased data. Specifically, we propose novel definitions and metrics to measure the bias in an attributed network, which leads to the optimization objective to mitigate bias. We then develop a framework EDITS to mitigate the bias in attributed networks while maintaining the performance of GNNs in downstream tasks. EDITS works in a model-agnostic manner, i.e., it is independent of any specific GNN. Experiments demonstrate the validity of the proposed bias metrics and the superiority of EDITS on both bias mitigation and utility maintenance. Open-source implementation: https://github.com/yushundong/EDITS.
Title: Variational Reasoning about User Preferences for Conversational Recommendation
Authors: Zhaochun Ren; Zhi Gang Tian; Dongdong Li; Pengjie Ren; Liu Yang; Xin Xin; Huasheng Liang; Maarten de Rijke; Zhumin Chen
Affiliations: Shandong University; Shandong University; Shandong University; Shandong University; Shandong University; Shandong University; Tencent (China); University of Amsterdam; Shandong University
Year: 2022
Domain: Recommender Systems

Abstract: Conversational recommender systems (CRSs) provide recommendations through interactive conversations. CRSs typically provide recommendations through relatively straightforward interactions, where the system continuously inquires about a user's explicit attribute-aware preferences and then decides which items to recommend. In addition, topic tracking is often used to provide naturally sounding responses. However, merely tracking topics is not enough to recognize a user's real preferences in a dialogue.
Title: On the Dangers of Stochastic Parrots
Authors: Emily M. Bender; Timnit Gebru; Angelina McMillan-Major; Shmargaret Shmitchell
Affiliations: University of Washington; University of Washington
Year: 2021
Domain: LLMs & NLP

Abstract: The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.
Title: Machine learning and algorithmic fairness in public and population health
Authors: Vishwali Mhasawade; Yuan Zhao; Rumi Chunara
Affiliations: New York University; New York University; New York University
Year: 2021
Domain: Health & Clinical AI

Abstract: The quality of software is becoming vital as it becomes more pervasive in human’s daily operations. Today’s software is employed differently than in the past because of automation, machine learning advancements, and the availability of copious quantities of data, all of which allow the program to make more independent judgments. The concept of fairness in software engineering refers to the different approaches used to address the problem of bias to ensure zero discrimination while using the software in question. Testing in software engineering that specifically examines a system’s fairness is known as “fairness testing”. It can also be defined as the process of revealing imperfections that can lead to some friction between the needed and existing fairness conditions, in a software system. As part of this process, it is common practice to test for bias based on demographics like color and gender to make sure the system is not prejudiced in any way. In the construction of any system that might have significant consequences for people’s lives, testing for fairness is a crucial stage that can assist guarantee the system is fair and impartial in its decision-making. Unfortunately, throughout the software development life cycle, fairness is not prioritized or even considered. There have been many instances of unfair software appearing. Using algorithms, Amazon.com, Inc. decided in 2016 which areas of the United States would get free same-day delivery. At times when every other community around it was participating, the algorithms decided against include minority areas. The same kind of software is now being utilized to assign suspects with risk assessment ratings. At each level of the criminal justice process, from determining guilt to setting bond amounts to sentence, these scores are used to guide judgments about who may be released free. This paper looks into some of the important methodologies that can be used for fairness testing in software engineering, and come up with a survey on fairness while using software systems.
Title: Ethical Machine Learning in Healthcare
Authors: Irene Y. Chen; Emma Pierson; Sherri Rose; Shalmali Joshi; Kadija Ferryman; Marzyeh Ghassemi
Affiliations: Massachusetts Institute of Technology; Microsoft (United States); Stanford University; Vector Institute; New York University; Massachusetts Institute of Technology
Year: 2021
Domain: Health & Clinical AI

Abstract: The use of machine learning (ML) in healthcare raises numerous ethical concerns, especially as models can amplify existing health inequities. Here, we outline ethical considerations for equitable ML in the advancement of healthcare. Specifically, we frame ethics of ML in healthcare through the lens of social justice. We describe ongoing efforts and outline challenges in a proposed pipeline of ethical ML in health, ranging from problem selection to postdeployment considerations. We close by summarizing recommendations to address these challenges.
Title: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations
Authors: Laleh Seyyed-Kalantari; Haoran Zhang; Matthew B. A. McDermott; Irene Y. Chen; Marzyeh Ghassemi
Affiliations: University of Toronto; Massachusetts Institute of Technology; Massachusetts Institute of Technology; Massachusetts Institute of Technology; Vector Institute
Year: 2021
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) systems have increasingly achieved expert-level performance in medical imaging applications. However, there is growing concern that such AI systems may reflect and amplify human bias, and reduce the quality of their performance in historically under-served populations such as female patients, Black patients, or patients of low socioeconomic status. Such biases are especially troubling in the context of underdiagnosis, whereby the AI algorithm would inaccurately label an individual with a disease as healthy, potentially delaying access to care. Here, we examine algorithmic underdiagnosis in chest X-ray pathology classification across three large chest X-ray datasets, as well as one multi-source dataset. We find that classifiers produced using state-of-the-art computer vision techniques consistently and selectively underdiagnosed under-served patient populations and that the underdiagnosis rate was higher for intersectional under-served subpopulations, for example, Hispanic female patients. Deployment of AI systems using medical imaging for disease diagnosis with such biases risks exacerbation of existing care biases and can potentially lead to unequal access to medical treatment, thereby raising ethical concerns for the use of these models in the clinic.
Title: Addressing bias in big data and AI for health care: A call for open science
Authors: Natalia Norori; Qiyang Hu; Florence M. Aellen; Francesca Dalia Faraci; Athina Tzovara
Affiliations: University of Bern; University of Bern; University of Bern; University of Applied Sciences and Arts of Southern Switzerland; University of Bern
Year: 2021
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) has an astonishing potential in assisting clinical decision making and revolutionizing the field of health care. A major open challenge that AI will need to address before its integration in the clinical routine is that of algorithmic bias. Most AI algorithms need big datasets to learn from, but several groups of the human population have a long history of being absent or misrepresented in existing biomedical datasets. If the training data is misrepresentative of the population variability, AI is prone to reinforcing bias, which can lead to fatal outcomes, misdiagnoses, and lack of generalization. Here, we describe the challenges in rendering AI algorithms fairer, and we propose concrete steps for addressing bias using tools from the field of open science.
Title: A survey on datasets for fairness-aware machine learning
Authors: Tai Le Quy; Arjun Roy; Vasileios Iosifidis; Wenbin Zhang; Eirini Ntoutsi
Affiliations: Leibniz University Hannover; Leibniz University Hannover; Leibniz University Hannover; Carnegie Mellon University; Freie Universität Berlin
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Abstract As decision‐making increasingly relies on machine learning (ML) and (big) data, the issue of fairness in data‐driven artificial intelligence systems is receiving increasing attention from both research and industry. A large variety of fairness‐aware ML solutions have been proposed which involve fairness‐related interventions in the data, learning algorithms, and/or model outputs. However, a vital part of proposing new approaches is evaluating them empirically on benchmark datasets that represent realistic and diverse settings. Therefore, in this paper, we overview real‐world datasets used for fairness‐aware ML. We focus on tabular data as the most common data representation for fairness‐aware ML. We start our analysis by identifying relationships between the different attributes, particularly with respect to protected attributes and class attribute, using a Bayesian network. For a deeper understanding of bias in the datasets, we investigate interesting relationships using exploratory analysis. This article is categorized under: Commercial, Legal, and Ethical Issues &gt; Fairness in Data Mining Fundamental Concepts of Data and Knowledge &gt; Data Concepts Technologies &gt; Data Preprocessing
Title: Racial Bias in Pulse Oximetry Measurement
Authors: Michael W. Sjoding; Robert P. Dickson; Theodore J. Iwashyna; Steven E. Gay; Thomas S. Valley
Affiliations: University of Michigan; University of Michigan; University of Michigan; University of Michigan; University of Michigan
Year: 2020
Domain: Health & Clinical AI

Abstract: Oxygen is among the most frequently administered medical therapies, with a level that is commonly adjusted according to the reading on a pulse oximeter that measures patients oxygen saturation. Questions about pulse oximeter technology have been raised, given its original development in populations that were not racially diverse. The clinical significance of potential racial bias in pulse oximetry measurement is unknown.
Title: Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Authors: Darshali A. Vyas; Leo G. Eisenstein; David S. Jones
Affiliations: Harvard University; Harvard University; Harvard University
Year: 2020
Domain: Health & Clinical AI

Abstract: Hidden in Plain Sight Diagnostic algorithms and practice guidelines that adjust or “correct” their outputs on the basis of a patient’s race or ethnicity guide decisions in ways that may direct more...
Title: Model Cards for Model Reporting
Authors: Margaret Mitchell; Simone Wu; Andrew Zaldivar; Parker Barnes; Lucy Vasserman; Ben Hutchinson; Elena Spitzer; Inioluwa Deborah Raji; Timnit Gebru
Affiliations: Google; Google; Google; Google; Google; Google; Google; University of Toronto; Google
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.
Title: Dissecting racial bias in an algorithm used to manage the health of populations
Authors: Ziad Obermeyer; Brian W. Powers; Christine Vogeli; Sendhil Mullainathan
Affiliations: Brigham and Women's Hospital; Brigham and Women's Hospital; Massachusetts General Hospital; University of Chicago
Year: 2019
Domain: Health & Clinical AI

Abstract: Racial bias in health algorithms The U.S. health care system uses commercial algorithms to guide health decisions. Obermeyer et al. find evidence of racial bias in one widely used algorithm, such that Black patients assigned the same level of risk by the algorithm are sicker than White patients (see the Perspective by Benjamin). The authors estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. Bias occurs because the algorithm uses health costs as a proxy for health needs. Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients. Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care. Science , this issue p. 447 ; see also p. 421
Title: The Measure and Mismeasure of Fairness
Authors: Sam Corbett‐Davies; Gaebler, Johann D.; Nilforoshan, Hamed; Shroff, Ravi; Goel, Sharad
Affiliations: N/A; Harvard University; Stanford University; New York University; Harvard University
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: The field of fair machine learning aims to ensure that decisions guided by algorithms are equitable. Over the last decade, several formal, mathematical definitions of fairness have gained prominence. Here we first assemble and categorize these definitions into two broad families: (1) those that constrain the effects of decisions on disparities; and (2) those that constrain the effects of legally protected characteristics, like race and gender, on decisions. We then show, analytically and empirically, that both families of definitions typically result in strongly Pareto dominated decision policies. For example, in the case of college admissions, adhering to popular formal conceptions of fairness would simultaneously result in lower student-body diversity and a less academically prepared class, relative to what one could achieve by explicitly tailoring admissions policies to achieve desired outcomes. In this sense, requiring that these fairness definitions hold can, perversely, harm the very groups they were designed to protect. In contrast to axiomatic notions of fairness, we argue that the equitable design of algorithms requires grappling with their context-specific consequences, akin to the equitable design of policy. We conclude by listing several open challenges in fair machine learning and offering strategies to ensure algorithms are better aligned with policy goals.