cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Evaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Authors: Jonathan Huang; Galal Galal; Mozziyar Etemadi; Mahesh Vaidyanathan
Affiliations: Northwestern University; Northwestern University; Northwestern University; Northwestern University
Year: 2022
Domain: Health & Clinical AI

Abstract: BACKGROUND: Racial bias is a key concern regarding the development, validation, and implementation of machine learning (ML) models in clinical settings. Despite the potential of bias to propagate health disparities, racial bias in clinical ML has yet to be thoroughly examined and best practices for bias mitigation remain unclear. OBJECTIVE: Our objective was to perform a scoping review to characterize the methods by which the racial bias of ML has been assessed and describe strategies that may be used to enhance algorithmic fairness in clinical ML. METHODS: A scoping review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) Extension for Scoping Reviews. A literature search using PubMed, Scopus, and Embase databases, as well as Google Scholar, identified 635 records, of which 12 studies were included. RESULTS: Applications of ML were varied and involved diagnosis, outcome prediction, and clinical score prediction performed on data sets including images, diagnostic studies, clinical text, and clinical variables. Of the 12 studies, 1 (8%) described a model in routine clinical use, 2 (17%) examined prospectively validated clinical models, and the remaining 9 (75%) described internally validated models. In addition, 8 (67%) studies concluded that racial bias was present, 2 (17%) concluded that it was not, and 2 (17%) assessed the implementation of bias mitigation strategies without comparison to a baseline model. Fairness metrics used to assess algorithmic racial bias were inconsistent. The most commonly observed metrics were equal opportunity difference (5/12, 42%), accuracy (4/12, 25%), and disparate impact (2/12, 17%). All 8 (67%) studies that implemented methods for mitigation of racial bias successfully increased fairness, as measured by the authors' chosen metrics. Preprocessing methods of bias mitigation were most commonly used across all studies that implemented them. CONCLUSIONS: The broad scope of medical ML applications and potential patient harms demand an increased emphasis on evaluation and mitigation of racial bias in clinical ML. However, the adoption of algorithmic fairness principles in medicine remains inconsistent and is limited by poor data availability and ML model reporting. We recommend that researchers and journal editors emphasize standardized reporting and data availability in medical ML studies to improve transparency and facilitate evaluation for racial bias.
Title: Towards Debiasing Sentence Representations
Authors: Paul Pu Liang; Irene Mengze Li; Emily Zheng; Yao Chong Lim; Ruslan Salakhutdinov; Louis-Philippe Morency
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2020
Domain: LLMs & NLP

Abstract: As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes. Previous work has revealed the presence of social biases in widely used word embeddings involving gender, race, religion, and other social constructs. While some methods were proposed to debias these word-level embeddings, there is a need to perform debiasing at the sentence-level given the recent shift towards new contextualized sentence representations such as ELMo and BERT. In this paper, we investigate the presence of social biases in sentence-level representations and propose a new method, Sent-Debias, to reduce these biases. We show that Sent-Debias is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks such as sentiment analysis, linguistic acceptability, and natural language understanding. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
Title: Algorithmic Bias in Autonomous Systems
Authors: David Danks; Alex John London
Affiliations: Carnegie Mellon University; Carnegie Mellon University
Year: 2017
Domain: General Fairness & Bias Mitigation

Abstract: Algorithms play a key role in the functioning of autonomous systems, and so concerns have periodically been raised about the possibility of algorithmic bias. However, debates in this area have been hampered by different meanings and uses of the term, "bias." It is sometimes used as a purely descriptive term, sometimes as a pejorative term, and such variations can promote confusion and hamper discussions about when and how to respond to algorithmic bias. In this paper, we first provide a taxonomy of different types and sources of algorithmic bias, with a focus on their different impacts on the proper functioning of autonomous systems. We then use this taxonomy to distinguish between algorithmic biases that are neutral or unobjectionable, and those that are problematic in some way and require a response. In some cases, there are technological or algorithmic adjustments that developers can use to compensate for problematic bias. In other cases, however, responses require adjustments by the agent, whether human or autonomous system, who uses the results of the algorithm. There is no "one size fits all" solution to algorithmic bias.
Title: Meritocratic Fairness for Infinite and Contextual Bandits
Authors: Matthew Joseph; Michael Kearns; Jamie Morgenstern; Seth Neel; Aaron Roth
Affiliations: University of Pennsylvania; University of Pennsylvania; University of Pennsylvania; University of Pennsylvania; University of Pennsylvania
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: We study fairness in linear bandit problems. Starting from the notion of meritocratic fairness introduced in~\citeJKMR16, we carry out a more refined analysis of a more general problem, achieving better performance guarantees with fewer modelling assumptions on the number and structure of available choices as well as the number selected. We also analyze the previously-unstudied question of fairness in infinite linear bandit problems, obtaining instance-dependent regret upper bounds as well as lower bounds demonstrating that this instance-dependence is necessary. The result is a framework for meritocratic fairness in an online linear setting that is substantially more powerful, general, and realistic than the current state of the art.
Title: Trustworthy AI: A Computational Perspective
Authors: Haochen Liu; Yiqi Wang; Wenqi Fan; Xiaorui Liu; Yaxin Li; Shaili Jain; Yunhao Liu; Anil K. Jain; Jiliang Tang
Affiliations: Michigan State University; Michigan State University; The Hong Kong Polytechnic University; Michigan State University; Michigan State University; Twitter; Tsinghua University; Michigan State University; Michigan State University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: In the past few decades, artificial intelligence (AI) technology has experienced swift developments, changing everyone's daily life and profoundly altering the course of human society. The intention of developing AI is to benefit humans, by reducing human labor, bringing everyday convenience to human lives, and promoting social good. However, recent research and AI applications show that AI can cause unintentional harm to humans, such as making unreliable decisions in safety-critical scenarios or undermining fairness by inadvertently discriminating against one group. Thus, trustworthy AI has attracted immense attention recently, which requires careful consideration to avoid the adverse effects that AI may bring to humans, so that humans can fully trust and live in harmony with AI technologies. Recent years have witnessed a tremendous amount of research on trustworthy AI. In this survey, we present a comprehensive survey of trustworthy AI from a computational perspective, to help readers understand the latest technologies for achieving trustworthy AI. Trustworthy AI is a large and complex area, involving various dimensions. In this work, we focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety &amp; Robustness, (ii) Non-discrimination &amp; Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability &amp; Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems. We also discuss the accordant and conflicting interactions among different dimensions and discuss potential aspects for trustworthy AI to investigate in the future.
Title: A Survey on Datasets for Fairness-Aware ML
Authors: Tai Le Quy; Arjun Roy; Vasileios Iosifidis; Wenbin Zhang; Eirini Ntoutsi
Affiliations: Leibniz University Hannover; Leibniz University Hannover; Leibniz University Hannover; Carnegie Mellon University; Freie Universität Berlin
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: As decision-making increasingly relies on Machine Learning (ML) and (big) data, the issue of fairness in data-driven Artificial Intelligence (AI) systems is receiving increasing attention from both research and industry. A large variety of fairness-aware machine learning solutions have been proposed which involve fairness-related interventions in the data, learning algorithms and/or model outputs. However, a vital part of proposing new approaches is evaluating them empirically on benchmark datasets that represent realistic and diverse settings. Therefore, in this paper, we overview real-world datasets used for fairness-aware machine learning. We focus on tabular data as the most common data representation for fairness-aware machine learning. We start our analysis by identifying relationships between the different attributes, particularly w.r.t. protected attributes and class attribute, using a Bayesian network. For a deeper understanding of bias in the datasets, we investigate the interesting relationships using exploratory analysis.
Title: A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers
Authors: Zhenpeng Chen; Jie M. Zhang; Federica Sarro; Mark Harman
Affiliations: University College London; King’s College London; University College London; University College London
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation methods for Machine Learning (ML) classifiers, evaluated with 11 ML performance metrics (e.g., accuracy), 4 fairness metrics, and 20 types of fairness-performance trade-off assessment, applied to 8 widely-adopted software decision tasks. The empirical coverage is much more comprehensive, covering the largest numbers of bias mitigation methods, evaluation metrics, and fairness-performance trade-off measures compared to previous work on this important software property. We find that (1) the bias mitigation methods significantly decrease ML performance in 53% of the studied scenarios (ranging between 42%~66% according to different ML performance metrics); (2) the bias mitigation methods significantly improve fairness measured by the 4 used metrics in 46% of all the scenarios (ranging between 24%~59% according to different fairness metrics); (3) the bias mitigation methods even lead to decrease in both fairness and ML performance in 25% of the scenarios; (4) the effectiveness of the bias mitigation methods depends on tasks, models, the choice of protected attributes, and the set of metrics used to assess fairness and ML performance; (5) there is no bias mitigation method that can achieve the best trade-off in all the scenarios. The best method that we find outperforms other methods in 30% of the scenarios. Researchers and practitioners need to choose the bias mitigation method best suited to their intended application scenario(s).
Title: Ignorance and Prejudice in Software Fairness
Authors: Jie M. Zhang; Mark Harman
Affiliations: University College London; University College London
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning software can be unfair when making human-related decisions, having prejudices over certain groups of people. Existing work primarily focuses on proposing fairness metrics and presenting fairness improvement approaches. It remains unclear how key aspect of any machine learning system, such as feature set and training data, affect fairness. This paper presents results from a comprehensive study that addresses this problem. We find that enlarging the feature set plays a significant role in fairness (with an average effect rate of 38%). Importantly, and contrary to widely-held beliefs that greater fairness often corresponds to lower accuracy, our findings reveal that an enlarged feature set has both higher accuracy and fairness. Perhaps also surprisingly, we find that a larger training data does not help to improve fairness. Our results suggest a larger training data set has more unfairness than a smaller one when feature sets are insufficient; an important cautionary finding for practising software engineers.
Title: Removing Biased Data to Improve Fairness and Accuracy
Authors: Sahil Verma; Michael D. Ernst; René Just
Affiliations: University of Washington; University of Washington; University of Washington
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning systems are often trained using data collected from historical decisions. If past decisions were biased, then automated systems that learn from historical data will also be biased. We propose a black-box approach to identify and remove biased training data. Machine learning models trained on such debiased data (a subset of the original training data) have low individual discrimination, often 0%. These models also have greater accuracy and lower statistical disparity than models trained on the full historical data. We evaluated our methodology in experiments using 6 real-world datasets. Our approach outperformed seven previous approaches in terms of individual discrimination and accuracy.
Title: FAIR MIXUP: Fairness via Interpolation
Authors: Ching-Yao Chuang; Youssef Mroueh
Affiliations: Massachusetts Institute of Technology; IBM Research AI
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Training classifiers under fairness constraints such as group fairness, regularizes the disparities of predictions between the groups. Nevertheless, even though the constraints are satisfied during training, they might not generalize at evaluation time. To improve the generalizability of fair classifiers, we propose fair mixup, a new data augmentation strategy for imposing the fairness constraint. In particular, we show that fairness can be achieved by regularizing the models on paths of interpolated samples between the groups. We use mixup, a powerful data augmentation strategy to generate these interpolates. We analyze fair mixup and empirically show that it ensures a better generalization for both accuracy and fairness measurement in tabular, vision, and language benchmarks.
Title: Mitigating Unwanted Biases with Adversarial Learning
Authors: Brian Hu Zhang; Blake Lemoine; Margaret Mitchell
Affiliations: Stanford University; Google; Google
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning is a tool for building models that accurately represent input training data. When undesired biases concerning demographic groups are in the training data, well-trained models will reflect those biases. We present a framework for mitigating such biases by including a variable for the group of interest and simultaneously learning a predictor and an adversary. The input to the network X, here text or census data, produces a prediction Y, such as an analogy completion or income bracket, while the adversary tries to model a protected variable Z, here gender or zip code. The objective is to maximize the predictor's ability to predict Y while minimizing the adversary's ability to predict Z. Applied to analogy completion, this method results in accurate predictions that exhibit less evidence of stereotyping Z. When applied to a classification task using the UCI Adult (Census) Dataset, it results in a predictive model that does not lose much accuracy while achieving very close to equality of odds (Hardt, et al., 2016). The method is flexible and applicable to multiple definitions of fairness as well as a wide range of gradient-based learning models, including both regression and classification tasks.
Title: Black Box Fairness Testing of Machine Learning Models
Authors: Aniya Aggarwal; Pranay Lohia; Seema Nagar; Kuntal Dey; Diptikalyan Saha
Affiliations: IBM Research - India; IBM Research - India; IBM Research - India; IBM Research - India; IBM Research - India
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Any given AI system cannot be accepted unless its trustworthiness is proven. An important characteristic of a trustworthy AI system is the absence of algorithmic bias. 'Individual discrimination' exists when a given individual different from another only in 'protected attributes' (e.g., age, gender, race, etc.) receives a different decision outcome from a given machine learning (ML) model as compared to the other individual. The current work addresses the problem of detecting the presence of individual discrimination in given ML models. Detection of individual discrimination is test-intensive in a black-box setting, which is not feasible for non-trivial systems. We propose a methodology for auto-generation of test inputs, for the task of detecting individual discrimination. Our approach combines two well-established techniques - symbolic execution and local explainability for effective test case generation. We empirically show that our approach to generate test cases is very effective as compared to the best-known benchmark systems that we examine.
Title: MAAT: A Novel Ensemble Approach to Addressing Fairness and Performance Bugs for Machine Learning Software
Authors: Zhenpeng Chen; Jie M. Zhang; Federica Sarro; Mark Harman
Affiliations: University College London; King's College London; University College London; University College London
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Machine Learning (ML) software can lead to unfair and unethical decisions, making software fairness bugs an increasingly significant concern for software engineers. However, addressing fairness bugs often comes at the cost of introducing more ML performance (e.g., accuracy) bugs. In this paper, we propose MAAT, a novel ensemble approach to improving fairness-performance trade-off for ML software. Conventional ensemble methods combine different models with identical learning objectives. MAAT, instead, combines models optimized for different objectives: fairness and ML performance. We conduct an extensive evaluation of MAAT with 5 state-of-the-art methods, 9 software decision tasks, and 15 fairness-performance measurements. The results show that MAAT significantly outperforms the state-of-the-art. In particular, MAAT beats the trade-off baseline constructed by a recent benchmarking tool in 92.2% of the overall cases evaluated, 12.2 percentage points more than the best technique currently available. Moreover, the superiority of MAAT over the state-of-the-art holds on all the tasks and measurements that we study. We have made publicly available the code and data of this work to allow for future replication and extension.
Title: Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods
Authors: Max Hort; Jie M. Zhang; Federica Sarro; Mark Harman
Affiliations: University College London; University College London; University College London; University College London
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: The increasingly wide uptake of Machine Learning (ML) has raised the significance of the problem of tackling bias (i.e., unfairness), making it a primary software engineering concern. In this paper, we introduce Fairea, a model behaviour mutation approach to benchmarking ML bias mitigation methods. We also report on a large-scale empirical study to test the effectiveness of 12 widely-studied bias mitigation methods. Our results reveal that, surprisingly, bias mitigation methods have a poor effectiveness in 49% of the cases. In particular, 15% of the mitigation cases have worse fairness-accuracy trade-offs than the baseline established by Fairea; 34% of the cases have a decrease in accuracy and an increase in bias.
Title: Machine Learning Fairness Notions: Bridging the Gap with Real-World Applications
Authors: Karima Makhlouf; Sami Zhioua; Catuscia Palamidessi
Affiliations: École Polytechnique; École Polytechnique; École Polytechnique
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: RATIONALE: Studies on racial fairness in asthma hospitalization are limited due to the lack of diverse datasets and comprehensive statistical validation of underlying analytical models. We sought to examine racial and ethnic fairness when predicting asthma hospitalization outcomes among U.S. population of diverse race and ethnicity. METHODS: We have developed statistical diagnostic models to examine racial and ethnic fairness in healthcare utilization using multiple machine learning algorithms. We have accessed the fairness performance measures of our approach in an ongoing population cohort with recurrent asthma encounters in the All of Us Research Program. We implemented advance data modelling strategies to mitigate racial and ethnic fairness in asthma hospitalization predictions. RESULTS: Data consisted of 82,188 asthma encounters (37.4% White, 40.9% Black, 20.7% Hispanic, 0.7% Asian, and 0.3% MENA) with median hospital stay of 1 day and 17,862 asthma readmissions Compared to White patients, the baseline algorithm for readmission performed similar when evaluated using AUROC and specificity in all groups, but poorly with sensitivity and proportional parity among Asian, HL and MENA patients. In the race-neutral advance readmission model, Asian, HL, and MENA experienced substantial improvements in sensitivity (range: 36%-50%), and proportional parity (range: 22%-35%) compared to the baseline model in White and Black patients. CONCLUSIONS: Overall, racial unfairness in predicting asthma readmission were found in classical race-based approach. Mitigation strategies with advance modelling techniques may help to achieve equitable outcomes in asthma hospitalization outcomes, specially in underrepresented groups.