cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Towards Personalized Fairness Based on Causal Notion
Authors: Yunqi Li; Hanxiong Chen; Shuyuan Xu; Yingqiang Ge; Yongfeng Zhang
Affiliations: Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2021
Domain: Recommender Systems

Abstract: Recommender systems are gaining increasing and critical impacts on human and society since a growing number of users use them for information seeking and decision making. Therefore, it is crucial to address the potential unfairness problems in recommendations. Just like users have personalized preferences on items, users' demands for fairness are also personalized in many scenarios. Therefore, it is important to providepersonalized fair recommendations for users to satisfy theirpersonalized fairness demands. Besides, previous works on fair recommendation mainly focus on association-based fairness. However, it is important to advance from associative fairness notions to causal fairness notions for assessing fairness more properly in recommender systems. Based on the above considerations, this paper focuses on achieving personalized counterfactual fairness for users in recommender systems. To this end, we introduce a framework for achieving counterfactually fair recommendations through adversary learning by generating feature-independent user embeddings for recommendation. The framework allows recommender systems to achieve personalized fairness for users while also covering non-personalized situations. Experiments on two real-world datasets with shallow and deep recommendation algorithms show that our method can generate fairer recommendations for users with a desirable recommendation performance.
Title: Explainable Recommender Systems with Knowledge Graphs and Language Models
Authors: Giacomo Balloccu; Ludovico Boratto; Gianni Fenu; Francesca Maridina Malloci; Mirko Marras
Affiliations: University of Cagliari; University of Cagliari; University of Cagliari; University of Cagliari; University of Cagliari
Year: 2024
Domain: Recommender Systems

Abstract: Objective: The study aims to investigate whether machine learning-based predictive models for cardiovascular disease (CVD) risk assessment show equivalent performance across demographic groups (such as race and gender) and if bias mitigation methods can reduce any bias present in the models. This is important as systematic bias may be introduced when collecting and preprocessing health data, which could affect the performance of the models on certain demographic sub-cohorts. The study is to investigate this using electronic health records data and various machine learning models. Methods: The study used large de-identified Electronic Health Records data from Vanderbilt University Medical Center. Machine learning (ML) algorithms including logistic regression, random forest, gradient-boosting trees, and long short-term memory were applied to build multiple predictive models. Model bias and fairness were evaluated using equal opportunity difference (EOD, 0 indicates fairness) and disparate impact (DI, 1 indicates fairness). In our study, we also evaluated the fairness of a non-ML baseline model, the American Heart Association (AHA) Pooled Cohort Risk Equations (PCEs). Moreover, we compared the performance of three different de-biasing methods: removing protected attributes (e.g., race and gender), resampling the imbalanced training dataset by sample size, and resampling by the proportion of people with CVD outcomes. Results: The study cohort included 109,490 individuals (mean [SD] age 47.4 [14.7] years; 64.5% female; 86.3% White; 13.7% Black). The experimental results suggested that most ML models had smaller EOD and DI than PCEs. For ML models, the mean EOD ranged from −0.001 to 0.018 and the mean DI ranged from 1.037 to 1.094 across race groups. There was a larger EOD and DI across gender groups, with EOD ranging from 0.131 to 0.136 and DI ranging from 1.535 to 1.587. For debiasing methods, removing protected attributes didn’t significantly reduced the bias for most ML models. Resampling by sample size also didn’t consistently decrease bias. Resampling by case proportion reduced the EOD and DI for gender groups but slightly reduced accuracy in many cases. Conclusions: Among the VUMC cohort, both PCEs and ML models were biased against women, suggesting the need to investigate and correct gender disparities in CVD risk prediction. Resampling by proportion reduced the bias for gender groups but not for race groups
Title: Knowledge Graphs and Explainable AI in Healthcare
Authors: Enayat Rajabi; Somayeh Kafaie
Affiliations: Cape Breton University; Saint Mary's University
Year: 2022
Domain: Health & Clinical AI

Abstract: Building trust and transparency in healthcare can be achieved using eXplainable Artificial Intelligence (XAI), as it facilitates the decision-making process for healthcare professionals. Knowledge graphs can be used in XAI for explainability by structuring information, extracting features and relations, and performing reasoning. This paper highlights the role of knowledge graphs in XAI models in healthcare, considering a state-of-the-art review. Based on our review, knowledge graphs have been used for explainability to detect healthcare misinformation, adverse drug reactions, drug-drug interactions and to reduce the knowledge gap between healthcare experts and AI-based models. We also discuss how to leverage knowledge graphs in pre-model, in-model, and post-model XAI models in healthcare to make them more explainable.
Title: Constructing Knowledge Graphs and Their Biomedical Applications
Authors: David N. Nicholson; Casey S. Greene
Affiliations: Translational Therapeutics (United States); Translational Therapeutics (United States)
Year: 2020
Domain: Health & Clinical AI

Abstract: Knowledge graphs can support many biomedical applications. These graphs represent biomedical concepts and relationships in the form of nodes and edges. In this review, we discuss how these graphs are constructed and applied with a particular focus on how machine learning approaches are changing these processes. Biomedical knowledge graphs have often been constructed by integrating databases that were populated by experts via manual curation, but we are now seeing a more robust use of automated systems. A number of techniques are used to represent knowledge graphs, but often machine learning methods are used to construct a low-dimensional representation that can support many different applications. This representation is designed to preserve a knowledge graph's local and/or global structure. Additional machine learning methods can be applied to this representation to make predictions within genomic, pharmaceutical, and clinical domains. We frame our discussion first around knowledge graph construction and then around unifying representational learning techniques and unifying applications. Advances in machine learning for biomedicine are creating new opportunities across many domains, and we note potential avenues for future work with knowledge graphs that appear particularly promising.
Title: Algorithmic Fairness Datasets: The Story So Far
Authors: Alessandro Fabris; S. Messina; Gianmaria Silvello; Gian Antonio Susto
Affiliations: University of Padua; University of Padua; University of Padua; University of Padua
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Abstract Data-driven algorithms are studied and deployed in diverse domains to support critical decisions, directly impacting people’s well-being. As a result, a growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair machine learning and equitable algorithm design hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community, as a whole, suffers from a collective data documentation debt caused by a lack of information on specific resources ( opacity ) and scatteredness of available information ( sparsity ). In this work, we target this data documentation debt by surveying over two hundred datasets employed in algorithmic fairness research, and producing standardized and searchable documentation for each of them. Moreover we rigorously identify the three most popular fairness datasets, namely Adult, COMPAS, and German Credit, for which we compile in-depth documentation. This unifying documentation effort supports multiple contributions. Firstly, we summarize the merits and limitations of Adult, COMPAS, and German Credit, adding to and unifying recent scholarship, calling into question their suitability as general-purpose fairness benchmarks. Secondly, we document hundreds of available alternatives, annotating their domain and supported fairness tasks, along with additional properties of interest for fairness practitioners and researchers, including their format, cardinality, and the sensitive attributes they encode. We summarize this information, zooming in on the tasks, domains, and roles of these resources. Finally, we analyze these datasets from the perspective of five important data curation topics: anonymization, consent, inclusivity, labeling of sensitive attributes, and transparency. We discuss different approaches and levels of attention to these topics, making them tangible, and distill them into a set of best practices for the curation of novel resources.
Title: Review of Mathematical Frameworks for Fairness in Machine Learning
Authors: Eustasio del Barrio; Paula Gordaliza; Jean-Michel Loubes
Affiliations: Institut de Mathématiques de Toulouse
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: A review of the main fairness definitions and fair learning methodologies proposed in the literature over the last years is presented from a mathematical point of view. Following our independence-based approach, we consider how to build fair algorithms and the consequences on the degradation of their performance compared to the possibly unfair case. This corresponds to the price for fairness given by the criteria $\textit{statistical parity}$ or $\textit{equality of odds}$. Novel results giving the expressions of the optimal fair classifier and the optimal fair predictor (under a linear regression gaussian model) in the sense of $\textit{equality of odds}$ are presented.
Title: Did You Do Your Homework? Raising Awareness on Software Fairness and Discrimination
Authors: Max Hort; Federica Sarro
Affiliations: University College London; University College London
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Machine Learning is a vital part of various modern day decision making software. At the same time, it has shown to exhibit bias, which can cause an unjust treatment of individuals and population groups. One method to achieve fairness in machine learning software is to provide individuals with the same degree of benefit, regardless of sensitive attributes (e.g., students receive the same grade, independent of their sex or race). However, there can be other attributes that one might want to discriminate against (e.g., students with homework should receive higher grades). We will call such attributes anti-protected attributes. When reducing the bias of machine learning software, one risks the loss of discriminatory behaviour of anti-protected attributes. To combat this, we use grid search to show that machine learning software can be debiased (e.g., reduce gender bias) while also improving the ability to discriminate against anti-protected attributes.
Title: Fairness in Machine Learning: A Survey
Authors: Simon Caton; Christian Haas
Affiliations: University College Dublin; Vienna University of Economics and Business
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: When Machine Learning technologies are used in contexts that affect citizens, companies as well as researchers need to be confident that there will not be any unexpected social implications, such as bias towards gender, ethnicity, and/or people with disabilities. There is significant literature on approaches to mitigate bias and promote fairness, yet the area is complex and hard to penetrate for newcomers to the domain. This article seeks to provide an overview of the different schools of thought and approaches that aim to increase the fairness of Machine Learning. It organizes approaches into the widely accepted framework of pre-processing, in-processing, and post-processing methods, subcategorizing into a further 11 method areas. Although much of the literature emphasizes binary classification, a discussion of fairness in regression, recommender systems, and unsupervised learning is also provided along with a selection of currently available open source libraries. The article concludes by summarizing open challenges articulated as five dilemmas for fairness research.
Title: Privileged and Unprivileged Groups: An Empirical Study on the Impact of the Age Attribute on Fairness
Authors: Max Hort; Federica Sarro
Affiliations: University College London; University College London
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Recent advances in software fairness investigate bias in the treatment of different population groups, which are devised based on attributes such as gender, race and age. Groups are divided into privileged groups (favourable treatment) and unprivileged groups (unfavourable treatment). To truthfully represent the real world and to measure the degree of bias according to age (young vs. old), one needs to pick a threshold to separate those groups.
Title: Fairway: A Way to Build Fair ML Software
Authors: Joymallya Chakraborty; Suvodeep Majumder; Zhe Yu; Tim Menzies
Affiliations: North Carolina State University; North Carolina State University; North Carolina State University; North Carolina State University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning software is increasingly being used to make decisions that affect people's lives. But sometimes, the core part of this software (the learned model), behaves in a biased manner that gives undue advantages to a specific group of people (where those groups are determined by sex, race, etc.). This "algorithmic discrimination" in the AI software systems has become a matter of serious concern in the machine learning and software engineering community. There have been works done to find "algorithmic bias" or "ethical bias" in the software system. Once the bias is detected in the AI software system, the mitigation of bias is extremely important. In this work, we a)explain how ground-truth bias in training data affects machine learning model fairness and how to find that bias in AI software,b)propose a method Fairway which combines pre-processing and in-processing approach to remove ethical bias from training data and trained model. Our results show that we can find bias and mitigate bias in a learned model, without much damaging the predictive performance of that model. We propose that (1) testing for bias and (2) bias mitigation should be a routine part of the machine learning software development life cycle. Fairway offers much support for these two purposes.
Title: Do the Machine Learning Models on a Crowd-Sourced Platform Exhibit Bias?
Authors: Sumon Biswas; Hridesh Rajan
Affiliations: Iowa State University; Iowa State University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: The artifact contains code and data for the following paper published in ESEC/FSE 2020. The package can also be obtained from the GitHub repository: https://github.com/sumonbis/ML-Fairness. Title: Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness Authors: Sumon Biswas and Hridesh Rajan
Title: Fair Preprocessing: Understanding Compositional Fairness of Data Transformers in ML Pipeline
Authors: Sumon Biswas; Hridesh Rajan
Affiliations: Iowa State University; Iowa State University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: In recent years, many incidents have been reported where machine learning models exhibited discrimination among people based on race, sex, age, etc. Research has been conducted to measure and mitigate unfairness in machine learning models. For a machine learning task, it is a common practice to build a pipeline that includes an ordered set of data preprocessing stages followed by a classifier. However, most of the research on fairness has considered a single classifier based prediction task. What are the fairness impacts of the preprocessing stages in machine learning pipeline? Furthermore, studies showed that often the root cause of unfairness is ingrained in the data itself, rather than the model. But no research has been conducted to measure the unfairness caused by a specific transformation made in the data preprocessing stage. In this paper, we introduced the causal method of fairness to reason about the fairness impact of data preprocessing stages in ML pipeline. We leveraged existing metrics to define the fairness measures of the stages. Then we conducted a detailed fairness evaluation of the preprocessing stages in 37 pipelines collected from three different sources. Our results show that certain data transformers are causing the model to exhibit unfairness. We identified a number of fairness patterns in several categories of data transformers. Finally, we showed how the local fairness of a preprocessing stage composes in the global fairness of the pipeline. We used the fairness composition to choose appropriate downstream transformer that mitigates unfairness in the machine learning pipeline.
Title: Bias in Machine Learning Software: Why? How? What to Do?
Authors: Joymallya Chakraborty; Suvodeep Majumder; Tim Menzies
Affiliations: North Carolina State University; North Carolina State University; North Carolina State University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Increasingly, software is making autonomous decisions in case of criminal sentencing, approving credit cards, hiring employees, and so on. Some of these decisions show bias and adversely affect certain social groups (e.g. those defined by sex, race, age, marital status). Many prior works on bias mitigation take the following form: change the data or learners in multiple ways, then see if any of that improves fairness. Perhaps a better approach is to postulate root causes of bias and then applying some resolution strategy. This paper postulates that the root causes of bias are the prior decisions that affect- (a) what data was selected and (b) the labels assigned to those examples. Our Fair-SMOTE algorithm removes biased labels; and rebalances internal distributions such that based on sensitive attribute, examples are equal in both positive and negative classes. On testing, it was seen that this method was just as effective at reducing bias as prior approaches. Further, models generated via Fair-SMOTE achieve higher performance (measured in terms of recall and F1) than other state-of-the-art fairness improvement algorithms. To the best of our knowledge, measured in terms of number of analyzed learners and datasets, this study is one of the largest studies on bias mitigation yet presented in the literature.
Title: Fairness-Aware Machine Learning: An Extensive Overview
Authors: Jannik Dunkelau; Michael Leuschel
Affiliations: Heinrich-Heine-Universität Düsseldorf; Heinrich-Heine-Universität Düsseldorf
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: We provide an overview of the state-of-the-art in fairness-aware machine learning and examine a wide variety of research articles in the area. We survey different fairness notions, algorithms for pre-, in-, and post-processing of the data and models, and provide an overview of available frameworks.
Title: An Information Theoretic Approach to Reducing Algorithmic Bias for Machine Learning
Authors: Jinyoung Kim; Sung-Bae Cho
Affiliations: University College London; University College London
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Inherent bias in the artificial intelligence (AI)-model brings inaccuracies and variabilities during clinical deployment of the model. It is challenging to recognize the source of bias in AI-model due to variations in datasets and black box nature of system design. Additionally, there is no distinct process to identify the potential source of bias in the AI-model. To the best of our knowledge, this is the first review of its kind that addresses the bias in AI-model by categorizing 48 studies into three classes, namely, point-based, image-based, and hybrid-based AI-models. Selection strategy using PRISMA is adopted to select the 72 crucial AI studies for identifying bias in AI models. Using the three classes, bias is identified in these studies based on 44 critical AI attributes. Bias in the AI-models is computed by analytical, butterfly, and ranking-based bias models. These bias models were evaluated using two experts and compared using variability analysis. AI-studies that lacked sufficient AI-attributes are more prone to risk-of-bias (RoB) in all three classes. Studies with high RoB loses fins in the butterfly model. It has been analyzed that the majority of the studies in healthcare suffer from data bias and algorithmic bias due to incomplete specifications mentioned in the design protocol and weak AI design exploited for prediction.
Title: Representation Bias in Data: A Survey on Identification and Resolution Techniques
Authors: Nima Shahbazi; Lin Yin; Abolfazl Asudeh; H. V. Jagadish
Affiliations: North Carolina State University; North Carolina State University; North Carolina State University; North Carolina State University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Data-driven algorithms are only as good as the data they work with, while datasets, especially social data, often fail to represent minorities adequately. Representation Bias in data can happen due to various reasons, ranging from historical discrimination to selection and sampling biases in the data acquisition and preparation methods. Given that “bias in, bias out,” one cannot expect AI-based solutions to have equitable outcomes for societal applications, without addressing issues such as representation bias. While there has been extensive study of fairness in machine learning models, including several review papers, bias in the data has been less studied. This article reviews the literature on identifying and resolving representation bias as a feature of a dataset, independent of how consumed later. The scope of this survey is bounded to structured (tabular) and unstructured (e.g., image, text, graph) data. It presents taxonomies to categorize the studied techniques based on multiple design dimensions and provides a side-by-side comparison of their properties. There is still a long way to fully address representation bias issues in data. The authors hope that this survey motivates researchers to approach these challenges in the future by observing existing work within their respective domains.