cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Algorithmic Fairness
Authors: Dana Pessach; Erez Shmueli
Affiliations: Tel Aviv University; Tel Aviv University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: This paper studies how counterfactual explanations can be used to assess the fairness of a model. Using machine learning for high-stakes decisions is a threat to fairness as these models can amplify bias present in the dataset, and there is no consensus on a universal metric to detect this. The appropriate metric and method to tackle the bias in a dataset will be case-dependent, and it requires insight into the nature of the bias first. We aim to provide this insight by integrating explainable AI (XAI) research with the fairness domain. More specifically, apart from being able to use (Predictive) Counterfactual Explanations to detect explicit bias when the model is directly using the sensitive attribute, we show that it can also be used to detect implicit bias when the model does not use the sensitive attribute directly but does use other correlated attributes leading to a substantial disadvantage for a protected group. We call this metric PreCoF, or Predictive Counterfactual Fairness. Our experimental results show that our metric succeeds in detecting occurrences of implicit bias in the model by assessing which attributes are more present in the explanations of the protected group compared to the unprotected group. These results could help policymakers decide on whether this discrimination is justified or not.
Title: A Survey on Fairness for Machine Learning on Graphs
Authors: Manvi Choudhary; Charlotte Laclau; Christine Largeron
Affiliations: Iowa State University; Iowa State University
Year: 2024
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Nowadays, the analysis of complex phenomena modeled by graphs plays a crucial role in many real-world application domains where decisions can have a strong societal impact. However, numerous studies and papers have recently revealed that machine learning models could lead to potential disparate treatment between individuals and unfair outcomes. In that context, algorithmic contributions for graph mining are not spared by the problem of fairness and present some specific challenges related to the intrinsic nature of graphs: (1) graph data is non-IID, and this assumption may invalidate many existing studies in fair machine learning, (2) suited metric definitions to assess the different types of fairness with relational data and (3) algorithmic challenge on the difficulty of finding a good trade-off between model accuracy and fairness. This survey is the first one dedicated to fairness for relational data. It aims to present a comprehensive review of state-of-the-art techniques in fairness on graph mining and identify the open challenges and future trends. In particular, we start by presenting several sensible application domains and the associated graph mining tasks with a focus on edge prediction and node classification in the sequel. We also recall the different metrics proposed to evaluate potential bias at different levels of the graph mining process; then we provide a comprehensive overview of recent contributions in the domain of fair machine learning for graphs, that we classify into pre-processing, in-processing and post-processing models. We also propose to describe existing graph data, synthetic and real-world benchmarks. Finally, we present in detail five potential promising directions to advance research in studying algorithmic fairness on graphs.
Title: Slice Tuner: A Selective Data Acquisition Framework for Fair ML Models
Authors: Ki Hyun Tae; Steven Euijong Whang
Affiliations: North Carolina State University; North Carolina State University; North Carolina State University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: As machine learning becomes democratized in the era of Software 2.0, a serious bottleneck is acquiring enough data to ensure accurate and fair models. Recent techniques including crowdsourcing provide cost-effective ways to gather such data. However, simply acquiring data as much as possible is not necessarily an effective strategy for optimizing accuracy and fairness. For example, if an online app store has enough training data for certain slices of data (say American customers), but not for others, obtaining more American customer data will only bias the model training. Instead, we contend that one needs to selectively acquire data and propose Slice Tuner, which acquires possibly-different amounts of data per slice such that the model accuracy and fairness on all slices are optimized. This problem is different than labeling existing data (as in active learning or weak supervision) because the goal is obtaining the right amounts of new data. At its core, Slice Tuner maintains learning curves of slices that estimate the model accuracies given more data and uses convex optimization to find the best data acquisition strategy. The key challenges of estimating learning curves are that they may be inaccurate if there is not enough data, and there may be dependencies among slices where acquiring data for one slice influences the learning curves of others. We solve these issues by iteratively and efficiently updating the learning curves as more data is acquired. We evaluate Slice Tuner on real datasets using crowdsourcing for data acquisition and show that Slice Tuner significantly outperforms baselines in terms of model accuracy and fairness, even when the learning curves cannot be reliably estimated.
Title: Optimized Score Transformation for Consistent Fair Classification
Authors: Dennis Wei; Karthikeyan Natesan Ramamurthy; Flávio P. Calmon
Affiliations: Cornell University; Google (United States); Google (United States); Microsoft Research (United Kingdom); Cornell University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: This paper considers fair probabilistic binary classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing a cross-entropy objective. The formulation can be applied directly to post-process classifier outputs and we also explore a pre-processing extension, thus allowing maximum freedom in selecting a classification algorithm. We derive a closed-form expression for the optimal transformed scores and a convex optimization problem for the transformation parameters. In the population limit, the transformed score function is the fairness-constrained minimizer of cross-entropy with respect to the true conditional probability of the outcome. In the finite sample setting, we propose a method called FairScoreTransformer to approach this solution using a combination of standard probabilistic classifiers and ADMM. We provide several consistency and finite-sample guarantees for FairScoreTransformer, relating to the transformation parameters and transformed score function that it obtains. Comprehensive experiments comparing to 10 existing methods show that FairScoreTransformer has advantages for score-based metrics such as Brier score and AUC while remaining competitive for binary label-based metrics such as accuracy.
Title: Counterfactual Fairness
Authors: Matt J. Kusner; Joshua R. Loftus; Chris Russell; Ricardo Silva
Affiliations: Université du Québec à Montréal; Higher Colleges of Technology; Institut Polytechnique de Paris
Year: 2017
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning can impact people with legal or ethical consequences when it is used to automate decisions in areas such as insurance, lending, hiring, and predictive policing. In many of these scenarios, previous decisions have been made that are unfairly biased against certain subpopulations, for example those of a particular race, gender, or sexual orientation. Since this past data may be biased, machine learning predictors must account for this to avoid perpetuating or creating discriminatory practices. In this paper, we develop a framework for modeling fairness using tools from causal inference. Our definition of counterfactual fairness captures the intuition that a decision is fair towards an individual if it is the same in (a) the actual world and (b) a counterfactual world where the individual belonged to a different demographic group. We demonstrate our framework on a real-world problem of fair prediction of success in law school.
Title: Algorithmic Fairness: Choices Assumptions and Definitions
Authors: Shira Mitchell; Eric Potash; Solon Barocas; Alexander D'Amour; Kristian Lum
Affiliations: University of Zurich; University of Zurich; University of Zurich
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: A recent wave of research has attempted to define fairness quantitatively. In particular, this work has explored what fairness might mean in the context of decisions based on the predictions of statistical and machine learning models. The rapid growth of this new field has led to wildly inconsistent motivations, terminology, and notation, presenting a serious challenge for cataloging and comparing definitions. This article attempts to bring much-needed order. First, we explicate the various choices and assumptions made—often implicitly—to justify the use of prediction-based decision-making. Next, we show how such choices and assumptions can raise fairness concerns and we present a notationally consistent catalog of fairness definitions from the literature. In doing so, we offer a concise reference for thinking through the choices, assumptions, and fairness considerations of prediction-based decision-making.
Title: On the Applicability of Machine Learning Fairness Notions
Authors: Karima Makhlouf; Sami Zhioua; Catuscia Palamidessi
Affiliations: Carnegie Mellon University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Machine Learning (ML) based predictive systems are increasingly used to support decisions with a critical impact on individuals' lives such as college admission, job hiring, child custody, criminal risk assessment, etc. As a result, fairness emerged as an important requirement to guarantee that ML predictive systems do not discriminate against specific individuals or entire sub-populations, in particular, minorities. Given the inherent subjectivity of viewing the concept of fairness, several notions of fairness have been introduced in the literature. This paper is a survey of fairness notions that, unlike other surveys in the literature, addresses the question of "which notion of fairness is most suited to a given real-world scenario and why?". Our attempt to answer this question consists in (1) identifying the set of fairness-related characteristics of the real-world scenario at hand, (2) analyzing the behavior of each fairness notion, and then (3) fitting these two elements to recommend the most suitable fairness notion in every specific setup. The results are summarized in a decision diagram that can be used by practitioners and policy makers to navigate the relatively large catalogue of ML fairness notions.
Title: Approval of AI/ML-Based Medical Devices in USA & Europe
Authors: Urs J. Muehlematter; Paola Daniore; Kerstin Noëlle Vokinger
Affiliations: Brigham and Women's Hospital; Brigham and Women's Hospital; George Washington University; Brigham and Women's Hospital; Brigham and Women's Hospital; Brigham and Women's Hospital; Brigham and Women's Hospital
Year: 2021
Domain: Health & Clinical AI

Abstract: There has been a surge of interest in artificial intelligence and machine learning (AI/ML)-based medical devices. However, it is poorly understood how and which AI/ML-based medical devices have been approved in the USA and Europe. We searched governmental and non-governmental databases to identify 222 devices approved in the USA and 240 devices in Europe. The number of approved AI/ML-based devices has increased substantially since 2015, with many being approved for use in radiology. However, few were qualified as high-risk devices. Of the 124 AI/ML-based devices commonly approved in the USA and Europe, 80 were first approved in Europe. One possible reason for approval in Europe before the USA might be the potentially relatively less rigorous evaluation of medical devices in Europe. The substantial number of approved devices highlight the need to ensure rigorous regulation of these devices. Currently, there is no specific regulatory pathway for AI/ML-based medical devices in the USA or Europe. We recommend more transparency on how devices are regulated and approved to enable and improve public trust, efficacy, safety, and quality of AI/ML-based medical devices. A comprehensive, publicly accessible database with device details for Conformité Européene (CE)-marked medical devices in Europe and US Food and Drug Administration approved devices is needed.
Title: AI in Medicine: Structural Challenges in Improving Patient Care
Authors: Alex John London
Affiliations: United States Food and Drug Administration; United States Food and Drug Administration; United States Food and Drug Administration; United States Food and Drug Administration
Year: 2022
Domain: Health & Clinical AI

Abstract: There is considerable enthusiasm about the prospect that artificial intelligence (AI) will help to improve the safety and efficacy of health services and the efficiency of health systems. To realize this potential, however, AI systems will have to overcome structural problems in the culture and practice of medicine and the organization of health systems that impact the data from which AI models are built, the environments into which they will be deployed, and the practices and incentives that structure their development. This perspective elaborates on some of these structural challenges and provides recommendations to address potential shortcomings.
Title: FDA-Approved ML Algorithms in Neuroradiology
Authors: Alexander G. Yearley; Caroline M.W. Goedmakers; Armon Panahi; Joanne Doucette; Aakanksha Rana; Kavitha Ranganathan; Timothy R. Smith
Affiliations: Fraunhofer Institute for Digital Medicine; Technische Universität Berlin; Fraunhofer Institute for Digital Medicine; University Hospital Carl Gustav Carus; Technische Universität Berlin; Humboldt-Universität zu Berlin; Fraunhofer Institute for Digital Medicine; RWTH Aachen University; Medical University of Graz; Technische Universität Berlin; Technische Universität Berlin; Medical University of Graz; Humboldt-Universität zu Berlin; Helmholtz-Zentrum Dresden-Rossendorf; RWTH Aachen University; RWTH Aachen University; RWTH Aachen University; Olympus (Germany); RWTH Aachen University; Medical University of Graz; Humboldt-Universität zu Berlin; Humboldt-Universität zu Berlin
Year: 2023
Domain: Health & Clinical AI

Abstract: Large Language Models (LLMs) have significantly advanced natural language processing applications, yet their widespread use raises concerns regarding inherent biases that may reduce utility or harm for particular social groups. Despite the advancement in addressing LLM bias, existing research has two major limitations. First, existing LLM bias evaluation focuses on the U.S. cultural context, making it challenging to reveal stereotypical biases of LLMs toward other cultures, leading to unfair development and use of LLMs. Second, current bias evaluation often assumes models are familiar with the target social groups. When LLMs encounter words beyond their knowledge boundaries that are unfamiliar in their training data, they produce irrelevant results in the local context due to hallucinations and overconfidence, which are not necessarily indicative of inherent bias. This research addresses these limitations with a Local Integrated Bias Recognition and Assessment Framework (LIBRA) for measuring bias using datasets sourced from local corpora without crowdsourcing. Implementing this framework, we develop a dataset comprising over 360,000 test cases in the New Zealand context. Furthermore, we propose the Enhanced Idealized CAT Score (EiCAT), integrating the iCAT score with a beyond knowledge boundary score (bbs) and a distribution divergence-based bias measurement to tackle the challenge of LLMs encountering words beyond knowledge boundaries. Our results show that the BERT family, GPT-2, and Llama-3 models seldom understand local words in different contexts. While Llama-3 exhibits larger bias, it responds better to different cultural contexts.
Title: Data Drift in Medical ML: Implications and Remedies
Authors: Berkman Sahiner; Weijie Chen; Ravi K. Samala; Nicholas Petrick
Affiliations: Cardiovascular Institute of the South; Stanford University; Stanford Medicine; Cardiovascular Institute of the South
Year: 2023
Domain: Health & Clinical AI

Abstract: Data drift refers to differences between the data used in training a machine learning (ML) model and that applied to the model in real-world operation. Medical ML systems can be exposed to various forms of data drift, including differences between the data sampled for training and used in clinical operation, differences between medical practices or context of use between training and clinical use, and time-related changes in patient populations, disease patterns, and data acquisition, to name a few. In this article, we first review the terminology used in ML literature related to data drift, define distinct types of drift, and discuss in detail potential causes within the context of medical applications with an emphasis on medical imaging. We then review the recent literature regarding the effects of data drift on medical ML systems, which overwhelmingly show that data drift can be a major cause for performance deterioration. We then discuss methods for monitoring data drift and mitigating its effects with an emphasis on pre- and post-deployment techniques. Some of the potential methods for drift detection and issues around model retraining when drift is detected are included. Based on our review, we find that data drift is a major concern in medical ML deployment and that more research is needed so that ML models can identify drift early, incorporate effective mitigation strategies and resist performance decay.
Title: Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology
Authors: André Homeyer; Christian Geißler; Lars Ole Schwen; Falk Zakrzewski; Theodore Evans; Klaus Strohmenger; Max Westphal; Roman D. Bülow; Michaela Kargl; Aray Karjauv; Isidre Munné-Bertran; Carl Orge Retzlaff; Adrià Romero-López; Tomasz Sołtysiński; Markus Plass; Rita Carvalho; Peter Steinbach; Yu-Chia Lan; Nassim Bouteldja; David Haber; Mateo Rojas-Carulla; Alireza Vafaei Sadr; Matthias Kraft; Daniel Krüger; Rutger Fick; Tobias Lang; Peter Boor; Heimo Müller; Peter Hufnagl; Norman Zerbe
Affiliations: Fraunhofer Institute for Digital Medicine; Technische Universität Berlin; Fraunhofer Institute for Digital Medicine; University Hospital Carl Gustav Carus; Technische Universität Berlin; Humboldt-Universität zu Berlin; Fraunhofer Institute for Digital Medicine; RWTH Aachen University; Medical University of Graz; Technische Universität Berlin; Technische Universität Berlin; Medical University of Graz; Humboldt-Universität zu Berlin; Helmholtz-Zentrum Dresden-Rossendorf; RWTH Aachen University; RWTH Aachen University; RWTH Aachen University; Olympus (Germany); RWTH Aachen University; Medical University of Graz; Humboldt-Universität zu Berlin; Humboldt-Universität zu Berlin
Year: 2022
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.
Title: Shifting ML in Healthcare: From Development to Deployment
Authors: Angela Zhang; Lei Xing; James Zou; Joseph C. Wu
Affiliations: Cardiovascular Institute of the South; Stanford University; Stanford Medicine; Cardiovascular Institute of the South
Year: 2022
Domain: Health & Clinical AI

Abstract: In the past decade, the application of machine learning (ML) to healthcare has helped drive the automation of physician tasks as well as enhancements in clinical capabilities and access to care. This progress has emphasized that, from model development to model deployment, data play central roles. In this Review, we provide a data-centric view of the innovations and challenges that are defining ML for healthcare. We discuss deep generative models and federated learning as strategies to augment datasets for improved model performance, as well as the use of the more recent transformer models for handling larger datasets and enhancing the modelling of clinical text. We also discuss data-focused problems in the deployment of ML, emphasizing the need to efficiently deliver data to ML models for timely clinical predictions and to account for natural data shifts that can deteriorate model performance.
Title: Towards a Fair Marketplace: Fairness & Satisfaction in Recommender Systems
Authors: Rishabh Mehrotra; James O. McInerney; Hugues Bouchard; Mounia Lalmas; Fernando Díaz
Affiliations: Microsoft (Canada)
Year: 2022
Domain: Recommender Systems

Abstract: Two-sided marketplaces are platforms that have customers not only on the demand side (e.g. users), but also on the supply side (e.g. retailer, artists). While traditional recommender systems focused specifically towards increasing consumer satisfaction by providing relevant content to consumers, two-sided marketplaces face the problem of additionally optimizing for supplier preferences, and visibility. Indeed, the suppliers would want afair opportunity to be presented to users. Blindly optimizing for consumer relevance may have a detrimental impact on supplier fairness. Motivated by this problem, we focus on the trade-off between objectives of consumers and suppliers in the case of music streaming services, and consider the trade-off betweenrelevance of recommendations to the consumer (i.e. user) andfairness of representation of suppliers (i.e. artists) and measure their impact on consumersatisfaction.