cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Bias Discovery in Machine Learning Models for Mental Health
Authors: Pablo Mosteiro; Jesse Kuiper; Judith Masthoff; Floortje Scheepers; Marco Spruit
Affiliations: Utrecht University; Utrecht University; Utrecht University; University Medical Center Utrecht; Leiden University
Year: 2022
Domain: Health & Clinical AI

Abstract: Fairness and bias are crucial concepts in artificial intelligence, yet they are relatively ignored in machine learning applications in clinical psychiatry. We computed fairness metrics and present bias mitigation strategies using a model trained on clinical mental health data. We collected structured data related to the admission, diagnosis, and treatment of patients in the psychiatry department of the University Medical Center Utrecht. We trained a machine learning model to predict future administrations of benzodiazepines on the basis of past data. We found that gender plays an unexpected role in the predictions—this constitutes bias. Using the AI Fairness 360 package, we implemented reweighing and discrimination-aware regularization as bias mitigation strategies, and we explored their implications for model performance. This is the first application of bias exploration and mitigation in a machine learning model trained on real clinical psychiatry data.
Title: Bias in machine learning models can be significantly mitigated by careful training: Evidence from neuroimaging studies
Authors: Rongguang Wang; Pratik Chaudhari; Christos Davatzikos
Affiliations: University of Pennsylvania; University of Pennsylvania; University of Pennsylvania
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Despite the great promise that machine learning has offered in many fields of medicine, it has also raised concerns about potential biases and poor generalization across genders, age distributions, races and ethnicities, hospitals, and data acquisition equipment and protocols. In the current study, and in the context of three brain diseases, we provide evidence which suggests that when properly trained, machine learning models can generalize well across diverse conditions and do not necessarily suffer from bias. Specifically, by using multistudy magnetic resonance imaging consortia for diagnosing Alzheimer's disease, schizophrenia, and autism spectrum disorder, we find that well-trained models have a high area-under-the-curve (AUC) on subjects across different subgroups pertaining to attributes such as gender, age, racial groups and different clinical studies and are unbiased under multiple fairness metrics such as demographic parity difference, equalized odds difference, equal opportunity difference, etc. We find that models that incorporate multisource data from demographic, clinical, genetic factors, and cognitive scores are also unbiased. These models have a better predictive AUC across subgroups than those trained only with imaging features, but there are also situations when these additional features do not help.
Title: Causal Fairness Analysis: A Causal Toolkit for Fair Machine Learning
Authors: Drago Plečko; Elias Bareinboim
Affiliations: ETH Zurich; University of Missouri
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Decision-making systems based on AI and machine learning have been used throughout a wide range of real-world scenarios, including healthcare, law enforcement, education, and finance. It is no longer far-fetched to envision a future where autonomous systems will drive entire business decisions and, more broadly, support large-scale decision-making infrastructure to solve society’s most challenging problems. Issues of unfairness and discrimination are pervasive when decisions are being made by humans, and remain (or are potentially amplified) when decisions are made using machines with little transparency, accountability, and fairness. In this monograph, we introduce a framework for causal fairness analysis with the intent of filling in this gap, i.e., understanding, modeling, and possibly solving issues of fairness in decision-making settings. The main insight of our approach will be to link the quantification of the disparities present in the observed data with the underlying, often unobserved, collection of causal mechanisms that generate the disparity in the first place, a challenge we call the Fundamental Problem of Causal Fairness Analysis (FPCFA). In order to solve the FPCFA, we study the problem of decomposing variations and empirical measures of fairness that attribute such variations to structural mechanisms and different units of the population. Our effort culminates in the Fairness Map, the first systematic attempt to organize and explain the relationship between various criteria found in the literature. Finally, we study which causal assumptions are minimally needed for performing causal fairness analysis and propose the Fairness Cookbook, which allows one to assess the existence of disparate impact and disparate treatment.
Title: Deep Multimodal Data Fusion
Authors: Fei Zhao; Chengcui Zhang; Baocheng Geng
Affiliations: University of Alabama at Birmingham; University of Alabama at Birmingham; University of Alabama at Birmingham
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction, combination/fusion), and decision-making (e.g., majority vote). As architectures become more and more sophisticated, multimodal neural networks can integrate feature extraction, feature fusion, and decision-making processes into one single model. The boundaries between those processes are increasingly blurred. The conventional multimodal data fusion taxonomy (e.g., early/late fusion), based on which the fusion occurs in, is no longer suitable for the modern deep learning era. Therefore, based on the main-stream techniques used, we propose a new fine-grained taxonomy grouping the state-of-the-art (SOTA) models into five classes: Encoder-Decoder methods, Attention Mechanism methods, Graph Neural Network methods, Generative Neural Network methods, and other Constraint-based methods. Most existing surveys on multimodal data fusion are only focused on one specific task with a combination of two specific modalities. Unlike those, this survey covers a broader combination of modalities, including Vision + Language (e.g., videos, texts), Vision + Sensors (e.g., images, LiDAR), and so on, and their corresponding tasks (e.g., video captioning, object detection). Moreover, a comparison among these methods is provided, as well as challenges and future directions in this area.
Title: Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates
Authors: Alex Jiahong Lu; Cameron Moy; Mark S. Ackerman; Jeffrey D. Morenoff; Tawanna R. Dillahunt
Affiliations: University of Michigan–Ann Arbor; Rutgers, The State University of New Jersey; University of Pennsylvania; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In Detroit, the largest Black-majority city in the United States, municipal authorities have deployed an array of surveillance technologies with the promise of containing crime and improving community safety. This article draws from a cross-sectional survey of over two thousand Detroit residents and multi-year community-based fieldwork in Detroit’s Eastside to examine local perceptions of policing surveillance technologies. Our survey reveals that respondents, notably those in more vulnerable positions, report higher perceived safety levels with policing surveillance cameras in their neighborhoods. However, when triangulating these results with insights from our fieldwork, we argue that these survey findings should not be taken as public support for surveillance. Alongside this seeming buy-in is a widely shared “better than nothing” imaginary among residents from impacted communities. “Better than nothing,” for the residents, is a pragmatic compromise and maneuver between being aware of the inherent flaws of surveillance technologies and settling for any available resource or hope. This notion of “better than nothing” unveils residents’ prolonged wait for digital justice and institutional accountability, which we show is where racialized infrastructural harm and exploitation are enacted along the temporal dimension. Our findings offer practical insights for counter-surveillance advocacy efforts.
Title: Ethnic Classifications in Algorithmic Fairness: Concepts, Measures and Implications in Practice
Authors: Sofia Jaime; Christoph Kern
Affiliations: University of Mannheim
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: We address the challenges and implications of ensuring fairness in algorithmic decision-making (ADM) practices related to ethnicity. Expanding beyond the U.S.-centric approach to race, we provide an overview of ethnic classification schemes in European countries and emphasize how the distinct approaches to ethnicity in Europe can impact fairness assessments in ADM. Drawing on large-scale German survey data, we highlight differences in ethnic disadvantage across subpopulations defined by different measures of ethnicity. We build prediction models in the labor market, health, and finance domain and investigate the fairness implications of different ethnic classification schemes across multiple prediction tasks and fairness metrics. Our results show considerable variation in fairness scores across ethnic classifications, where error disparities for the same model can be twice as large when using different operationalizations of ethnicity. We argue that ethnic classifications differ in their ability to identify ethnic disadvantage across ADM domains and advocate for context-sensitive operationalizations of ethnicity and its transparent reporting in fair machine learning (ML) applications.
Title: Ethnic disparity in diagnosing asymptomatic bacterial vaginosis using machine learning
Authors: Cameron Celeste; Dion Ming; Justin Broce; Diandra P. Ojo; Emma Drobina; Adetola Louis‐Jacques; Juan E. Gilbert; Ruogu Fang; Ivana Parker
Affiliations: University of Florida; University of Florida; University of Florida; University of Florida; University of Florida; University of Florida; University of Florida; University of Florida; University of Florida
Year: 2023
Domain: Health & Clinical AI

Abstract: While machine learning (ML) has shown great promise in medical diagnostics, a major challenge is that ML models do not always perform equally well among ethnic groups. This is alarming for women's health, as there are already existing health disparities that vary by ethnicity. Bacterial Vaginosis (BV) is a common vaginal syndrome among women of reproductive age and has clear diagnostic differences among ethnic groups. Here, we investigate the ability of four ML algorithms to diagnose BV. We determine the fairness in the prediction of asymptomatic BV using 16S rRNA sequencing data from Asian, Black, Hispanic, and white women. General purpose ML model performances vary based on ethnicity. When evaluating the metric of false positive or false negative rate, we find that models perform least effectively for Hispanic and Asian women. Models generally have the highest performance for white women and the lowest for Asian women. These findings demonstrate a need for improved methodologies to increase model fairness for predicting BV.
Title: Evaluating and mitigating bias in machine learning models for cardiovascular disease prediction
Authors: Fuchen Li; Patrick Wu; Henry H. Ong; Josh F. Peterson; Wei‐Qi Wei; Juan Zhao
Affiliations: Vanderbilt University; Vanderbilt University Medical Center; Vanderbilt University Medical Center; Vanderbilt University Medical Center; Vanderbilt University Medical Center; Vanderbilt University Medical Center
Year: 2023
Domain: Health & Clinical AI

Abstract: This paper advocates for inclusive AI development, emphasizing its necessity for global equity, ethical soundness, and social relevance. We detail MIT Critical Data's approach to equitable AI development, focusing on healthcare. Our methods prioritize diverse collaboration and community engagement. Through global datathons, open-source datasets, and accessible education, we empower the global majority to actively participate in shaping AI that benefits all. Significant results, including numerous publications and established community hubs, demonstrate the impact of this approach. We argue that inclusivity in AI is not only achievable but crucial for its future success and fairness, particularly in serving the global majority.
Title: Evidence of What, for Whom? The Socially Contested Role of Algorithmic Bias in a Predictive Policing Tool
Authors: Marta Ziosi; Dasha Pruss
Affiliations: University of Oxford; Harvard University Press
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: This paper presents a critical, qualitative study of the social role of algorithmic bias in the context of the Chicago crime prediction algorithm, a predictive policing tool that forecasts when and where in the city crime is most likely to occur. Through interviews with 18 Chicago-area community organizations, academic researchers, and public sector actors, we show that stakeholders from different groups articulate diverse problem diagnoses of the tool’s algorithmic bias, strategically using it as evidence to advance criminal justice interventions that align with stakeholders’ positionality and political ends. Drawing inspiration from Catherine D’Ignazio’s taxonomy of “refusing and using” data, we find that stakeholders use evidence of algorithmic bias to reform the policies around police patrol allocation; reject algorithm-based policing interventions; reframe crime as a structural rather than interpersonal problem; reveal data on authority figures in an effort to subvert their power; repair and heal families and communities; and, in the case of more powerful actors, to reaffirm their own authority or existing power structures. We identify the implicit assumptions and scope of these varied uses of algorithmic bias as evidence, showing that they require different (and sometimes conflicting) values about policing and AI. This divergence reflects long-standing tensions in the criminal justice reform landscape between the values of liberation and healing often centered by system-impacted communities and the values of surveillance and deterrence often instantiated in data-driven reform measures. We advocate for centering the interests and experiential knowledge of communities impacted by incarceration to ensure that evidence of algorithmic bias can serve as a device to challenge the status quo.
Title: Fairness and bias correction in machine learning for depression prediction across four study populations
Authors: Vien Ngoc Dang; Anna Cascarano; Rosa H. Mulder; Charlotte A. M. Cecil; María A. Zuluaga; Jerónimo Hernández-González; Karim Lekadir
Affiliations: Universitat de Barcelona; Universitat de Barcelona; Erasmus MC; Erasmus University Rotterdam; EURECOM; Universitat de Girona; Universitat de Barcelona
Year: 2024
Domain: Health & Clinical AI

Abstract: A significant level of stigma and inequality exists in mental healthcare, especially in under-served populations. Inequalities are reflected in the data collected for scientific purposes. When not properly accounted for, machine learning (ML) models learned from data can reinforce these structural inequalities or biases. Here, we present a systematic study of bias in ML models designed to predict depression in four different case studies covering different countries and populations. We find that standard ML approaches regularly present biased behaviors. We also show that mitigation techniques, both standard and our own post-hoc method, can be effective in reducing the level of unfair bias. There is no one best ML model for depression prediction that provides equality of outcomes. This emphasizes the importance of analyzing fairness during model selection and transparent reporting about the impact of debiasing interventions. Finally, we also identify positive habits and open challenges that practitioners could follow to enhance fairness in their models.
Title: Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
Authors: Sierra Wyllie; Ilia Shumailov; Nicolas Papernot
Affiliations: University of Toronto; University of Oxford; University of Toronto
Year: 2024
Domain: Health & Clinical AI

Abstract: Model-induced distribution shifts (MIDS) occur as previous model outputs pollute new model training sets over generations of models. This is known as model collapse in the case of generative models, and performative prediction or unfairness feedback loops for supervised models. When a model induces a distribution shift, it also encodes its mistakes, biases, and unfairnesses into the ground truth of its data ecosystem. We introduce a framework that allows us to track multiple MIDS over many generations, finding that they can lead to loss in performance, fairness, and minoritized group representation, even in initially unbiased datasets. Despite these negative consequences, we identify how models might be used for positive, intentional, interventions in their data ecosystems, providing redress for historical discrimination through a framework called algorithmic reparation (AR). We simulate AR interventions by curating representative training batches for stochastic gradient descent to demonstrate how AR can improve upon the unfairnesses of models and data ecosystems subject to other MIDS. Our work takes an important step towards identifying, mitigating, and taking accountability for the unfair feedback loops enabled by the idea that ML systems are inherently neutral and objective.
Title: Fairness in AI-Based Mental Health: Clinician Perspectives and Bias Mitigation
Authors: Gizem Soğancıoğlu; Pablo Mosteiro; Albert Ali Salah; Floortje Scheepers; Heysem Kaya
Affiliations: Utrecht University; Utrecht University; Utrecht University; University Medical Center Utrecht; Utrecht University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: There is limited research on fairness in automated decision-making systems in the clinical domain, particularly in the mental health domain. Our study explores clinicians' perceptions of AI fairness through two distinct scenarios: violence risk assessment and depression phenotype recognition using textual clinical notes. We engage with clinicians through semi-structured interviews to understand their fairness perceptions and to identify appropriate quantitative fairness objectives for these scenarios. Then, we compare a set of bias mitigation strategies developed to improve at least one of the four selected fairness objectives. Our findings underscore the importance of carefully selecting fairness measures, as prioritizing less relevant measures can have a detrimental rather than a beneficial effect on model behavior in real-world clinical use.
Title: Fairness in Graph Machine Learning: Recent Advances and Future Prospectives
Authors: Yushun Dong; Öykü Deniz Köse; Yanning Shen; Jundong Li
Affiliations: University of Virginia; University of California, Irvine; University of California, Irvine; University of Virginia
Year: 2023
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Graph machine learning algorithms have become popular tools in helping us gain a deeper understanding of the ubiquitous graph data. Despite their effectiveness, most graph machine learning algorithms lack considerations for fairness, which can result in discriminatory outcomes against certain demographic subgroups or individuals. As a result, there is a growing societal concern about mitigating the bias exhibited in these algorithms. To tackle the problem of algorithmic bias in graph machine learning algorithms, this tutorial aims to provide a comprehensive overview of recent research progress in measuring and mitigating the bias in machine learning algorithms on graphs. Specifically, this tutorial first introduces several widely-used fairness notions and the corresponding metrics. Then, we present a well-organized review of the theoretical understanding of bias in graph machine learning algorithms, followed by a summary of existing techniques to debias graph machine learning algorithms. Furthermore, we demonstrate how different real-world applications benefit from these graph machine learning algorithms after debiasing. Finally, we provide insights on current research challenges and open questions to encourage further advances.
Title: Fairness in Graph Mining: A Survey
Authors: Yushun Dong; Jing Ma; Song Wang; Chen Chen; Jundong Li
Affiliations: University of Virginia; University of Virginia; University of Virginia; Biocom; University of Virginia
Year: 2023
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Graph mining algorithms have been playing a significant role in myriad fields over the years. However, despite their promising performance on various graph analytical tasks, most of these algorithms lack fairness considerations. As a consequence, they could lead to discrimination towards certain populations when exploited in human-centered applications. Recently, algorithmic fairness has been extensively studied in graph-based applications. In contrast to algorithmic fairness on independent and identically distributed (i.i.d.) data, fairness in graph mining has exclusive backgrounds, taxonomies, and fulfilling techniques. In this survey, we provide a comprehensive and up-to-date introduction of existing literature under the context of fair graph mining. Specifically, we propose a novel taxonomy of fairness notions on graphs, which sheds light on their connections and differences. We further present an organized summary of existing techniques that promote fairness in graph mining. Finally, we discuss current research challenges and open questions, aiming at encouraging cross-breeding ideas and further advances.
Title: Fairness in Reinforcement Learning: A Survey
Authors: Anka Reuel; Devin Ma
Affiliations: Stanford University; California University of Pennsylvania
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: While our understanding of fairness in machine learning has significantly progressed, our understanding of fairness in reinforcement learning (RL) remains nascent. Most of the attention has been on fairness in one-shot classification tasks; however, real-world, RL-enabled systems (e.g., autonomous vehicles) are much more complicated in that agents operate in dynamic environments over a long period of time. To ensure the responsible development and deployment of these systems, we must better understand fairness in RL. In this paper, we survey the literature to provide the most up-to-date snapshot of the frontiers of fairness in RL. We start by reviewing where fairness considerations can arise in RL, then discuss the various definitions of fairness in RL that have been put forth thus far. We continue to highlight the methodologies researchers used to implement fairness in single- and multi-agent RL systems and showcase the distinct application domains that fair RL has been investigated in. Finally, we critically examine gaps in the literature, such as understanding fairness in the context of RLHF, that still need to be addressed in future work to truly operationalize fair RL in real-world systems.