cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Fairness definitions explained
Authors: Sahil Verma; Julia Rubin
Affiliations: Indian Institute of Technology Kanpur; University of British Columbia
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: Algorithm fairness has started to attract the attention of researchers in AI, Software Engineering and Law communities, with more than twenty different notions of fairness proposed in the last few years. Yet, there is no clear agreement on which definition to apply in each situation. Moreover, the detailed differences between multiple definitions are difficult to grasp. To address this issue, this paper collects the most prominent definitions of fairness for the algorithmic classification problem, explains the rationale behind these definitions, and demonstrates each of them on a single unifying case-study. Our analysis intuitively explains why the same case can be considered fair according to some definitions and unfair according to others.
Title: Word embeddings quantify 100 years of gender and ethnic stereotypes
Authors: Nikhil Garg; Londa Schiebinger; Dan Jurafsky; James Zou
Affiliations: Stanford University; Stanford University; Stanford University; Chan Zuckerberg Initiative (United States)
Year: 2018
Domain: LLMs & NLP

Abstract: Word embeddings are a powerful machine-learning framework that represents each English word by a vector. The geometric relationship between these vectors captures meaningful semantic relationships between the corresponding words. In this paper, we develop a framework to demonstrate how the temporal dynamics of the embedding helps to quantify changes in stereotypes and attitudes toward women and ethnic minorities in the 20th and 21st centuries in the United States. We integrate word embeddings trained on 100 y of text data with the US Census to show that changes in the embedding track closely with demographic and occupation shifts over time. The embedding captures societal shifts-e.g., the women's movement in the 1960s and Asian immigration into the United States-and also illuminates how specific adjectives and occupations became more closely associated with certain populations over time. Our framework for temporal analysis of word embedding opens up a fruitful intersection between machine learning and quantitative social science.
Title: AI can be sexist and racist — it’s time to make it fair
Authors: James Zou; Londa Schiebinger
Affiliations: Stanford University; Health Innovations (United States)
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data, argue James Zou and Londa Schiebinger. Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data.
Title: Ensuring Fairness in Machine Learning to Advance Health Equity
Authors: Alvin Rajkomar; Michaela Hardt; Michael Howell; Greg S. Corrado; Marshall H. Chin
Affiliations: University of California, San Francisco; Google (United States); Google (United States); Google (United States); University of Chicago
Year: 2018
Domain: Health & Clinical AI

Abstract: Machine learning is used increasingly in clinical care to improve diagnosis, treatment selection, and health system efficiency. Because machine-learning models learn from historically collected data, populations that have experienced human and structural biases in the past-called protected groups-are vulnerable to harm by incorrect predictions or withholding of resources. This article describes how model design, biases in data, and the interactions of model predictions with clinicians and patients may exacerbate health care disparities. Rather than simply guarding against these harms passively, machine-learning systems should be used proactively to advance health equity. For that goal to be achieved, principles of distributive justice must be incorporated into model design, deployment, and evaluation. The article describes several technical implementations of distributive justice-specifically those that ensure equality in patient outcomes, performance, and resource allocation-and guides clinicians as to when they should prioritize each principle. Machine learning is providing increasingly sophisticated decision support and population-level monitoring, and it should encode principles of justice to ensure that models benefit all patients.
Title: The accuracy, fairness, and limits of predicting recidivism
Authors: Julia Dressel; Hany Farid
Affiliations: Dartmouth College; Dartmouth College
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: Algorithms for predicting recidivism are commonly used to assess a criminal defendant's likelihood of committing a crime. These predictions are used in pretrial, parole, and sentencing decisions. Proponents of these systems argue that big data and advanced machine learning make these analyses more accurate and less biased than humans. We show, however, that the widely used commercial risk assessment software COMPAS is no more accurate or fair than predictions made by people with little or no criminal justice expertise. We further show that a simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.
Title: Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments
Authors: Alexandra Chouldechova
Affiliations: Carnegie Mellon University
Year: 2017
Domain: General Fairness & Bias Mitigation

Abstract: Recidivism prediction instruments (RPIs) provide decision-makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. Although such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy concerns potential discriminatory bias in the risk assessments that are produced. This article discusses several fairness criteria that have recently been applied to assess the fairness of RPIs. We demonstrate that the criteria cannot all be simultaneously satisfied when recidivism prevalence differs across groups. We then show how disparate impact can arise when an RPI fails to satisfy the criterion of error rate balance.
Title: Equality of Opportunity in Supervised Learning
Authors: Moritz Hardt; Eric Price; Nathan Srebro
Affiliations: Google (United States); The University of Texas at Austin; Toyota Technological Institute at Chicago
Year: 2016
Domain: General Fairness & Bias Mitigation

Abstract: We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. In line with other studies, our notion is oblivious: it depends only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation of individualfeatures. We study the inherent limits of defining and identifying biases based on such oblivious measures, outlining what can and cannot be inferred from different oblivious tests. We illustrate our notion using a case study of FICO credit scores.
Title: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Authors: Tolga Bolukbasi; Kai-Wei Chang; James Zou; Venkatesh Saligrama; Adam Tauman Kalai
Affiliations: Microsoft Research New England (United States); Microsoft Research New England (United States); Microsoft Research New England (United States); Microsoft Research New England (United States)
Year: 2016
Domain: LLMs & NLP

Abstract: The blind application of machine learning runs the risk of amplifying biases present in data. Such a danger is facing us with word embedding, a popular framework to represent text data as vectors which has been used in many machine learning and natural language processing tasks. We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent. This raises concerns because their widespread use, as we describe, often tends to amplify these biases. Geometrically, gender bias is first shown to be captured by a direction in the word embedding. Second, gender neutral words are shown to be linearly separable from gender definition words in the word embedding. Using these properties, we provide a methodology for modifying an embedding to remove gender stereotypes, such as the association between between the words receptionist and female, while maintaining desired associations such as between the words queen and female. We define metrics to quantify both direct and indirect gender biases in embeddings, and develop algorithms to "debias" the embedding. Using crowd-worker evaluation as well as standard benchmarks, we empirically demonstrate that our algorithms significantly reduce gender bias in embeddings while preserving the its useful properties such as the ability to cluster related concepts and to solve analogy tasks. The resulting embeddings can be used in applications without amplifying gender bias.
Title: Fairness through awareness
Authors: Cynthia Dwork; Moritz Hardt; Toniann Pitassi; Omer Reingold; Richard S. Zemel
Affiliations: Microsoft (United States); IBM Research - Almaden; University of Toronto; Microsoft (United States); University of Toronto
Year: 2012
Domain: General Fairness & Bias Mitigation

Abstract: We study fairness in classification, where individuals are classified, e.g., admitted to a university, and the goal is to prevent discrimination against individuals based on their membership in some group, while maintaining utility for the classifier (the university). The main conceptual contribution of this paper is a framework for fair classification comprising (1) a (hypothetical) task-specific metric for determining the degree to which individuals are similar with respect to the classification task at hand; (2) an algorithm for maximizing utility subject to the fairness constraint, that similar individuals are treated similarly. We also present an adaptation of our approach to achieve the complementary goal of "fair affirmative action," which guarantees statistical parity (i.e., the demographics of the set of individuals receiving any classification are the same as the demographics of the underlying population), while treating similar individuals as similarly as possible. Finally, we discuss the relationship of fairness to privacy: when fairness implies privacy, and how tools developed in the context of differential privacy may be applied to fairness.
Title: “How Biased are Your Features?”: Computing Fairness Influence Functions with Global Sensitivity Analysis
Authors: Bishwamittra Ghosh; Debabrota Basu; Kuldeep S. Meel
Affiliations: National University of Singapore; Centre National de la Recherche Scientifique; National University of Singapore
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Fairness in machine learning has attained significant focus due to the widespread application in high-stake decision-making tasks. Unregulated machine learning classifiers can exhibit bias towards certain demographic groups in data, thus the quantification and mitigation of classifier bias is a central concern in fairness in machine learning. In this paper, we aim to quantify the influence of different features in a dataset on the bias of a classifier. To do this, we introduce the Fairness Influence Function (FIF). This function breaks down bias into its components among individual features and the intersection of multiple features. The key idea is to represent existing group fairness metrics as the difference of the scaled conditional variances in the classifier’s prediction and apply a decomposition of variance according to global sensitivity analysis. To estimate FIFs, we instantiate an algorithm that applies variance decomposition of classifier’s prediction following local regression. Experiments demonstrate that captures FIFs of individual feature and intersectional features, provides a better approximation of bias based on FIFs, demonstrates higher correlation of FIFs with fairness interventions, and detects changes in bias due to fairness affirmative/punitive actions in the classifier.
Title: A Critical Survey on Fairness Benefits of Explainable AI
Authors: Luca Deck; Jakob Schoeffer; Maria De‐Arteaga; Niklas Kühl
Affiliations: University of Bayreuth; The University of Texas at Austin; The University of Texas at Austin; University of Bayreuth
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In this critical survey, we analyze typical claims on the relationship between explainable AI (XAI) and fairness to disentangle the multidimensional relationship between these two concepts. Based on a systematic literature review and a subsequent qualitative content analysis, we identify seven archetypal claims from 175 scientific articles on the alleged fairness benefits of XAI. We present crucial caveats with respect to these claims and provide an entry point for future discussions around the potentials and limitations of XAI for specific fairness desiderata. Importantly, we notice that claims are often (i) vague and simplistic, (ii) lacking normative grounding, or (iii) poorly aligned with the actual capabilities of XAI. We suggest to conceive XAI not as an ethical panacea but as one of many tools to approach the multidimensional, sociotechnical challenge of algorithmic fairness. Moreover, when making a claim about XAI and fairness, we emphasize the need to be more specific about what kind of XAI method is used, which fairness desideratum it refers to, how exactly it enables fairness, and who is the stakeholder that benefits from XAI.
Title: A deep-learning method for the end-to-end prediction of intracranial aneurysm rupture risk
Authors: Peiying Li; Yongchang Liu; Jiafeng Zhou; Shikui Tu; Bing Zhao; Jieqing Wan; Yunjun Yang; Lei Xu
Affiliations: Shanghai Jiao Tong University; Shanghai Jiao Tong University; Wenzhou Medical University; Shanghai Jiao Tong University; Shanghai Jiao Tong University; Shanghai Jiao Tong University; Wenzhou Medical University; Shanghai Jiao Tong University
Year: 2023
Domain: Health & Clinical AI

Abstract: It is critical to accurately predict the rupture risk of an intracranial aneurysm (IA) for timely and appropriate treatment because the fatality rate after rupture is 50%. Existing methods relying on morphological features (e.g., height-width ratio) measured manually by neuroradiologists are labor intensive and have limited use for risk assessment. Therefore, we propose an end-to-end deep-learning method, called TransIAR net, to automatically learn the morphological features from 3D computed tomography angiography (CTA) data and accurately predict the status of IA rupture. We devise a multiscale 3D convolutional neural network (CNN) to extract the structural patterns of the IA and its neighborhood with a dual branch of shared network structures. Moreover, we learn the spatial dependence within the IA neighborhood with a transformer encoder. Our experiments demonstrated that the features learned by TransIAR are more effective and robust than handcrafted features, resulting in a 10%−15% improvement in the accuracy of rupture status prediction.
Title: A Framework for Assurance Audits of Algorithmic Systems
Authors: Khoa T. Lam; Benjamin Lange; Borhane Blili-Hamelin; Jovana Davidovic; Shea Brown; Ali Hasan
Affiliations: University of Iowa; University of Iowa; University of Iowa
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: An increasing number of regulations propose ‘AI audits’ as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently lacks agreed-upon practices, procedures, taxonomies, and standards. We propose the ‘criterion audit’ as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations’ ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which ‘bias audits’ can be performed for in-scope hiring algorithms, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion—informed by experiences in performing these audits in practice—highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of audits.
Title: A scientometric analysis of fairness in health AI literature
Authors: Isabelle Rose I. Alberto; Nicole Rose I. Alberto; Yüksel Altınel; Sarah Blacker; William W. Binotti; Leo Anthony Celi; Tiffany Chua; Amelia Fiske; Molly Griffin; Gülce Karaca; Nkiruka Mokolo; David Kojo N. Naawu; Jonathan Patscheider; Anton Petushkov; Justin Quion; Charles Senteio; Simon Taisbak; İsmail Tırnova; H. Tokashiki; Adrian Velasquez; Antonio Yaghy; Keagan Yap
Affiliations: Bağcılar Eğitim ve Araştırma Hastanesi; York University; Tufts Medical Center; Beth Israel Deaconess Medical Center; University of San Francisco; Technical University of Munich; Beth Israel Deaconess Medical Center; Massachusetts General Hospital; Meharry Medical College; Meharry Medical College; University of Michigan; University of the East Ramon Magsaysay Memorial Medical Center; Rutgers, The State University of New Jersey; Başkent University; Carney Hospital; Brown University; Cambridge College
Year: 2024
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) and machine learning are central components of today's medical environment. The fairness of AI, i.e. the ability of AI to be free from bias, has repeatedly come into question. This study investigates the diversity of members of academia whose scholarship poses questions about the fairness of AI. The articles that combine the topics of fairness, artificial intelligence, and medicine were selected from Pubmed, Google Scholar, and Embase using keywords. Eligibility and data extraction from the articles were done manually and cross-checked by another author for accuracy. Articles were selected for further analysis, cleaned, and organized in Microsoft Excel; spatial diagrams were generated using Public Tableau. Additional graphs were generated using Matplotlib and Seaborn. Linear and logistic regressions were conducted using Python to measure the relationship between funding status, number of citations, and the gender demographics of the authorship team. We identified 375 eligible publications, including research and review articles concerning AI and fairness in healthcare. Analysis of the bibliographic data revealed that there is an overrepresentation of authors that are white, male, and are from high-income countries, especially in the roles of first and last author. Additionally, analysis showed that papers whose authors are based in higher-income countries were more likely to be cited more often and published in higher impact journals. These findings highlight the lack of diversity among the authors in the AI fairness community whose work gains the largest readership, potentially compromising the very impartiality that the AI fairness community is working towards.
Title: A scoping review of fair machine learning techniques when using real-world data
Authors: Yu Huang; Jingchuan Guo; Wei‐Han Chen; Hsin-Yueh Lin; Huilin Tang; Fei Wang; Hua Xu; Jiang Bian
Affiliations: University of Florida Health; University of Florida; University of Florida; University of Florida; University of Florida; Cornell University; Yale University; University of Florida Health
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: OBJECTIVE: The integration of artificial intelligence (AI) and machine learning (ML) in health care to aid clinical decisions is widespread. However, as AI and ML take important roles in health care, there are concerns about AI and ML associated fairness and bias. That is, an AI tool may have a disparate impact, with its benefits and drawbacks unevenly distributed across societal strata and subpopulations, potentially exacerbating existing health inequities. Thus, the objectives of this scoping review were to summarize existing literature and identify gaps in the topic of tackling algorithmic bias and optimizing fairness in AI/ML models using real-world data (RWD) in health care domains. METHODS: We conducted a thorough review of techniques for assessing and optimizing AI/ML model fairness in health care when using RWD in health care domains. The focus lies on appraising different quantification metrics for accessing fairness, publicly accessible datasets for ML fairness research, and bias mitigation approaches. RESULTS: We identified 11 papers that are focused on optimizing model fairness in health care applications. The current research on mitigating bias issues in RWD is limited, both in terms of disease variety and health care applications, as well as the accessibility of public datasets for ML fairness research. Existing studies often indicate positive outcomes when using pre-processing techniques to address algorithmic bias. There remain unresolved questions within the field that require further research, which includes pinpointing the root causes of bias in ML models, broadening fairness research in AI/ML with the use of RWD and exploring its implications in healthcare settings, and evaluating and addressing bias in multi-modal data. CONCLUSION: This paper provides useful reference material and insights to researchers regarding AI/ML fairness in real-world health care data and reveals the gaps in the field. Fair AI/ML in health care is a burgeoning field that requires a heightened research focus to cover diverse applications and different types of RWD.