cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Net Benefit, Calibration, Threshold Selection, and Training Objectives for Algorithmic Fairness in Healthcare
Authors: Stephen Pfohl; Yizhe Xu; Agata Foryciarz; Nikolaos Ignatiadis; Julian Z. Genkins; Nigam H. Shah
Affiliations: Google; Stanford University; Stanford University; Stanford University; Stanford University; Stanford University
Year: 2022
Domain: Health & Clinical AI

Abstract: A growing body of work uses the paradigm of algorithmic fairness to frame the development of techniques to anticipate and proactively mitigate the introduction or exacerbation of health inequities that may follow from the use of model-guided decision-making. We evaluate the interplay between measures of model performance, fairness, and the expected utility of decision-making to offer practical recommendations for the operationalization of algorithmic fairness principles for the development and evaluation of predictive models in healthcare. We conduct an empirical case-study via development of models to estimate the ten-year risk of atherosclerotic cardiovascular disease to inform statin initiation in accordance with clinical practice guidelines. We demonstrate that approaches that incorporate fairness considerations into the model training objective typically do not improve model performance or confer greater net benefit for any of the studied patient populations compared to the use of standard learning paradigms followed by threshold selection concordant with patient preferences, evidence of intervention effectiveness, and model calibration. These results hold when the measured outcomes are not subject to differential measurement error across patient populations and threshold selection is unconstrained, regardless of whether differences in model performance metrics, such as in true and false positive error rates, are present. In closing, we argue for focusing model development efforts on developing calibrated models that predict outcomes well for all patient populations while emphasizing that such efforts are complementary to transparent reporting, participatory design, and reasoning about the impact of model-informed interventions in context.
Title: Bias in Automated Speaker Recognition
Authors: Wiebke Toussaint Hutiri; Aaron Yi Ding
Affiliations: Delft University of Technology; Delft University of Technology
Year: 2022
Domain: LLMs & NLP

Abstract: Automated speaker recognition uses data processing to identify speakers by their voice. Today, automated speaker recognition is deployed on billions of smart devices and in services such as call centres. Despite their wide-scale deployment and known sources of bias in related domains like face recognition and natural language processing, bias in automated speaker recognition has not been studied systematically. We present an in-depth empirical and analytical study of bias in the machine learning development workflow of speaker verification, a voice biometric and core task in automated speaker recognition. Drawing on an established framework for understanding sources of harm in machine learning, we show that bias exists at every development stage in the well-known VoxCeleb Speaker Recognition Challenge, including data generation, model building, and implementation. Most affected are female speakers and non-US nationalities, who experience significant performance degradation. Leveraging the insights from our findings, we make practical recommendations for mitigating bias in automated speaker recognition, and outline future research directions.
Title: Gender and Racial Bias in Visual Question Answering Datasets
Authors: Yusuke Hirota; Yuta Nakashima; Noa García
Affiliations: Osaka University; Osaka University; Osaka University
Year: 2022
Domain: LLMs & NLP

Abstract: Vision-and-language tasks have increasingly drawn more attention as a means to evaluate human-like reasoning in machine learning models. A popular task in the field is visual question answering (VQA), which aims to answer questions about images. However, VQA models have been shown to exploit language bias by learning the statistical correlations between questions and answers without looking into the image content: e.g., questions about the color of a banana are answered with yellow, even if the banana in the image is green. If societal bias (e.g., sexism, racism, ableism, etc.) is present in the training data, this problem may be causing VQA models to learn harmful stereotypes. For this reason, we investigate gender and racial bias in five VQA datasets. In our analysis, we find that the distribution of answers is highly different between questions about women and men, as well as the existence of detrimental gender-stereotypical samples. Likewise, we identify that specific race-related attributes are underrepresented, whereas potentially discriminatory samples appear in the analyzed datasets. Our findings suggest that there are dangers associated to using VQA datasets without considering and dealing with the potentially harmful stereotypes. We conclude the paper by proposing solutions to alleviate the problem before, during, and after the dataset collection process.
Title: Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification
Authors: Camille Harris; Matan Halevy; Ayanna Howard; Amy Bruckman; Diyi Yang
Affiliations: Georgia Institute of Technology; Georgia Institute of Technology; The Ohio State University; Georgia Institute of Technology; Georgia Institute of Technology
Year: 2022
Domain: LLMs & NLP

Abstract: Language usage on social media varies widely even within the context of American English. Despite this, the majority of natural language processing systems are trained only on “Standard American English,” or SAE, the construction of English most prominent among white Americans. For hate speech classification, prior work has shown that African American English (AAE) is more likely to be misclassified as hate speech. This has harmful implications for Black social media users as it reinforces and exacerbates existing notions of anti-Black racism. While past work has highlighted the relationship between AAE and hate speech classification, no work has explored the linguistic characteristics of AAE that lead to misclassification. Our work uses Twitter datasets for AAE dialect and hate speech classifiers to explore the fine-grained relationship between specific characteristics of AAE such as word choice and grammatical features and hate speech predictions. We further investigate these biases by removing profanity and examining the influence of four aspects of AAE grammar that are distinct from SAE. Results show that removing profanity accounts for a roughly 20 to 30% reduction in the percentage of samples classified as ’hate’ ’abusive’ or ’offensive,’ and that similar classification patterns are observed regardless of grammar categories.
Title: De-biasing "Bias" Measurement
Authors: Kristian Lum; Yunfeng Zhang; Amanda Bower
Affiliations: Twitter; Twitter; Twitter
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: When a model’s performance differs across socially or culturally relevant groups–like race, gender, or the intersections of many such groups–it is often called ”biased.” While much of the work in algorithmic fairness over the last several years has focused on developing various definitions of model fairness (the absence of group-wise model performance disparities) and eliminating such “bias,” much less work has gone into rigorously measuring it. In practice, it important to have high quality, human digestible measures of model performance disparities and associated uncertainty quantification about them that can serve as inputs into multi-faceted decision-making processes. In this paper, we show both mathematically and through simulation that many of the metrics used to measure group-wise model performance disparities are themselves statistically biased estimators of the underlying quantities they purport to represent. We argue that this can cause misleading conclusions about the relative group-wise model performance disparities along different dimensions, especially in cases where some sensitive variables consist of categories with few members. We propose the “double-corrected” variance estimator, which provides unbiased estimates and uncertainty quantification of the variance of model performance across groups. It is conceptually simple and easily implementable without statistical software package or numerical optimization. We demonstrate the utility of this approach through simulation and show on a real dataset that while statistically biased estimators of model group-wise model performance disparities indicate statistically significant between-group model performance disparities, when accounting for statistical bias in the estimator, the estimated group-wise disparities in model performance are no longer statistically significant.
Title: Markedness in Visual Semantic AI
Authors: Robert E. Wolfe; Aylin Caliskan
Affiliations: University of Washington; University of Washington
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: We evaluate the state-of-the-art multimodal ”visual semantic” model CLIP (”Contrastive Language Image Pretraining”) for biases related to the marking of age, gender, and race or ethnicity. Given the option to label an image as ”a photo of a person” or to select a label denoting race or ethnicity, CLIP chooses the ”person” label 47.9% of the time for White individuals, compared with 5.0% or less for individuals who are Black, East Asian, Southeast Asian, Indian, or Latino or Hispanic. The model is also more likely to rank the unmarked ”person” label higher than labels denoting gender for Male individuals (26.7% of the time) vs. Female individuals (15.2% of the time). Age also affects whether an individual is marked by the model: Female individuals under the age of 20 are more likely than Male individuals to be marked with a gender label, but less likely to be marked with an age label, while Female individuals over the age of 40 are more likely to be marked based on age than Male individuals. We trace our results back to the CLIP embedding space by examining the self-similarity (mean pairwise cosine similarity) for each social group, where higher self-similarity denotes greater attention directed by CLIP to the shared characteristics (i.e., age, race, or gender) of the social group. The results indicate that, as age increases, the self-similarity of representations of Female individuals increases at a higher rate than for Male individuals, with the disparity most pronounced at the ”more than 70” age range. Six of the ten least self-similar social groups are individuals who are White and Male, while all ten of the most self-similar social groups are individuals under the age of 10 or over the age of 70, and six of the ten are Female individuals. Our results yield evidence that bias in CLIP is intersectional: existing biases of self-similarity and markedness between Male and Female gender groups are further exacerbated when the groups compared are individuals who are White and Male and individuals who are Black and Female. CLIP is an English-language model trained on internet content gathered based on a query list generated from an American website (Wikipedia), and results indicate that CLIP reflects the biases of the language and society which produced this training data.
Title: Multi-Disciplinary Fairness Considerations in Machine Learning for Clinical Trials
Authors: Isabel Chien; Nina Deliu; Richard E. Turner; Adrian Weller; Sofía S. Villar; Niki Kilbertus
Affiliations: University of Cambridge; University of Cambridge; University of Cambridge; University of Cambridge; University of Cambridge; Helmholtz Zentrum München
Year: 2022
Domain: Health & Clinical AI

Abstract: While interest in the application of machine learning to improve healthcare has grown tremendously in recent years, a number of barriers prevent deployment in medical practice. A notable concern is the potential to exacerbate entrenched biases and existing health disparities in society. The area of fairness in machine learning seeks to address these issues of equity; however, appropriate approaches are context-dependent, necessitating domain-specific consideration. We focus on clinical trials, i.e., research studies conducted on humans to evaluate medical treatments. Clinical trials are a relatively under-explored application in machine learning for healthcare, in part due to complex ethical, legal, and regulatory requirements and high costs. Our aim is to provide a multi-disciplinary assessment of how fairness for machine learning fits into the context of clinical trials research and practice. We start by reviewing the current ethical considerations and guidelines for clinical trials and examine their relationship with common definitions of fairness in machine learning. We examine potential sources of unfairness in clinical trials, providing concrete examples, and discuss the role machine learning might play in either mitigating potential biases or exacerbating them when applied without care. Particular focus is given to adaptive clinical trials, which may employ machine learning. Finally, we highlight concepts that require further investigation and development, and emphasize new approaches to fairness that may be relevant to the design of clinical trials.
Title: What is Proxy Discrimination?
Authors: Michael Carl Tschantz
Affiliations: International Computer Science Institute
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: The near universal condemnation of proxy discrimination hides a disagreement over what it is. This work surveys various notions of proxy and proxy discrimination found in prior work and represents them in a common framework. These notions variously turn on statistical dependencies, causal effects, and intentions. It discusses the limitations and uses of each notation and of the concept as a whole.
Title: Achieving Fairness via Post-Processing in Web-Scale Recommender Systems
Authors: Preetam Nandy; Cyrus DiCiccio; Divya Venugopalan; Heloise Logan; Kinjal Basu; Noureddine El Karoui
Affiliations: LinkedIn; LinkedIn; LinkedIn; LinkedIn; LinkedIn; LinkedIn
Year: 2022
Domain: Recommender Systems

Abstract: Building fair recommender systems is a challenging and crucial area of study due to its immense impact on society. We extended the definitions of two commonly accepted notions of fairness to recommender systems, namely equality of opportunity and equalized odds. These fairness measures ensure that equally “qualified” (or “unqualified”) candidates are treated equally regardless of their protected attribute status (such as gender or race). We propose scalable methods for achieving equality of opportunity and equalized odds in rankings in the presence of position bias, which commonly plagues data generated from recommender systems. Our algorithms are model agnostic in the sense that they depend only on the final scores provided by a model, making them easily applicable to virtually all web-scale recommender systems. We conduct extensive simulations as well as real-world experiments to show the efficacy of our approach.
Title: Providing Item-Side Individual Fairness for Deep Recommender Systems
Authors: Xiuling Wang; Hui Wang
Affiliations: Stevens Institute of Technology; Stevens Institute of Technology
Year: 2022
Domain: Recommender Systems

Abstract: Recent advent of deep learning techniques have reinforced the development of new recommender systems. Although these systems have been demonstrated as efficient and effective, the issue of item popularity bias in these recommender systems has raised serious concerns. While most of the existing works focus on group fairness at item side, individual fairness at item side is left largely unexplored. To address this issue, in this paper, first, we define a new notion of individual fairness from the perspective of items, namely (α, β)-fairness, to deal with item popularity bias in recommendations. In particular, (α, β)-fairness requires that similar items should receive similar coverage in the recommendations, where α and β control item similarity and coverage similarity respectively, and both item and coverage similarity metrics are defined as task specific for deep recommender systems. Next, we design two bias mitigation methods, namely embedding-based re-ranking (ER) and greedy substitution (GS), for deep recommender systems. ER is an in-processing mitigation method that equips (α, β)-fairness as a constraint to the objective function of the recommendation algorithm, while GS is a post-processing approach that accepts the biased recommendations as the input, and substitutes high-coverage items with low-coverage ones in the recommendations to satisfy (α, β)-fairness. We evaluate the performance of both mitigation algorithms on two real-world datasets and a set of state-of-the-art deep recommender systems. Our results demonstrate that both ER and GS outperform the existing minimum-coverage (MC) mitigation solutions [Koutsopoulos and Halkidi 2018; Patro et al. 2020] in terms of both fairness and accuracy of recommendations. Furthermore, ER delivers the best trade-off between fairness and recommendation accuracy among a set of alternative mitigation methods, including GS, the hybrid of ER and GS, and the existing MC solutions.
Title: Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models
Authors: Emily Black; Hadi Elzayn; Alexandra Chouldechova; Jacob Goldin; Daniel E. Ho
Affiliations: Carnegie Mellon University; Stanford University; Carnegie Mellon University; Stanford University; Stanford University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: This study examines issues of algorithmic fairness in the context of systems that inform tax audit selection by the United States Internal Revenue Service (IRS). While the field of algorithmic fairness has developed primarily around notions of treating like individuals alike, we instead explore the concept of vertical equity—appropriately accounting for relevant differences across individuals—which is a central component of fairness in many public policy settings. Applied to the design of the U.S. individual income tax system, vertical equity relates to the fair allocation of tax and enforcement burdens across taxpayers of different income levels. Through a unique collaboration with the Treasury Department and IRS, we use access to detailed, anonymized individual taxpayer microdata, risk-selected audits, and random audits from 2010-14 to study vertical equity in tax administration. In particular, we assess how the adoption of modern machine learning methods for selecting taxpayer audits may affect vertical equity. Our paper makes four contributions. First, we show how the adoption of more flexible machine learning (classification) methods—as opposed to simpler models—shapes vertical equity by shifting audit burdens from high to middle-income taxpayers. Second, given concerns about high audit rates of low-income taxpayers, we investigate how existing algorithmic fairness techniques would change the audit distribution. We find that such methods can mitigate some disparities across income buckets, but that these come at a steep cost to performance. Third, we show that the choice of whether to treat risk of underreporting as a classification or regression problem is highly consequential. Moving from a classification approach to a regression approach to predict the expected magnitude of underreporting shifts the audit burden substantially toward high income individuals, while increasing revenue. Last, we investigate the role of differential audit cost in shaping the distribution of audits. Audits of lower income taxpayers, for instance, are typically conducted by mail and hence pose much lower cost to the IRS. We show that a narrow focus on return-on-investment can undermine vertical equity. Our results have implications for ongoing policy debates and the design of algorithmic tools across the public sector.
Title: Female, White, 27? Bias Evaluation on Data and Algorithms for Affect Recognition in Faces
Authors: Jaspar Pahl; Ines Rieger; Anna Möller; Thomas Wittenberg; Ute Schmid
Affiliations: Fraunhofer Institute for Integrated Circuits; Fraunhofer Institute for Integrated Circuits; Friedrich-Alexander-Universität Erlangen-Nürnberg; Friedrich-Alexander-Universität Erlangen-Nürnberg; Fraunhofer Institute for Integrated Circuits
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Nowadays, Artificial Intelligence (AI) algorithms show a strong performance for many use cases, making them desirable for real-world scenarios where the algorithms provide high-impact decisions. However, one major drawback of AI algorithms is their susceptibility to bias and resulting unfairness. This has a huge influence for their application, as they have a higher failure rate for certain subgroups. In this paper, we focus on the field of affective computing and particularly on the detection of bias for facial expressions. Depending on the deployment scenario, bias in facial expression models can have a disadvantageous impact and it is therefore essential to evaluate the bias and limitations of the model. In order to analyze the metadata distribution in affective computing datasets, we annotate several benchmark training datasets, containing both Action Units and categorical emotions, with age, gender, ethnicity, glasses, and beards. We show that there is a significantly skewed distribution, particularly for ethnicity and age. Based on this metadata annotation, we evaluate two trained state-of-the-art affective computing algorithms. Our evaluation shows that the strongest bias is in age, with the best performance for persons under 34 and a sharp decrease for older persons. Furthermore, we see an ethnicity bias with varying direction depending on the algorithm, a slight gender bias and worse performance for facial parts occluded by glasses.
Title: Adaptive Sampling Strategies to Construct Equitable Training Datasets
Authors: William Cai; Ro Encarnación; Bobbie Chern; Sam Corbett-Davies; Miranda Bogen; Stevie Bergman; Sharad Goel
Affiliations: ; ; ; ; ; ; Harvard University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: In domains ranging from computer vision to natural language processing, machine learning models have been shown to exhibit stark disparities, often performing worse for members of traditionally underserved groups. One factor contributing to these performance gaps is a lack of representation in the data the models are trained on. It is often unclear, however, how to operationalize representativeness in specific applications. Here we formalize the problem of creating equitable training datasets, and propose a statistical framework for addressing this problem. We consider a setting where a model builder must decide how to allocate a fixed data collection budget to gather training data from different subgroups. We then frame dataset creation as a constrained optimization problem, in which one maximizes a function of group-specific performance metrics based on (estimated) group-specific learning rates and costs per sample. This flexible approach incorporates preferences of model-builders and other stakeholders, as well as the statistical properties of the learning task. When data collection decisions are made sequentially, we show that under certain conditions this optimization problem can be efficiently solved even without prior knowledge of the learning rates. To illustrate our approach, we conduct a simulation study of polygenic risk scores on synthetic genomic data—an application domain that often suffers from non-representative data collection. When optimizing policies for overall or group-specific average health, we find that our adaptive approach outperforms heuristic strategies, including equal and representative sampling. In this sense, equal treatment with respect to sampling decisions does not guarantee equal or equitable outcomes.