cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Fairness Indicators for Systematic Assessments of Visual Feature Extractors
Authors: Priya Goyal; Adriana Romero; Caner Hazırbas; Levent Sagun; Nicolas Usunier
Affiliations: Meta AI Research; Meta AI Research; Meta AI Research; Meta AI Research; Meta AI Research
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Does everyone equally benefit from computer vision systems? Answers to this question become more and more important as computer vision systems are deployed at large scale, and can spark major concerns when they exhibit vast performance discrepancies between people from various demographic and social backgrounds.
Title: A Data-Driven Analysis of the Interplay between Criminological Theory and Predictive Policing Algorithms
Authors: Adriane Chapman; Philip J Grylls; Pamela Ugwudike; David Gammack; Jacqui Ayling
Affiliations: University of Southampton; University of Southampton; University of Southampton; University of Southampton; University of Southampton
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Previous studies have focused on the biases and feedback loops that occur in predictive policing algorithms. These studies show how systemically and institutionally biased data leads to these feedback loops when predictive policing algorithms are applied in real life. We take a step back, and show that the choice in algorithm can be embedded in a specific criminological theory, and that the choice of a model on its own even without biased data can create biased feedback loops. By synthesizing “historical” data, in which we control the relationships between crimes, location and time, we show that the current predictive policing algorithms create biased feedback loops even with completely random data. We then review the process of creation and deployment of these predictive systems, and highlight when good practices, such as fitting a model to data, “go bad” within the context of larger system development and deployment. Using best practices from previous work on assessing and mitigating the impact of new technologies, we highlight where the design of these algorithms has broken down. The study also found that multidisciplinary analysis of such systems is vital for uncovering these issues and shows that any study of equitable AI should involve a systematic and holistic analysis of their design rationalities.
Title: Evidence for Hypodescent in Visual Semantic AI
Authors: Robert E. Wolfe; Mahzarin R. Banaji; Aylin Caliskan
Affiliations: University of Washington; Harvard University; University of Washington
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: We examine the state-of-the-art multimodal ”visual semantic” model CLIP (”Contrastive Language Image Pretraining”) for the rule of hypodescent, or one-drop rule, whereby multiracial people are more likely to be assigned a racial or ethnic label corresponding to a minority or disadvantaged racial or ethnic group than to the equivalent majority or advantaged group. A face morphing experiment grounded in psychological research demonstrating hypodescent indicates that, at the midway point of 1,000 series of morphed images, CLIP associates 69.7% of Black-White female images with a Black text label over a White text label, and similarly prefers Latina (75.8%) and Asian (89.1%) text labels at the midway point for Latina-White female and Asian-White female morphs, reflecting hypodescent. Additionally, assessment of the underlying cosine similarities in the model reveals that association with White is correlated with association with ”person,” with Pearson’s ρ as high as 0.82, p < 10− 90 over a 21,000-image morph series, indicating that a White person corresponds to the default representation of a person in CLIP. Finally, we show that the stereotype-congruent pleasantness association of an image correlates with association with the Black text label in CLIP, with Pearson’s ρ = 0.48, p < 10− 90 for 21,000 Black-White multiracial male images, and ρ = 0.41, p < 10− 90 for Black-White multiracial female images. CLIP is trained on English-language text gathered using data collected from an American website (Wikipedia), and our findings demonstrate that CLIP embeds the values of American racial hierarchy, reflecting the implicit and explicit beliefs that are present in human minds. We contextualize these findings within the history of and psychology of hypodescent. Overall, the data suggests that AI supervised using natural language will, unless checked, learn biases that reflect racial hierarchies.
Title: Is Calibration a Fairness Requirement?
Authors: Michele Loi; Christoph Heitz
Affiliations: Politecnico di Milano; ZHAW Zurich University of Applied Sciences
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: In this paper, we provide a moral analysis of two criteria of statistical fairness debated in the machine learning literature: 1) calibration between groups and 2) equality of false positive and false negative rates between groups. In our paper, we focus on moral arguments in support of either measure. The conflict between group calibration vs. false positive and false negative rate equality is one of the core issues in the debate about group fairness definitions among practitioners. For any thorough moral analysis, the meaning of the term “fairness” has to be made explicit and defined properly. For our paper, we equate fairness with (non-)discrimination, which is a legitimate understanding in the discussion about group fairness. More specifically, we equate it with “prima facie wrongful discrimination” in the sense this is used in Prof. Lippert-Rasmussen's treatment of this definition. In this paper, we argue that a violation of group calibration may be unfair in some cases, but not unfair in others. Our argument analyzes in great detail two specific hypothetical examples of usage of predictions in decision making. The most important practical implication is that between-group calibration is defensible as a bias standard in some cases but not others; we show this by referring to examples in which the violation of between-group calibration is discriminatory, and others in which it is not. This is in line with claims already advanced in the literature, that algorithmic fairness should be defined in a way that is sensitive to context. The most important practical implication is that arguments based on examples in which fairness requires between-group calibration, or equality in the false-positive/false-negative rates, do no generalize. For it may be that group calibration is a fairness requirement in one case, but not in another.
Title: Subverting Fair Image Search with Generative Adversarial Perturbations
Authors: Avijit Ghosh; Matthew Jagielski; Christo Wilson
Affiliations: Northeastern University; Google; Northeastern University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: In this work we explore the intersection fairness and robustness in the context of ranking: when a ranking model has been calibrated to achieve some definition of fairness, is it possible for an external adversary to make the ranking model behave unfairly without having access to the model or training data? To investigate this question, we present a case study in which we develop and then attack a state-of-the-art, fairness-aware image search engine using images that have been maliciously modified using a Generative Adversarial Perturbation (GAP) model [75]. These perturbations attempt to cause the fair re-ranking algorithm to unfairly boost the rank of images containing people from an adversary-selected subpopulation.
Title: FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes
Authors: Alan Mishler; Edward H. Kennedy
Affiliations: JPMorgan Chase; Carnegie Mellon University
Year: 2022
Domain: Health & Clinical AI

Abstract: Methods for building fair predictors often involve tradeoffs between fairness and accuracy and between different fairness criteria. Recent work seeks to characterize these tradeoffs in specific problem settings, but these methods often do not accommodate users who wish to improve the fairness of an existing benchmark model without sacrificing accuracy, or vice versa. These results are also typically restricted to observable accuracy and fairness criteria. We develop a flexible framework for fair ensemble learning that allows users to efficiently explore the fairness-accuracy space or to improve the fairness or accuracy of a benchmark model. Our framework can simultaneously target multiple observable or counterfactual fairness criteria, and it enables users to combine a large number of previously trained and newly trained predictors. We provide theoretical guarantees that our estimators converge at fast rates. We apply our method on both simulated and real data, with respect to both observable and counterfactual accuracy and fairness criteria. We show that, surprisingly, multiple unfairness measures can sometimes be minimized simultaneously with little impact on accuracy, relative to unconstrained predictors or existing benchmark models.
Title: Selection in the Presence of Implicit Bias: The Advantage of Intersectional Constraints
Authors: Anay Mehrotra; Bary Pradelski; Nisheeth K. Vishnoi
Affiliations: Yale University; CNRS; Yale University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: In selection processes such as hiring, promotion, and college admissions, implicit bias toward socially-salient attributes such as race, gender, or sexual orientation of candidates is known to produce persistent inequality and reduce aggregate utility for the decision maker. Interventions such as the Rooney Rule and its generalizations, which require the decision maker to select at least a specified number of individuals from each affected group, have been proposed to mitigate the adverse effects of implicit bias in selection. Recent works have established that such lower-bound constraints can be very effective in improving aggregate utility in the case when each individual belongs to at most one affected group. However, in several settings, individuals may belong to multiple affected groups and, consequently, face more extreme implicit bias due to this intersectionality. We consider independently drawn utilities and show that, in the intersectional case, the aforementioned non-intersectional constraints can only recover part of the total utility achievable in the absence of implicit bias. On the other hand, we show that if one includes appropriate lower-bound constraints on the intersections, almost all the utility achievable in the absence of implicit bias can be recovered. Thus, intersectional constraints can offer a significant advantage over a reductionist dimension-by-dimension non-intersectional approach to reducing inequality.
Title: ABCinML: Anticipatory Bias Correction in Machine Learning Applications
Authors: Abdulaziz A. Almuzaini; Chidansh Bhatt; David M. Pennock; Vivek Kumar Singh
Affiliations: Rutgers University; IBM; Rutgers University; Rutgers University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: The idealization of a static machine-learned model, trained once and deployed forever, is not practical. As input distributions change over time, the model will not only lose accuracy, any constraints to reduce bias against a protected class may fail to work as intended. Thus, researchers have begun to explore ways to maintain algorithmic fairness over time. One line of work focuses on dynamic learning: retraining after each batch, and the other on robust learning which tries to make algorithms robust against all possible future changes. Dynamic learning seeks to reduce biases soon after they have occurred and robust learning often yields (overly) conservative models. We propose an anticipatory dynamic learning approach for correcting the algorithm to mitigate bias before it occurs. Specifically, we make use of anticipations regarding the relative distributions of population subgroups (e.g., relative ratios of male and female applicants) in the next cycle to identify the right parameters for an importance weighing fairness approach. Results from experiments over multiple real-world datasets suggest that this approach has promise for anticipatory bias correction.
Title: Trade-offs between Group Fairness Metrics in Societal Resource Allocation
Authors: Tasfia Mashiat; Xavier Gitiaux; Huzefa Rangwala; Patrick J. Fowler; Sanmay Das
Affiliations: George Mason University; George Mason University; George Mason University; Washington University in St. Louis; George Mason University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: We consider social resource allocations that deliver an array of scarce supports to a diverse population. Such allocations pervade social service delivery, such as provision of homeless services and assignment of refugees to cities, among others. At issue is whether allocations are fair across sociodemographic groups and intersectional identities. Our paper shows that necessary trade-offs exist for fairness in the context of scarcity; many reasonable definitions of equitable outcomes cannot hold simultaneously except under stringent conditions. For example, defining fairness in terms of improvement over a baseline inherently conflicts with defining fairness in terms of loss compared with the best possible outcome. Moreover, we demonstrate that the fairness trade-offs stem from heterogeneity across groups in intervention responses. Administrative records on homeless service delivery offer a real-world example. Building on prior work, we measure utilities for each household as the probability of reentry into homeless services if given three homeless services. Heterogeneity in utility distributions (conditional on received services) for several sociodemographic groups (e.g. single women with children versus without children) generates divergence across fairness metrics. We argue that such heterogeneity, and thus, fairness trade-offs, pervade many social policy contexts.
Title: Minimax Demographic Group Fairness in Federated Learning
Authors: Afroditi Papadaki; Natalia J. Martinez; Martín Bertrán; Guillermo Sapiro; Miguel Tréfaut Rodrigues
Affiliations: University College London; Duke University; Duke University; Duke University; University College London
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Federated learning is an increasingly popular paradigm that enables a large number of entities to collaboratively learn better models. In this work, we study minimax group fairness in federated learning scenarios where different participating entities may only have access to a subset of the population groups during the training phase. We formally analyze how our proposed group fairness objective differs from existing federated learning fairness criteria that impose similar performance across participants instead of demographic groups. We provide an optimization algorithm – FedMinMax – for solving the proposed problem that provably enjoys the performance guarantees of centralized learning algorithms. We experimentally compare the proposed approach against other state-of-the-art methods in terms of group fairness in various federated learning setups, showing that our approach exhibits competitive or superior performance.
Title: Fair Representation Clustering with Several Protected Classes
Authors: Zhen Dai; Yury Makarychev; Ali Vakilian
Affiliations: University of Chicago; Toyota Technological Institute at Chicago; Toyota Technological Institute at Chicago
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: We study the problem of fair k-median where each cluster is required to have a fair representation of individuals from different groups. In the fair representation k-median problem, we are given a set of points X in a metric space. Each point x ∈ X belongs to one of ℓ groups. Further, we are given fair representation parameters αj and βj for each group j ∈ [ℓ]. We say that a k-clustering C1, ⋅⋅⋅, Ck fairly represents all groups if the number of points from group j in cluster Ci is between αj|Ci| and βj|Ci| for every j ∈ [ℓ] and i ∈ [k]. The goal is to find a set of k centers and an assignment such that the clustering defined by fairly represents all groups and minimizes the ℓ1-objective ∑x ∈ Xd(x, ϕ(x)).
Title: Improving Fairness in AI Models on Electronic Health Records: The Case for Federated Learning Methods
Authors: Raphael Poulain; Mirza Farhan Bin Tarek; Rahmatollah Beheshti
Affiliations: University of Delaware; University of Delaware; University of Delaware
Year: 2023
Domain: Health & Clinical AI

Abstract: Developing AI tools that preserve fairness is of critical importance, specifically in high-stakes applications such as those in healthcare. However, health AI models' overall prediction performance is often prioritized over the possible biases such models could have. In this study, we show one possible approach to mitigate bias concerns by having healthcare institutions collaborate through a federated learning paradigm (FL; which is a popular choice in healthcare settings). While FL methods with an emphasis on fairness have been previously proposed, their underlying model and local implementation techniques, as well as their possible applications to the healthcare domain remain widely underinvestigated. Therefore, we propose a comprehensive FL approach with adversarial debiasing and a fair aggregation method, suitable to various fairness metrics, in the healthcare domain where electronic health records are used. Not only our approach explicitly mitigates bias as part of the optimization process, but an FL-based paradigm would also implicitly help with addressing data imbalance and increasing the data size, offering a practical solution for healthcare applications. We empirically demonstrate our method's superior performance on multiple experiments simulating large-scale real-world scenarios and compare it to several baselines. Our method has achieved promising fairness performance with the lowest impact on overall discrimination performance (accuracy). Our code is available at https://github.com/healthylaife/FairFedAvg.
Title: Discrimination through Image Selection by Job Advertisers on Facebook
Authors: Varun Nagaraj Rao; Aleksandra Korolova
Affiliations: Princeton University; Princeton University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Targeted advertising platforms are widely used by job advertisers to reach potential employees; thus issues of discrimination due to targeting that have surfaced have received widespread attention. Advertisers could misuse targeting tools to exclude people based on gender, race, location and other protected attributes from seeing their job ads. In response to legal actions, Facebook disabled the ability for explicit targeting based on many attributes for some ad categories, including employment. Although this is a step in the right direction, prior work has shown that discrimination can take place not just due to the explicit targeting tools of the platforms, but also due to the impact of the biased ad delivery algorithm. Thus, one must look at the potential for discrimination more broadly, and not merely through the lens of the explicit targeting tools.
Title: Group-Fair Classification with Strategic Agents
Authors: Andrew Estornell; Sanmay Das; Yang Liu; Yevgeniy Vorobeychik
Affiliations: Washington University in St. Louis; George Mason University; University of California, Santa Cruz; Washington University in St. Louis
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The use of algorithmic decision making systems in domains which impact the financial, social, and political well-being of people has created a demand for these to be “fair” under some accepted notion of equity. This demand has in turn inspired a large body of work focused on the development of fair learning algorithms which are then used in lieu of their conventional counterparts. Most analysis of such fair algorithms proceeds from the assumption that the people affected by the algorithmic decisions are represented as immutable feature vectors. However, strategic agents may possess both the ability and the incentive to manipulate this observed feature vector in order to attain a more favorable outcome. We explore the impact that strategic agent behavior can have on group-fair classification. We find that in many settings strategic behavior can lead to fairness reversal, with a conventional classifier exhibiting higher fairness than a classifier trained to satisfy group fairness. Further, we show that fairness reversal occurs as a result of a group-fair classifier becoming more selective, achieving fairness largely by excluding individuals from the advantaged group. In contrast, if group fairness is achieved by the classifier becoming more inclusive, fairness reversal does not occur.
Title: In the Name of Fairness: Assessing the Bias in Clinical Record De-Identification
Authors: Yuxin Xiao; Shulammite Lim; Tom Pollard; Marzyeh Ghassemi
Affiliations: Massachusetts Institute of Technology; Massachusetts Institute of Technology; Massachusetts Institute of Technology; Massachusetts Institute of Technology
Year: 2023
Domain: Health & Clinical AI

Abstract: Data sharing is crucial for open science and reproducible research, but the legal sharing of clinical data requires the removal of protected health information from electronic health records. This process, known as de-identification, is often achieved through the use of machine learning algorithms by many commercial and open-source systems. While these systems have shown compelling results on average, the variation in their performance across different demographic groups has not been thoroughly examined. In this work, we investigate the bias of de-identification systems on names in clinical notes via a large-scale empirical analysis. To achieve this, we create 16 name sets that vary along four demographic dimensions: gender, race, name popularity, and the decade of popularity. We insert these names into 100 manually curated clinical templates and evaluate the performance of nine public and private de-identification methods. Our findings reveal that there are statistically significant performance gaps along a majority of the demographic dimensions in most methods. We further illustrate that de-identification quality is affected by polysemy in names, gender context, and clinical note characteristics. To mitigate the identified gaps, we propose a simple and method-agnostic solution by fine-tuning de-identification methods with clinical context and diverse names. Overall, it is imperative to address the bias in existing methods immediately so that downstream stakeholders can build high-quality systems to serve all demographic parties fairly.