cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: One Model Many Scores: Using Multiverse Analysis to Prevent Fairness Hacking and Evaluate the Influence of Model Design Decisions
Authors: Jan Simson; Florian Pfisterer; Christoph Kern
Affiliations: Munich Center for Machine Learning; Ludwig-Maximilians-Universität München; Munich Center for Machine Learning
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: A vast number of systems across the world use algorithmic decision making (ADM) to (partially) automate decisions that have previously been made by humans. The downstream effects of ADM systems critically depend on the decisions made during a systems’ design, implementation, and evaluation, as biases in data can be mitigated or reinforced along the modeling pipeline. Many of these decisions are made implicitly, without knowing exactly how they will influence the final system. To study this issue, we draw on insights from the field of psychology and introduce the method of multiverse analysis for algorithmic fairness. In our proposed method, we turn implicit decisions during design and evaluation into explicit ones and demonstrate their fairness implications. By combining decisions, we create a grid of all possible “universes” of decision combinations. For each of these universes, we compute metrics of fairness and performance. Using the resulting dataset, one can investigate the variability and robustness of fairness scores and see how and which decisions impact fairness. We demonstrate how multiverse analyses can be used to better understand fairness implications of design and evaluation decisions using an exemplary case study of predicting public health care coverage for vulnerable populations. Our results highlight how decisions regarding the evaluation of a system can lead to vastly different fairness metrics for the same model. This is problematic, as a nefarious actor could optimise or “hack” a fairness metric to portray a discriminating model as fair merely by changing how it is evaluated. We illustrate how a multiverse analysis can help to address this issue.
Title: Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans
Authors: Messi Ho Jun Lee; Jacob Montgomery; Calvin K. Lai
Affiliations: Washington University in St. Louis; Washington University in St. Louis; Washington University in St. Louis
Year: 2024
Domain: LLMs & NLP

Abstract: Large language models (LLMs) are becoming pervasive in everyday life, yet their propensity to reproduce biases inherited from training data remains a pressing concern. Prior investigations into bias in LLMs have focused on the association of social groups with stereotypical attributes. However, this is only one form of human bias such systems may reproduce. We investigate a new form of bias in LLMs that resembles a social psychological phenomenon where socially subordinate groups are perceived as more homogeneous than socially dominant groups. We had ChatGPT, a state-of-the-art LLM, generate texts about intersectional group identities and compared those texts on measures of homogeneity. We consistently found that ChatGPT portrayed African, Asian, and Hispanic Americans as more homogeneous than White Americans, indicating that the model described racial minority groups with a narrower range of human experience. ChatGPT also portrayed women as more homogeneous than men, but these differences were small. Finally, we found that the effect of gender differed across racial/ethnic groups such that the effect of gender was consistent within African and Hispanic Americans but not within Asian and White Americans. We argue that the tendency of LLMs to describe groups as less diverse risks perpetuating stereotypes and discriminatory behavior.
Title: The Impact of Differential Feature Under-reporting on Algorithmic Fairness
Authors: Nil-Jana Akpinar; Zachary C. Lipton; Alexandra Chouldechova
Affiliations: Amazon (United States); Carnegie Mellon University; Carnegie Mellon University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Predictive risk models in the public sector are commonly developed using administrative data that is more complete for subpopulations that more greatly rely on public services. In the United States, for instance, information on health care utilization is routinely available to government agencies for individuals supported by Medicaid and Medicare, but not for the privately insured. Critiques of public sector algorithms have identified such “differential feature under-reporting” as a driver of disparities in algorithmic decision-making. Yet this form of data bias remains understudied from a technical viewpoint. While prior work has examined the fairness impacts of additive feature noise and features that are clearly marked as missing, little is known about the setting of data missingness absent indicators (i.e. differential feature under-reporting). In this work, we study an analytically tractable model of differential feature under-reporting to characterizethe impact of under-report on algorithmic fairness. We demonstrate how standard missing data methods typically fail to mitigate bias in this setting, and propose a new set of augmented loss and imputation methods. Our results show that, in real world data settings, under-reporting typically exacerbates disparities. The proposed solution methods show some success in mitigating disparities attributable to feature under-reporting.
Title: NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs
Authors: Maria Antoniak; Aakanksha Naik; Carla S. Alvarado; Lucy Lu Wang; Irene Y. Chen
Affiliations: Allen Institute; Allen Institute; Association of American Medical Colleges; University of Washington; University of California, San Francisco
Year: 2024
Domain: Health & Clinical AI

Abstract: Ethical frameworks for the use of natural language processing (NLP) are urgently needed to shape how large language models (LLMs) and similar tools are used for healthcare applications. Healthcare faces existing challenges including the balance of power in clinician-patient relationships, systemic health disparities, historical injustices, and economic constraints. Drawing directly from the voices of those most affected, and focusing on a case study of a specific healthcare setting, we propose a set of guiding principles for the use of NLP in maternal healthcare. We led an interactive session centered on an LLM-based chatbot demonstration during a full-day workshop with 39 participants, and additionally surveyed 30 healthcare workers and 30 birthing people about their values, needs, and perceptions of NLP tools in the context of maternal health. We conducted quantitative and qualitative analyses of the survey results and interactive discussions to consolidate our findings into a set of guiding principles. We propose nine principles for ethical use of NLP for maternal healthcare, grouped into three themes: (i) recognizing contextual significance (ii) holistic measurements, and (iii) who/what is valued. For each principle, we describe its underlying rationale and provide practical advice. This set of principles can provide a methodological pattern for other researchers and serve as a resource to practitioners working on maternal health and other healthcare fields to emphasize the importance of technical nuance, historical context, and inclusive design when developing NLP technologies for clinical use.
Title: Equalised Odds is not Equal Individual Odds: Post-processing for Group and Individual Fairness
Authors: Edward Small; Kacper Sokol; Daniel Manning; Flora D. Salim; Jeffrey Chan
Affiliations: RMIT University; ETH Zurich; RMIT University; UNSW Sydney; RMIT University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Group fairness is achieved by equalising prediction distributions between protected sub-populations; individual fairness requires treating similar individuals alike. These two objectives, however, are incompatible when a scoring model is calibrated through discontinuous probability functions, where individuals can be randomly assigned an outcome determined by a fixed probability. This procedure may provide two similar individuals from the same protected group with classification odds that are disparately different – a clear violation of individual fairness. Assigning unique odds to each protected sub-population may also prevent members of one sub-population from ever receiving the chances of a positive outcome available to individuals from another sub-population, which we argue is another type of unfairness called individual odds. We reconcile all this by constructing continuous probability functions between group thresholds that are constrained by their Lipschitz constant. Our solution preserves the model’s predictive power, individual fairness and robustness while ensuring group fairness.
Title: Careless Whisper: Speech-to-Text Hallucination Harms
Authors: Allison Koenecke; Anna Seo Gyeong Choi; K. Mei; Hilke Schellmann; Mona Sloane
Affiliations: Cornell University; Cornell University; University of Washington; New York University; University of Virginia
Year: 2024
Domain: LLMs & NLP

Abstract: Speech-to-text services aim to transcribe input audio as accurately as possible. They increasingly play a role in everyday life, for example in personal voice assistants or in customer-company interactions. We evaluate Open AI’s Whisper, a state-of-the-art automated speech recognition service outperforming industry competitors, as of 2023. While many of Whisper’s transcriptions were highly accurate, we find that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences which did not exist in any form in the underlying audio. We thematically analyze the Whisper-hallucinated content, finding that 38% of hallucinations include explicit harms such as perpetuating violence, making up inaccurate associations, or implying false authority. We then study why hallucinations occur by observing the disparities in hallucination rates between speakers with aphasia (who have a lowered ability to express themselves using speech and voice) and a control group. We find that hallucinations disproportionately occur for individuals who speak with longer shares of non-vocal durations—a common symptom of aphasia. We call on industry practitioners to ameliorate these language-model-based hallucinations in Whisper, and to raise awareness of potential biases amplified by hallucinations in downstream applications of speech-to-text models.
Title: Unlawful Proxy Discrimination: A Framework for Challenging Inherently Discriminatory Algorithms
Authors: Hilde Weerts; Reuben Binns; Aislinn Kelly-Lyth; Jeremias Adams-Prassl
Affiliations: Eindhoven University of Technology; University of Oxford; Blackstone Chambers; University of Oxford
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Emerging scholarship suggests that the EU legal concept of direct discrimination - where a person is given different treatment on grounds of a protected characteristic - may apply to various algorithmic decision-making contexts. This has important implications: unlike indirect discrimination, there is generally no objective justification stage in the direct discrimination framework, which means that the deployment of directly discriminatory algorithms will usually be unlawful per se. In this paper, we focus on the most likely candidate for direct discrimination in the algorithmic context, termed inherent direct discrimination, where a proxy is inextricably linked to a protected characteristic. We draw on computer science literature to suggest that, in the algorithmic context, treatment on the grounds of needs to be understood in terms of two steps: proxy capacity and proxy use. Only where both elements can be made out can direct discrimination be said to be on grounds of a protected characteristic. We analyse the legal conditions of our proposed proxy capacity and proxy use tests. Based on this analysis, we discuss technical approaches and metrics that could be developed or applied to identify inherent direct discrimination in algorithmic decision-making.
Title: Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention
Authors: Cedric Deslandes Whitney; J Norman
Affiliations: Berkeley College; Berkeley College
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning systems require representations of the real world for training and testing - they require data, and lots of it. Collecting data at scale has logistical and ethical challenges, and synthetic data promises a solution to these challenges. Instead of needing to collect photos of real people’s faces to train a facial recognition system, a model creator could create and use photo-realistic, synthetic faces. The comparative ease of generating this synthetic data rather than relying on collecting data has made it a common practice. We present two key risks of using synthetic data in model development. First, we detail the high risk of false confidence when using synthetic data to increase dataset diversity and representation. We base this in the examination of a real world use-case of synthetic data, where synthetic datasets were generated for an evaluation of facial recognition technology. Second, we examine how using synthetic data risks circumventing consent for data usage. We illustrate this by considering the importance of consent to the U.S. Federal Trade Commission’s regulation of data collection and affected models. Finally, we discuss how these two risks exemplify how synthetic data complicates existing governance and ethical practice; by decoupling data from those it impacts, synthetic data is prone to consolidating power away those most impacted by algorithmically-mediated harm.
Title: Fairness without Sensitive Attributes via Knowledge Sharing
Authors: Hongliang Ni; Lei Han; Tong Chen; Shazia Sadiq; Gianluca Demartini
Affiliations: The University of Queensland; The University of Queensland; The University of Queensland; The University of Queensland; The University of Queensland
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: While model fairness improvement has been explored previously, existing methods invariably rely on adjusting explicit sensitive attribute values in order to improve model fairness in downstream tasks. However, we observe a trend in which sensitive demographic information becomes inaccessible as public concerns around data privacy grow. In this paper, we propose a confidence-based hierarchical classifier structure called “Reckoner” for reliable fair model learning under the assumption of missing sensitive attributes. We first present results showing that if the dataset contains biased labels or other hidden biases, classifiers significantly increase the bias gap across different demographic groups in the subset with higher prediction confidence. Inspired by these findings, we devised a dual-model system in which a version of the model initialised with a high-confidence data subset learns from a version of the model initialised with a low-confidence data subset, enabling it to avoid biased predictions. Our experimental results show that Reckoner consistently outperforms state-of-the-art baselines in COMPAS dataset and New Adult dataset, considering both accuracy and fairness metrics.
Title: BaBE: Enhancing Fairness via Estimation of Explaining Variables
Authors: Rūta Binkytė; Daniele Gorla; Catuscia Palamidessi
Affiliations: Helmholtz Center for Information Security; ; École Polytechnique
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: We consider the problem of unfair discrimination between two groups and propose a pre-processing method to achieve fairness. Corrective methods like statistical parity usually lead to bad accuracy and do not really achieve fairness in situations where there is a correlation between the sensitive attribute S and the legitimate attribute E (explanatory variable) that should determine the decision. To overcome these drawbacks, other notions of fairness have been proposed, in particular, conditional statistical parity and equal opportunity. However, E is often not directly observable in the data. We may observe some other variable Z representing E, but the problem is that Z may also be affected by S, hence Z itself can be biased. To deal with this problem, we propose BaBE (Bayesian Bias Elimination), an approach based on a combination of Bayes inference and the Expectation-Maximization method, to estimate the most likely value of E for a given Z for each group. The decision can then be based directly on the estimated E. We show, by experiments on synthetic and real data sets, that our approach provides a good level of fairness as well as high accuracy.
Title: Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology
Authors: Rishav Hada; Safiya Husain; Varun Gumma; Harshita Diddee; Aditya Yadavalli; Agrima Seth; Nidhi Kulkarni; Ujwal Gadiraju; Aditya Vashistha; Vivek Seshadri; Kalika Bali
Affiliations: Microsoft Research (India); ; Microsoft Research (India); Carnegie Mellon University; ; University of Michigan; ; Delft University of Technology; Cornell University; Microsoft Research (India); Microsoft Research (India)
Year: 2024
Domain: LLMs & NLP

Abstract: Existing research in measuring and mitigating gender bias predominantly centers on English, overlooking the intricate challenges posed by non-English languages and the Global South. This paper presents the first comprehensive study delving into the nuanced landscape of gender bias in Hindi, the third most spoken language globally. Our study employs diverse mining techniques, computational models, field studies and sheds light on the limitations of current methodologies. Given the challenges faced with mining gender biased statements in Hindi using existing methods, we conducted field studies to bootstrap the collection of such sentences. Through field studies involving rural and low-income community women, we uncover diverse perceptions of gender bias, underscoring the necessity for context-specific approaches. This paper advocates for a community-centric research design, amplifying voices often marginalized in previous studies. Our findings not only contribute to the understanding of gender bias in Hindi but also establish a foundation for further exploration of Indic languages. By exploring the intricacies of this understudied context, we call for thoughtful engagement with gender bias, promoting inclusivity and equity in linguistic and cultural contexts beyond the Global North.
Title: Balancing Act: Evaluating People's Perceptions of Fair Ranking Metrics
Authors: Mallak Alkhathlan; Kathleen Cachel; Hilson Shrestha; Lane Harrison; Elke A. Rundensteiner
Affiliations: Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic decision-making using rankings— prevalent in areas from hiring and bail to university admissions— raises concerns of potential bias. In this paper, we explore the alignment between people’s perceptions of fairness and two popular fairness metrics designed for rankings. In a crowdsourced experiment with 480 participants, people rated the perceived fairness of a hypothetical scholarship distribution scenario. Results suggest a strong inclination towards relying on explicit score values. There is also evidence of people’s preference for one fairness metric, NDKL, over the other metric, ARP. Qualitative results paint a more complex picture: some participants endorse meritocratic award schemes and express concerns about fairness metrics being used to modify rankings; while other participants acknowledge socio-economic factors in score-based rankings as justification for adjusting rankings. In summary, we find that operationalizing algorithmic fairness in practice is a balancing act between mitigating harms towards marginalized groups and societal conventions of leveraging traditional performance scores such as grades in decision-making contexts.
Title: From the Fair Distribution of Predictions to the Fair Distribution of Social Goods: Evaluating the Impact of Fair Machine Learning on Long-Term Unemployment
Authors: Sebastian Zezulka; Konstantin Genin
Affiliations: University of Tübingen; University of Tübingen
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Deploying an algorithmically informed policy is a significant intervention in society. Prominent methods for algorithmic fairness focus on the distribution of predictions at the time of training, rather than the distribution of social goods that arises after deploying the algorithm in a specific social context. However, requiring a ‘fair’ distribution of predictions may undermine efforts at establishing a fair distribution of social goods. First, we argue that addressing this problem requires a notion of prospective fairness that anticipates the change in the distribution of social goods after deployment. Second, we provide formal conditions under which this change is identified from pre-deployment data. That requires accounting for different kinds of performative effects. Here, we focus on the way predictions change policy decisions and, consequently, the causally downstream distribution of social goods. Throughout, we are guided by an application from public administration: the use of algorithms to predict who among the recently unemployed will remain unemployed in the long term and to target them with labor market programs. Third, using administrative data from the Swiss public employment service, we simulate how such algorithmically informed policies would affect gender inequalities in long-term unemployment. When risk predictions are required to be ‘fair’ according to statistical parity and equality of opportunity, targeting decisions are less effective, undermining efforts to both lower overall levels of long-term unemployment and to close the gender gap in long-term unemployment.
Title: The Impact of iBuying is About More Than Just Racial Disparities: Evidence from Mecklenburg County, NC
Authors: Isaac Slaughter; Eva Maxfield Brown; Nicholas Weber
Affiliations: University of Washington; University of Washington; University of Washington
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Instant buyers (iBuyers)—companies that buy and sell homes based on automated valuation models (AVMs)—now hold more than 5% market share in some USA cities. In this work, we investigate the fairness of iBuyers by constructing a dataset that links racial demographics from voter records with detailed property information on over 50,000 real estate transactions. Using Bayesian hierarchical modeling we find that: 1. iBuyers Decrease the Racial Sales Price Gap Between Black and White Home Sellers. Controlling for over 50 property features we find that iBuyers reduce the racial price gap that otherwise exists between homes sold by Black and White homeowners. This is not, however, a result of equity achieved through proprietary AVMs, but rather a result of both Black and White homeowners being similarly disadvantaged by iBuyers’ low purchase prices; and, 2. iBuyers Increase Property Conversion Rates from Individual to Institutional Ownership. We trace iBuyers’ purchases as well as their follow-on sales of homes in Mecklenburg County. In doing so, we show that iBuyers increase the rate at which properties are converted from being individually owned to institutionally owned. The eventual purchasers of iBuyer homes include national and international rental companies that have been tied to high eviction rates and poor property management. As with sale prices, we find that rather than reapportioning this social harm more equitably, iBuyers are simply increasing the rate at which homes bought from White homeowners are converted to institutional ownership. Ultimately, our analysis suggests that iBuyers are Equalizing Housing Outcomes by Extending Real Estate Harms Typically Isolated to Black Homeowners to White homeowners as Well.