cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Black-Box Access is Insufficient for Rigorous AI Audits
Authors: Stephen Casper; Carson Ezell; Charlotte Siegmann; Noam Kolt; Taylor Lynn Curtis; Ben Bucknall; Andreas Haupt; Kevin Wei; Jérémy Scheurer; Marius Hobbhahn; Lee Sharkey; Satyapriya Krishna; Marvin Von Hagen; Silas Alberti; Alan Chan; Qinyi Sun; Michael Gerovitch; David Bau; Max Tegmark; David Krueger; Dylan Hadfield-Menell
Affiliations: Massachusetts Institute of Technology; Harvard University Press; Massachusetts Institute of Technology; University of Toronto; Massachusetts Institute of Technology; Centre for the Governance of AI; Massachusetts Institute of Technology; Harvard University Press; Harvard University Press; Massachusetts Institute of Technology; Stanford University; Mila - Quebec Artificial Intelligence Institute; Massachusetts Institute of Technology; Massachusetts Institute of Technology; Northeastern University; Massachusetts Institute of Technology; University of Cambridge; Massachusetts Institute of Technology
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system’s inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to training and deployment information (e.g., methodology, code, documentation, data, deployment details, findings from internal evaluations) allows auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.
Title: A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Authors: Anna Gausen; Bhaskar Mitra; Siân Lindley
Affiliations: Imperial College London; Microsoft (Canada); Microsoft Research (United Kingdom)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Organisations generate vast amounts of information, which has resulted in a long-term research effort into knowledge access systems for enterprise settings. Recent developments in artificial intelligence, in relation to large language models, are poised to have significant impact on knowledge access. This has the potential to shape the workplace and knowledge in new and unanticipated ways. Many risks can arise from the deployment of these types of AI systems, due to interactions between the technical system and organisational power dynamics.
Title: Machine learning data practices through a data curation lens: An evaluation framework
Authors: Eshta Bhardwaj; Harshit Gujral; Siyi Wu; Ciara Zogheib; Tegan Maharaj; Christoph Becker
Affiliations: University of Toronto; University of Toronto; University of Toronto; University of Toronto; University of Toronto; University of Toronto
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Studies of dataset development in machine learning call for greater attention to the data practices that make model development possible and shape its outcomes. Many argue that the adoption of theory and practices from archives and data curation fields can support greater fairness, accountability, transparency, and more ethical machine learning. In response, this paper examines data practices in machine learning dataset development through the lens of data curation. We evaluate data practices in machine learning as data curation practices. To do so, we develop a framework for evaluating machine learning datasets using data curation concepts and principles through a rubric. Through a mixed-methods analysis of evaluation results for 25 ML datasets, we study the feasibility of data curation principles to be adopted for machine learning data work in practice and explore how data curation is currently performed. We find that researchers in machine learning, which often emphasizes model development, struggle to apply standard data curation principles. Our findings illustrate difficulties at the intersection of these fields, such as evaluating dimensions that have shared terms in both fields but non-shared meanings, a high degree of interpretative flexibility in adapting concepts without prescriptive restrictions, obstacles in limiting the depth of data curation expertise needed to apply the rubric, and challenges in scoping the extent of documentation dataset creators are responsible for. We propose ways to address these challenges and develop an overall framework for evaluation that outlines how data curation concepts and methods can inform machine learning data practices.
Title: Structural Interventions and the Dynamics of Inequality
Authors: Aurora Zhang; Anette Hosoi
Affiliations: Data & Society Research Institute; Massachusetts Institute of Technology
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Recent conversations in the algorithmic fairness literature have raised several concerns with standard conceptions of fairness. First, constraining predictive algorithms to satisfy fairness benchmarks may sometimes lead to non-optimal outcomes for disadvantaged groups. Second, technical interventions are often ineffective by themselves, especially when divorced from an understanding of structural processes that generate social inequality. Inspired by both these critiques, we construct a common decision-making model, using mortgage loans as a running example. We show that under some conditions, any choice of decision threshold will inevitably perpetuate existing disparities in financial stability unless one deviates from the Pareto optimal policy. This confirms the intuition that technical interventions, such as fairness constraints, often do not sufficiently address persistent underlying inequities. Then, we model the effects of three different types of interventions: (1) policy changes in the algorithm’s decision threshold, and external changes to parameters that govern the downstream effects of late payment for (2) the whole population or (3) disadvantaged subgroups. We show how different interventions are recommended depending on the difficulty of enacting structural change upon external parameters and depending on the policymaker’s preferences for equity or efficiency. Counterintuitively, we demonstrate that preferences for efficiency over equity may sometimes lead to recommendations for interventions that target the under-resourced group alone. Finally, we simulate the effects of interventions on a dataset that combines HMDA and Fannie Mae loan data. This research highlights the ways that structural inequality can be perpetuated by seemingly unbiased decision mechanisms, and it shows that in many situations, technical solutions must be paired with external, context-aware interventions to enact social change.
Title: Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Authors: Arianna Manzini; Geoff Keeling; Nahema Marchal; Kevin R. McKee; Verena Rieser; Iason Gabriel
Affiliations: Google DeepMind (United Kingdom); Google (United Kingdom); Google DeepMind (United Kingdom); Google DeepMind (United Kingdom); Google DeepMind (United Kingdom); Google DeepMind (United Kingdom)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: As AI assistants become increasingly sophisticated and deeply integrated into our lives, questions of trust rise to the forefront. In this paper, we build on philosophical studies of trust to investigate when user trust in AI assistants is justified. By moving beyond a focus on the technical artefact in isolation, we consider the broader societal system in which AI assistants are developed and deployed. We conceptualise user trust in AI assistants as encompassing two main targets, namely AI assistants and their developers. We argue that – as AI assistants become more human like and exhibit increased agency – discerning when user trust is justified requires consideration not only of competence, on the part of AI assistants and their developers, but also alignment between the competing interests, values or incentives of AI assistants, developers and users. To help users understand if and when their trust in the competence and alignment of AI assistants and developers is justified, we propose a sociotechnical approach that requires evidence to be collected at three levels: AI assistant design, organisational practices and third-party governance. Taken together, these measures can help harness the transformative potential of AI assistants while also ensuring their operation is ethical and value aligned.
Title: Algorithmic Misjudgement in Google Search Results: Evidence from Auditing the US Online Electoral Information Environment
Authors: Brooke Perreault; J. Lee; Ropafadzo Shava; Eni Mustafaraj
Affiliations: Wellesley College; Wellesley College; Wellesley College; Wellesley College
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Google Search is an important way that people seek information about politics [8], and Google states that it is “committed to providing timely and authoritative information on Google Search to help voters understand, navigate, and participate in democratic processes.”1 This paper studies the extent to which government-maintained web domains are represented in the online electoral information environment, as captured through 3.45 Google Search result pages collected during the 2022 US midterm elections for 786 locations across the United States. Focusing on state, county, and local government domains that provide locality-specific information, we study not only the extent to which these sources appear in organic search results, but also the extent to which these sources are correctly targeted to their respective constituents. We label misalignment between the geographic area that non-federal domains serve and the locations for which they appear in search results as algorithmic mistargeting, a subtype of algorithmic misjudgement in which the search algorithm targets locality-specific information to users in different (incorrect) locations. In the context of the 2022 US midterm elections, we find that 71% of all occurrences of state, county, and local government sources were mistargeted, with some domains appearing disproportionately often among organic results despite providing locality-specific information that may not be relevant to all voters. However, we also find that mistargeting often occurs in low ranks. We conclude by considering the potential consequences of extensive mistargeting of non-federal government sources and argue that ensuring the correct targeting of these sources to their respective constituents is a critical part of Google’s role in facilitating access to authoritative and locally-relevant electoral information.
Title: Racial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Represent
Authors: Jennifer Mickel
Affiliations: The University of Texas at Austin
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Racial diversity has become increasingly discussed within the AI and algorithmic fairness literature, yet little attention is focused on justifying the choices of racial categories and understanding how people are racialized into these chosen racial categories. Even less attention is given to how racial categories shift and how the racialization process changes depending on the context of a dataset or model. An unclear understanding of who comprises the racial categories chosen and how people are racialized into these categories can lead to varying interpretations of these categories. These varying interpretations can lead to harm when the understanding of racial categories and the racialization process is misaligned from the actual racialization process and racial categories used. Harm can also arise if the racialization process and racial categories used are irrelevant or do not exist in the context they are applied.
Title: Misgendered During Moderation: How Transgender Bodies Make Visible Cisnormative Content Moderation Policies and Enforcement in a Meta Oversight Board Case
Authors: Samuel Mayworm; Kendra Albert; Oliver L. Haimson
Affiliations: University of Michigan; Harvard University Press; University of Michigan
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Transgender and nonbinary social media users experience disproportionate content removals on social media platforms, even when content does not violate platforms’ guidelines. In 2022, the Oversight Board, which oversees Meta platforms’ content moderation decisions, invited public feedback on Instagram’s removal of two trans users’ posts featuring their bare chests, introducing a unique opportunity to hear trans users’ feedback on how nudity and sexual activity policies impacted them. We conducted a qualitative analysis of 83 comments made public during the Oversight Board’s public comment process. Commenters criticized Meta’s nudity policies as enforcing a cisnormative view of gender while making it unclear how images of trans users’ bodies are moderated, enabling the disproportionate removal of trans content and limiting trans users’ ability to use Meta’s platforms. Yet there was significant divergence among commenters about how to address cisnormative moderation. Some commenters suggested that Meta clarify nudity guidelines, while others suggested that Meta overhaul them entirely, removing gendered distinctions or fundamentally reconfiguring the platform’s relationship to sexual content. We then discuss how the Oversight Board’s public comment process demonstrates the value of incorporating trans people’s feedback while developing policies related to gender and nudity, while arguing that Meta must go beyond only revising policy language by reevaluating how cisnormative values are encoded in all aspects of its content moderation systems.
Title: A preprocessing Shapley value-based approach to detect relevant and disparity prone features in machine learning
Authors: Guilherme Dean Pelegrina; Miguel Couceiro; Leonardo Tomazeli Duarte
Affiliations: Universidade Estadual de Campinas (UNICAMP); Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento; Universidade Estadual de Campinas (UNICAMP)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Decision support systems became ubiquitous in every aspect of human lives. Their reliance on increasingly complex and opaque machine learning models raises transparency and fairness concerns with respect to unprivileged groups of people. This motivated several efforts to estimate importance of features towards the models’ performance and to detect unfair/disparate decisions. The latter is often dealt with by means of fairness metrics that rely on performance metrics with respect to predefined features that are considered protected (salient features such as age, gender, ethnicity, etc.) and/or sensitive (such as education, /occupation, banking information). However, such an approach is subjective (as fairness metrics depend on the choice features), there may be other features that lead to unfair (disparate) decisions and that may ask for suitable interpretations.
Title: Understanding Disparities in Post Hoc Machine Learning Explanation
Authors: Vishwali Mhasawade; Salman Rahman; Zoé Haskell-Craig; Rumi Chunara
Affiliations: New York University; New York University; New York University; New York University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Previous work has highlighted that existing post-hoc explanation methods exhibit disparities in explanation fidelity (across “race” and “gender” as sensitive attributes), and while a large body of work focuses on mitigating these issues at the explanation metric level, the role of the data generating process and black box model in relation to explanation disparities remains largely unexplored. Accordingly, through both simulations as well as experiments on a real-world dataset, we specifically assess challenges to explanation disparities that originate from properties of the data: limited sample size, covariate shift, concept shift, omitted variable bias, and challenges based on model properties: inclusion of the sensitive attribute and appropriate functional form. Through controlled simulation analyses, our study demonstrates that increased covariate shift, concept shift, and omission of covariates increase explanation disparities, with the effect pronounced higher for neural network models that are better able to capture the underlying functional form in comparison to linear models. We also observe consistent findings regarding the effect of concept shift and omitted variable bias on explanation disparities in the Adult income dataset. Overall, results indicate that disparities in model explanations can also depend on data and model properties. Based on this systematic investigation, we provide recommendations for the design of explanation methods that mitigate undesirable disparities.
Title: Fairness Beyond the Algorithmic Frame: Actionable Recommendations for an Intersectional Approach
Authors: Steven Vethman; Quirine T. S. Smit; Nina M. van Liebergen; Cor J. Veenman
Affiliations: Institut d'Etudes Politiques de Paris; Netherlands Organisation for Applied Scientific Research; Netherlands Organisation for Applied Scientific Research; Leiden University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Algorithm fairness is an established line of research in the machine learning domain with substantial work while the equivalent in the recommender system domain is relatively new. In this article, we consider rating-based recommender systems which model the recommendation process as a prediction problem. We consider different types of biases that can occur in this setting, discuss various fairness definitions, and also propose a novel bias mitigation strategy to address potential unfairness in a rating-based recommender system. Based on an analysis of fairness metrics used in machine learning and a discussion of their applicability in the recommender system domain, we map the proposed metrics from the two domains and identify commonly used concepts and definitions of fairness. Finally, to address unfairness and potential bias against certain groups in a recommender system, we develop a bias mitigation algorithm and conduct case studies on one synthetic and one empirical dataset to show its effectiveness. Our results show that unfairness can be significantly lowered through our approach and that bias mitigation is a fruitful area of research for recommender systems.
Title: Discrimination Induced by Algorithmic Recourse Objectives
Authors: Nicholas Perello; Cyrus Cousins; Yair Zick; Przemyslaw A. Grabowicz
Affiliations: University of Massachusetts Amherst; Duke University; University of Massachusetts Amherst; University College Dublin
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: We study fairness in algorithmic recourse, which provides users who receive undesirable outcomes from decision-making systems with explanations on how to achieve their desired results.To produce explanations that emulate real-world settings and preferences, the literature proposes various constraints, e.g., sparsity, on the objective of recourse methods.We argue that these constrained objectives can induce disparate recourse costs between protected groups, especially when groups' recourse objectives differ from the objectives of the methods providing them recourse.We formalize discrimination in algorithmic recourse with respect to legal notions and the cost of recourse.Namely, we introduce a novel notion of induced discrimination that captures the excess cost of recourse induced by having misaligned objectives between groups and methods.Using this definition, we show that recourse objectives can induce discriminatory recourse, independent from the bias of the model being explained.To inhibit discrimination, we extend prior work on welfare-based group-fair supervised learning to measure group welfare under multiple recourse objectives.We then propose a recourse method that maximizes group welfare and empirically show its robust performance over multiple recourse objectives.
Title: Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits
Authors: Marissa Gerchick; Ro Encarnación; Cole Tanigawa-Lau; Lena Armstrong; Ana Bedoya Gutiérrez; Danaë Metaxa
Affiliations: American Civil Liberties Union; University of Pennsylvania; Stanford University; Harvard University Press; American Civil Liberties Union; University of Pennsylvania
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: In this work, we "audit the audits," analyzing the documents produced pursuant to one of the United States' first enacted laws regulating the use of artificial intelligence in employment: New York City's Local Law 144.This law requires employers and employment agencies using certain types of automated tools to publish "bias audits" with statistics about how different sex and racial groups fare in the hiring process when the tools are used.We collect and conduct a comprehensive analysis of all Local Law 144 bias audits (N=116) made publicly available to our knowledge from the law taking effect in July 2023 until early November 2024, and describe the extensive challenges we faced in identifying, archiving, extracting information from, and ultimately analyzing these bias audits.We identify several ways that bias audits produced in accordance with Local Law 144 are incomplete evaluations of algorithmic bias, despite news coverage and characterizations by employers and vendors suggesting otherwise.We show that Local Law 144 bias audits are significantly hampered by several issues, including missing demographic data, opaque data aggregation, problematic uses of "test data, " and reliance on metrics that do not represent how automated hiring tools are used in practice.We analyze the reported results in Local Law 144 bias audits alongside the four-fifths rule often used as a measure for assessing adverse impact in employment contexts.Most audits do not report results that would not suggest violations of the four-fifths rule.Crucially, however, we show that these tools could often be in violation of the four-fifths rule when considering potential impacts of missing demographic data.We offer ten practical recommendations to strengthen future legislative efforts that mandate algorithm auditing in hiring and other areas, and contribute an open dataset and codebase for extracting and combining bias audit results to support future auditing efforts.