cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices
Authors: Will Hawkins; Brent Mittelstadt
Affiliations: University of Oxford; University of Oxford
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The technical progression of artificial intelligence (AI) research has been built on breakthroughs in fields such as computer science, statistics, and mathematics. However, in the past decade AI researchers have increasingly looked to the social sciences, turning to human interactions to solve the challenges of model development. Paying crowdsourcing workers to generate or curate data, or ‘data enrichment’, has become indispensable for many areas of AI research, from natural language processing to reinforcement learning from human feedback (RLHF). Other fields that routinely interact with crowdsourcing workers, such as Psychology, have developed common governance requirements and norms to ensure research is undertaken ethically. This study explores how, and to what extent, comparable research ethics requirements and norms have developed for AI research and data enrichment. We focus on the approach taken by two leading conferences: ICLR and NeurIPS, and journal publisher Springer. In a longitudinal study of accepted papers, and via a comparison with Psychology and CHI papers, this work finds that leading AI venues have begun to establish protocols for human data collection, but these are are inconsistently followed by authors. Whilst Psychology papers engaging with crowdsourcing workers frequently disclose ethics reviews, payment data, demographic data and other information, similar disclosures are far less common in leading AI venues despite similar guidance. The work concludes with hypotheses to explain these gaps in research ethics practices and considerations for its implications.
Title: Envisioning Equitable Speech Technologies for Black Older Adults
Authors: Robin Brewer; Christina Harrington; Courtney Heldreth
Affiliations: University of Michigan; Carnegie Mellon University; Google (United States)
Year: 2023
Domain: LLMs & NLP

Abstract: There is increasing concern that how researchers currently define and measure fairness is inadequate. Recent calls push to move beyond traditional concepts of fairness and consider related constructs through qualitative and community-based approaches, particularly for underrepresented communities most at-risk for AI harm. One in context, previous research has identified that voice technologies are unfair due to racial and age disparities. This paper uses voice technologies as a case study to unpack how Black older adults value and envision fair and equitable AI systems. We conducted design workshops and interviews with 16 Black older adults, exploring how participants envisioned voice technologies that better understand cultural context and mitigate cultural dissonance. Our findings identify tensions between what it means to have fair, inclusive, and representative voice technologies. This research raises questions about how and whether researchers can model cultural representation with large language models.
Title: Understanding accountability in algorithmic supply chains
Authors: Jennifer Cobbe; Michael Veale; Jatinder Singh
Affiliations: University of Cambridge; Faculty (United Kingdom); University of Cambridge
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Academic and policy proposals on algorithmic accountability often seek to understand algorithmic systems in their socio-technical context, recognising that they are produced by ‘many hands’. Increasingly, however, algorithmic systems are also produced, deployed, and used within a supply chain comprising multiple actors tied together by flows of data between them. In such cases, it is the working together of an algorithmic supply chain of different actors who contribute to the production, deployment, use, and functionality that drives systems and produces particular outcomes. We argue that algorithmic accountability discussions must consider supply chains and the difficult implications they raise for the governance and accountability of algorithmic systems. In doing so, we explore algorithmic supply chains, locating them in their broader technical and political economic context and identifying some key features that should be understood in future work on algorithmic governance and accountability (particularly regarding general purpose AI services). To highlight ways forward and areas warranting attention, we further discuss some implications raised by supply chains: challenges for allocating accountability stemming from distributed responsibility for systems between actors, limited visibility due to the accountability horizon, service models of use and liability, and cross-border supply chains and regulatory arbitrage.
Title: Rethinking Transparency as a Communicative Constellation
Authors: Florian Eyert; Paola Lopez
Affiliations: Weizenbaum Institute; University of Vienna
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In this paper we make the case for an expanded understanding of transparency. Within the now extensive FAccT literature, transparency has largely been understood in terms of explainability. While this approach has proven helpful in many contexts, it falls short of addressing some of the more fundamental issues in the development and application of machine learning, such as the epistemic limitations of predictions and the political nature of the selection of fairness criteria. In order to render machine learning systems more democratic, we argue, a broader understanding of transparency is needed. We therefore propose to view transparency as a communicative constellation that is a precondition for meaningful democratic deliberation. We discuss four perspective expansions implied by this approach and present a case study illustrating the interplay of heterogeneous actors involved in producing this constellation. Drawing from our conceptualization of transparency, we sketch implications for actor groups in different sectors of society.
Title: Domain Adaptive Decision Trees: Implications for Accuracy and Fairness
Authors: José M. Álvarez; Kristen M. Scott; Bettina Berendt; Salvatore Ruggieri
Affiliations: University of Pisa; KU Leuven; Weizenbaum Institute; University of Pisa
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In uses of pre-trained machine learning models, it is a known issue that the target population in which the model is being deployed may not have been reflected in the source population with which the model was trained. This can result in a biased model when deployed, leading to a reduction in model performance. One risk is that, as the population changes, certain demographic groups will be under-served or otherwise disadvantaged by the model, even as they become more represented in the target population. The field of domain adaptation proposes techniques for a situation where label data for the target population does not exist, but some information about the target distribution does exist. In this paper we contribute to the domain adaptation literature by introducing domain-adaptive decision trees (DADT). We focus on decision trees given their growing popularity due to their interpretability and performance relative to other more complex models. With DADT we aim to improve the accuracy of models trained in a source domain (or training data) that differs from the target domain (or test data). We propose an in-processing step that adjusts the information gain split criterion with outside information corresponding to the distribution of the target population. We demonstrate DADT on real data and find that it improves accuracy over a standard decision tree when testing in a shifted target population. We also study the change in fairness under demographic parity and equal opportunity. Results show an improvement in fairness with the use of DADT.
Title: Bias on Demand: A Modelling Framework That Generates Synthetic Data With Bias
Authors: Joachim Baumann; Alessandro Castelnovo; Riccardo Crupi; Nicole Inverardi; Daniele Regoli
Affiliations: University of Zurich
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Nowadays, Machine Learning (ML) systems are widely used in various businesses and are increasingly being adopted to make decisions that can significantly impact people’s lives. However, these decision-making systems rely on data-driven learning, which poses a risk of propagating the bias embedded in the data. Despite various attempts by the algorithmic fairness community to outline different types of bias in data and algorithms, there is still a limited understanding of how these biases relate to the fairness of ML-based decision-making systems. In addition, efforts to mitigate bias and unfairness are often agnostic to the specific type(s) of bias present in the data. This paper explores the nature of fundamental types of bias, discussing their relationship to moral and technical frameworks. To prevent harmful consequences, it is essential to comprehend how and where bias is introduced throughout the entire modelling pipeline and possibly how to mitigate it. Our primary contribution is a framework for generating synthetic datasets with different forms of biases. We use our proposed synthetic data generator to perform experiments on different scenarios to showcase the interconnection between biases and their effect on performance and fairness evaluations. Furthermore, we provide initial insights into mitigating specific types of bias through post-processing techniques. The implementation of the synthetic data generator and experiments can be found at https://github.com/rcrupiISP/BiasOnDemand.
Title: Co-Design Perspectives on Algorithm Transparency Reporting: Guidelines and Prototypes
Authors: Michal Luria
Affiliations: Center for Democracy & Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Recommendation algorithms by and large determine what people see on social media. Users know little about how these algorithms work or what information they use to make their recommendations. But what exactly should platforms share with users about recommendation algorithms that would be meaningful to them? Research has looked into frameworks for explainability of algorithms as well as design features across social media platforms that can contribute to their transparency and accountability. We build on these prior efforts to explore what a recommendation algorithm transparency report may include and how it should present information to users. Through a human-centered co-design research process we result in: (1) A set of guidelines for recommendation algorithm transparency reports; (2) initial suggestions, in the form of prototypes, for more engaging and interactive forms of transparency; (3) an evaluation of these prototypes’ strengths and weaknesses, and areas of exploration for future work.
Title: Using Supervised Learning to Estimate Inequality in the Size and Persistence of Income Shocks
Authors: David Bruns-Smith; Avi Feller; Emi Nakamura
Affiliations: University of California, Berkeley; Berkeley College
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Household responses to income shocks are important drivers of financial fragility, the evolution of wealth inequality, and the effectiveness of fiscal and monetary policy. Traditional approaches to measuring the size and persistence of income shocks are based on restrictive econometric models that impose strong homogeneity across households and over time. In this paper, we propose a more flexible, machine learning framework for estimating income shocks that allows for variation across all observable features and time horizons. First, we propose non-parametric estimands for shocks and shock persistence. We then show how to estimate these quantities by using off-the-shelf supervised learning tools to approximate the conditional expectation of future income given present information. We solve this income prediction problem in a large Icelandic administrative dataset, and then use the estimated shocks to document several features of labor income risk in Iceland that are not captured by standard economic income models.
Title: A Causal Perspective on Label Bias
Authors: Vishwali Mhasawade; Alexander D’Amour; Stephen Pfohl
Affiliations: New York University; Google (United States); Google (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Predictive models developed with machine learning techniques are commonly used to inform decision making and resource allocation in high-stakes contexts, such as healthcare and public health. One means through which this practice may propagate equity-related harms is when the data used for model development or evaluation exhibits label bias. In such cases, the target of prediction is a proxy label of a construct of interest that may be difficult or impossible to measure, while the relationship between the proxy and the construct of interest differs systematically across subgroups. Label bias can be especially challenging to identify and mitigate in practice because consequential fairness violations are masked when the model is evaluated with respect to the proxy label. In this work, we aim to develop further formal understanding of label bias to inform the development of approaches for the identification and mitigation of it. To do so, we present desiderata for unbiased and biased proxy labels, introduce candidate causal graphical criteria for label bias, and consider the extent to which proxy labels can be used to reason about fairness with respect to a true construct of interest. We validate our findings with a simulation study and experiments with synthetic health insurance data used in the context of a care management system.
Title: Fairness in Online Ad Delivery
Authors: Joachim Baumann; Piotr Sapieżyński; Christoph Heitz; Anikó Hannák
Affiliations: University of Zurich; Northeastern University; ZHAW Zurich University of Applied Sciences; University of Zurich
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Advertising funds a number of services that play a major role in our everyday online experiences, from social networking, to maps, search, and news. As the power and reach of advertising platforms grow, so do the concerns about the potential for discrimination associated with targeted advertising. However, despite our ever-improving ability to measure and describe instances of unfair distribution of high-stakes ads—such as employment, housing, or credit—we lack the tools to model and predict the extent to which alternative systems could address such problems. In this paper, we simulate an ad distribution system to model the effects that enforcing popularly proposed fairness approaches would have on the utility of the advertising platforms and their users. We show that in many realistic scenarios, achieving statistical parity would come at a much higher utility cost to platforms than enforcing predictive parity or equality of opportunity. Additionally, we identify a tradeoff between different notions of fairness, i.e., enforcing one criterion leads to worse outcomes with respect to other criteria. We further describe how pursuing fairness in situations where one group of users is more expensive to advertise to is likely to result in “leveling down” effects, i.e., not benefiting any group of users. We show that these negative effects can be prevented by ensuring that it is the platforms that carry the cost of fairness rather than passing it on to their users or advertisers. Overall, our findings contribute to ongoing discussions on fair ad delivery. We show that fairness is not satisfied by default, that limiting targeting options is not sufficient to address potential discrimination and bias in online ad delivery, and that choices made by regulators and platforms may backfire if potential side-effects are not properly considered.
Title: Constructing Capabilities: The Politics of Testing Infrastructures for Generative AI
Authors: Gabriel Grill
Affiliations: University of Michigan
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: The advertised and perceived capabilities of generative AI products like ChatGPT have recently stimulated considerable investments and discourse surrounding their potential to aid and replace work. The prominence of these systems, and their promise to be general-purpose, has resulted in an avalanche of tests to discover and certify their capabilities. This new testing regime is concerned with creating ever-more tasks for generative AI products instead of testing a model for one specialized task. Beyond efforts to understand products’ capabilities, the construction of tasks and corresponding tests are also performative enactments meant to convince others and thus to gain attention, scientific legitimacy, and investment. The current market concentration of a few big AI companies points to a concerning conflict of interest: those with a vested interest in the success of the technology also have control over globalized testing infrastructures and thereby the exclusive means to create extensive knowledge claims about these systems. In this paper, I theorize capabilities as contested constructions and situated accomplishments shaped by power imbalances. I further unpack the globalized testing infrastructures involved in the construction and stabilization of generative AI products’ capabilities. Furthermore, I discuss how the testing of these AI models and products is externalized, extracting value from the unpaid or under-paid labor of researcher and developer communities, content creators, subcontractors, and users. Lastly, I discuss a reflexive and critical approach to testing that challenges depoliticization and seeks to produce lasting critiques that serve more emancipatory goals.
Title: Crediting Humans: A Systematic Assessment of Influencing Factors for Human-in-the-Loop Figurations in Consumer Credit Lending Decisions
Authors: Theresa Züger; Philipp Mahlow; Daniel Pothmann; Katharina Mosene; Fabian Burmeister; Matthias C. Kettemann; Wolfgang Schulz
Affiliations: Alexander von Humboldt Institute for Internet and Society; Alexander von Humboldt Institute for Internet and Society; Alexander von Humboldt Institute for Internet and Society; Alexander von Humboldt Institute for Internet and Society; Universität Hamburg; Alexander von Humboldt Institute for Internet and Society; Alexander von Humboldt Institute for Internet and Society
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The concept of ‘human-in-the-loop’ (Hilo) has gained prominence as a regulatory mechanism for ensuring human control in automated systems, particularly in the context of automated decision-making (ADM) mechanisms or AI-supported systems. However, despite its increasing use in regulatory discourse, there is a lack of empirical understanding regarding the real-world conditions and influencing factors that affect decision-making processes in such hybrid systems. This paper aims to address this gap by focusing on the use of automation, and the inclusion of humans, in consumer-facing credit lending decisions, a key industry use case. By employing an interdisciplinary approach that combines legal perspectives, social sciences, and architectural modeling—a methodology rooted in computer science—this research offers a first systematic analysis of the factors that influence meaningful human control in
Title: Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods
Authors: Mahdi Dhaini; Ege Erdoğan; Nils Feldhus; Gjergji Kasneci
Affiliations: Technical University of Munich; Technical University of Munich; German Research Centre for Artificial Intelligence; Technical University of Munich
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Achieving fair use of AI systems is a multi-faceted challenge. Intersectionality, rooted in Black Feminist movements, is increasingly used to address the interconnected nature of discrimination such as racism, ableism, and sexism. Yet in AI research, intersectionality is often reduced to a narrow technical lens, focused on algorithmic bias between subgroups defined by protected attributes and addressed through fairness metrics. This algorithmic frame sidelines key aspects of intersectionality, such as power relations, social justice, and structural inequality. Still, AI experts play a central role in development and deployment, and therefore should act to limit unjust outcomes. This study offers actionable guidance for AI experts, grounded in a broader intersectional perspective. Through a thematic analysis of AI fairness papers on key aspects of intersectionality, evaluated through community engagement, we identify five themes with concrete recommendations: 1) insisting on collaboration in interdisciplinary teams, 2) embedding reflection and recognizing positionality, 3) approaching communities and facilitating co-ownership, 4) engaging with power dynamics and social context, and 5) assessing the framing and nuance of data and metrics. Participating experts noted barriers such as tech-optimism and fear of insufficient knowledge. Still, they valued the recommendations for communicating the importance of intersectionality and initiating more just AI practices. We call on AI experts to meet this challenge through interdisciplinary collaboration with diverse communities.
Title: Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Making
Authors: Julia Barnett; Kimon Kieslich; Natali Helberger; Nicholas Diakopoulos
Affiliations: Northwestern University; University of Amsterdam; University of Amsterdam; Northwestern University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The potential for negative impacts of AI has rapidly become more pervasive around the world, and this has intensified a need for responsible AI governance. While many regulatory bodies endorse risk-based approaches and a multitude of risk mitigation practices are proposed by companies and academic scholars, these approaches are commonly expert-centered and thus lack the inclusion of a significant group of stakeholders. Ensuring that AI policies align with democratic expectations requires methods that prioritize the voices and needs of those impacted. In this work we develop a participative and forward-looking approach to inform policy-makers and academics that grounds the needs of lay stakeholders at the forefront and enriches the development of risk mitigation strategies. Our approach (1) maps potential mitigation and prevention strategies of negative AI impacts that assign responsibility to various stakeholders, (2) explores the importance and prioritization thereof in the eyes of laypeople, and (3) presents these insights in policy fact sheets, i.e., a digestible format for informing policy processes. We emphasize that this approach is not targeted towards replacing policy-makers; rather our aim is to present an informative method that enriches mitigation strategies and enables a more participatory approach to policy development.