cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Recommend Me? Designing Fairness Metrics with Providers
Authors: Jessie J. Smith; Aishwarya Satwani; Robin Burke; Casey Fiesler
Affiliations: University of Colorado Boulder; University of Colorado Boulder; University of Colorado Boulder; University of Colorado Boulder
Year: 2024
Domain: Recommender Systems

Abstract: Fairness metrics have become a useful tool to measure how fair or unfair a machine learning system may be for its stakeholders. In the context of recommender systems, previous research has explored how various stakeholders experience algorithmic fairness or unfairness, but it is also important to capture these experiences in the design of fairness metrics. Therefore, we conducted four focus groups with providers (those whose items, content, or profiles are being recommended) of two different domains: content creators and dating app users. We explored how our participants experience unfairness on their associated platforms, and worked with them to co-design fairness goals, definitions, and metrics that might capture these experiences. This work represents an important step towards designing fairness metrics with the stakeholders who will be impacted by their operationalizations. We analyze the efficacy and challenges of enacting these metrics in practice and explore how future work might benefit from this methodology.
Title: Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronouns
Authors: Goya van Boven; Yupei Du; Dong Nguyen
Affiliations: Utrecht University; Utrecht University; Utrecht University
Year: 2024
Domain: LLMs & NLP

Abstract: Gender-neutral pronouns are increasingly being introduced across Western languages. Recent evaluations have however demonstrated that English NLP systems are unable to correctly process gender-neutral pronouns, with the risk of erasing and misgendering non-binary individuals. This paper examines a Dutch coreference resolution system’s performance on gender-neutral pronouns, specifically hen and die. In Dutch, these pronouns were only introduced in 2016, compared to the longstanding existence of singular they in English. We additionally compare two debiasing techniques for coreference resolution systems in non-binary contexts: Counterfactual Data Augmentation (CDA) and delexicalisation. Moreover, because pronoun performance can be hard to interpret from a general evaluation metric like lea, we introduce an innovative evaluation metric, the pronoun score, which directly represents the portion of correctly processed pronouns. Our results reveal diminished performance on gender-neutral pronouns compared to gendered counterparts. Nevertheless, although delexicalisation fails to yield improvements, CDA substantially reduces the performance gap between gendered and gender-neutral pronouns. We further show that CDA remains effective in low-resource settings, in which a limited set of debiasing documents is used. This efficacy extends to previously unseen neopronouns, which are currently infrequently used but may gain popularity in the future, underscoring the viability of effective debiasing with minimal resources and low computational costs.
Title: Learning Fair Ranking Policies via Differentiable Optimization of Ordered Weighted Averages
Authors: My H. Dinh; James Kotary; Ferdinando Fioretto
Affiliations: University of Virginia; University of Virginia; University of Virginia
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Learning to Rank (LTR) is one of the most widely used machine learning applications. It is a key component in platforms with profound societal impacts, including job search, healthcare information retrieval, and social media content feeds. Conventional LTR models have been shown to produce biases results, stimulating a discourse on how to address the disparities introduced by ranking systems that solely prioritize user relevance. However, while several models of fair learning to rank have been proposed, they suffer from deficiencies either in accuracy or efficiency, thus limiting their applicability to real-world ranking platforms. This paper shows how efficiently-solvable fair ranking models, based on the optimization of Ordered Weighted Average (OWA) functions, can be integrated into the training loop of an LTR model to achieve favorable balances between fairness, user utility, and runtime efficiency. In particular, this paper is the first to show how to backpropagate through constrained optimizations of OWA objectives, enabling their use in integrated prediction and decision models.
Title: Algorithmic Arbitrariness in Content Moderation
Authors: Juan Felipe Gomez; Caio C. Vieira Machado; Lucas Monteiro Paes; Flávio P. Calmon
Affiliations: Harvard University; Harvard University Press; Harvard University; Harvard University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning (ML) is widely used to moderate online content. Despite its scalability relative to human moderation, the use of ML introduces unique challenges to content moderation. One such challenge is predictive multiplicity: multiple competing models for content classification may perform equally well on average, yet assign conflicting predictions to the same content. This multiplicity can result from seemingly innocuous choices made during training, which do not meaningfully change the accuracy of the ML model, but can nevertheless change what the model gets wrong. We experimentally demonstrate how content moderation tools can arbitrarily classify samples as “toxic,” leading to arbitrary restrictions on speech. We use the principles set by the International Covenant on Civil and Political Rights (ICCPR), namely freedom of expression, non-discrimination, and procedural justice to interpret the effects of these findings in terms of Human Rights. We analyze (i) the extent of predictive multiplicity among popular state-of-the-art LLMs used for detecting “toxic” content; (ii) the disparate impact of this arbitrariness across social groups; and (iii) the magnitude of model multiplicity on content that is unanimously recognized as toxic by human annotators. Our findings indicate that the up-scaled algorithmic moderation risks legitimizing an “algorithmic leviathan”, where an algorithm disproportionately manages human rights. To mitigate such risks, our study underscores the need to identify and increase the transparency of arbitrariness in content moderation applications. Our findings have implications to content moderation and intermediary liability laws being discussed and passed in many countries, such as the Digital Services Act in the European Union, the Online Safety Act in the United Kingdom, and the recent TSE resolutions in Brazil.
Title: System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes
Authors: Arpit Agarwal; Nicolas Usunier; Alessandro Lazaric; Maximilian Nickel
Affiliations: FAIR at Meta; FAIR at Meta; FAIR at Meta; FAIR at Meta
Year: 2024
Domain: Recommender Systems

Abstract: Recommender systems are an important part of the modern human experience whose influence ranges from the food we eat to the news we read. Yet, there is still debate as to what extent online recommendation platforms are aligned with the goals of their users. A core issue fueling this debate is the challenge of inferring a user’s utility based on their engagement signals such as likes, shares, watch time etc., which are often the primary metric used by platforms to optimize content. This is because users’ utility-driven decision-processes (which we refer to as System-2), e.g., reading news that are accurate and relevant for them, are often confounded by their impulsive or unconscious decision-processes (which we refer to as System-1), e.g., spend time on click-bait news articles. As a result, it is difficult to infer whether an observed engagement is utility-driven or impulse-driven. In this paper we explore a new approach to recommender systems where we infer user’s utility based on their return probability to the platform rather than engagement signals. This approach is based on the intuition that users tend to return to a platform in the long run if it creates utility for them, while pure engagement-driven interactions, i.e., interactions that do not add meaningful utility, may affect user return in the short term but will not have a lasting effect. For this purpose, we propose a generative model in which past content interactions impact the arrival rates of users based on a self-exciting Hawkes process. These arrival rates to the platform are a combination of both System-1 and System-2 decision processes. The System-2 arrival intensity depends on the utility drawn from past content interactions and has a long lasting effect on return probability. In contrast, System-1 arrival intensity depends on the instantaneous gratification or moreishness and tends to vanish rapidly in time. We show analytically that given samples from this model it is provably possible to disentangle the System-1 and System-2 decision-processes and thus infer user’s utility, thereby allowing us to optimize content based on it. We conduct experiments on synthetic data to demonstrate the effectiveness of our approach over engagement optimization.
Title: Emotion AI in Job Interviews: Injustice, Emotional Labor, Identity, and Privacy
Authors: Alexis Shore Ingber; Nazanin Andalibi
Affiliations: University of Michigan; University of Michigan
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The use of emotion artificial intelligence to evaluate job-seekers during the interview process raises concerns about injustice, particularly among minoritized individuals.However, we know little about how job-seekers experience emotion AI interviews compared to non-emotion AI or human resources (HR) interviews.We address this gap through a 3(Evaluation Source: Emotion AI, AI, HR) x 2(Outcome Favorability: Recommended, Not Recommended) experiment.Participants (n = 456) underwent a mock interview assuming real-time evaluation.We find those who partook an interview evaluated by emotion AI perceived significantly lower levels of justice than that evaluated by HR, with this relationship partially dependent on outcome favorability.Additionally, identity attributes (e.g., gender, race, disability) and privacy concerns significantly influenced the information disclosed and emotional labor exerted during the interview.We discuss practical implications for emotion AI use in job interviews as well as broader methodological implications for capturing real-time engagement with AI systems. CCS Concepts Security and privacy
Title: Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participation
Authors: Ben Hutchinson; Celeste Rodríguez Louro; Glenys Collard; Ned Cooper
Affiliations: ; The University of Western Australia; The University of Western Australia; Australian National University
Year: 2025
Domain: LLMs & NLP

Abstract: In Australia, post-contact language varieties, including creoles and local varieties of international languages, emerged as a result of forced contact between Indigenous communities and English speakers. These contact varieties are widely used, yet are poorly supported by language technologies. This gap presents barriers to participation in civil and economic society for Indigenous communities using these varieties, and reproduces minoritisation of contemporary Indigenous sociolinguistic identities. This paper concerns three questions regarding this context. First, can speech technologies support speakers of Australian Aboriginal English, a local indigenised variety of English? Second, what risks are inherent in such a project? Third, what technology development practices are appropriate for this context, and how can researchers integrate meaningful community participation in order to mitigate risks? We argue that opportunities do exist—as well as risks—and demonstrate this through a case study exploring design practices in a project aiming to improve speech technologies for Australian Aboriginal English. We discuss how we integrated culturally appropriate and participatory processes throughout the project. We call for increased support for languages used by Indigenous communities, including contact varieties, which provide practical economic and socio-cultural benefits, provided that participatory and culturally safe practices are enacted.
Title: FairTranslate: an English-French Dataset for Gender Bias Evaluation in Machine Translation by Overcoming Gender Binarity
Authors: Fanny Jourdan; Y Chevalier; Cécile Favre
Affiliations: IRT M2P; Université Lumière Lyon 2; Université Claude Bernard Lyon 1
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) are increasingly leveraged for translation tasks but often fall short when translating inclusive language -such as texts containing the singular 'they' pronoun or otherwise reflecting fair linguistic protocols.Because these challenges span both computational and societal domains, it is imperative to critically evaluate how well LLMs handle inclusive translation with a well-founded framework.This paper presents FairTranslate, a novel, fully human-annotated dataset designed to evaluate non-binary gender biases in machine translation systems from English to French.FairTranslate includes 2418 English-French sentence pairs related to occupations, annotated with rich metadata such as the stereotypical alignment of the occupation, grammatical gender indicator ambiguity, and the ground-truth gender label (male, female, or inclusive).We evaluate four leading LLMs (Gemma2-2B, Mistral-7B, Llama3.1-8B,Llama3.3-70B) on this dataset under different prompting procedures.Our results reveal substantial biases in gender representation across LLMs, highlighting persistent challenges in achieving equitable outcomes in machine translation.These findings underscore the need for focused strategies and interventions aimed at ensuring fair and inclusive language usage in LLM-based translation systems.We make the FairTranslate dataset publicly available on Hugging Face, and disclose the code for all experiments on GitHub.
Title: Formalising Anti-Discrimination Law in Automated Decision Systems
Authors: Holli Sargeant; Måns Magnusson
Affiliations: University of Cambridge; Uppsala University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic discrimination is a critical concern as machine learning models are used in high-stakes decision-making in legally protected contexts.Although substantial research on algorithmic bias and discrimination has led to the development of fairness metrics, several critical legal issues remain unaddressed in practice.The paper addresses three key shortcomings in prevailing ML fairness paradigms:(1) the narrow reliance on prediction or outcome disparity as evidence for discrimination, (2) the lack of nuanced evaluation of estimation error and assumptions that the true causal structure and data-generating process are known, and (3) the overwhelming dominance of US-based analyses which has inadvertently fostered some misconceptions regarding lawful modelling practices in other jurisdictions.To address these gaps, we introduce a novel decisiontheoretic framework grounded in anti-discrimination law of the United Kingdom, which has global influence and aligns closely with European and Commonwealth legal systems.We propose the "conditional estimation parity" metric, which accounts for estimation error and the underlying data-generating process, aligning with UK legal standards.We apply our formalism to a real-world algorithmic discrimination case, demonstrating how technical and legal reasoning can be aligned to detect and mitigate unlawful discrimination.Our contributions offer actionable, legally-grounded guidance for ML practitioners, policymakers, and legal scholars seeking to develop non-discriminatory automated decision systems that are legally robust.
Title: Classifying Hate: Legal and Ethical Evaluations of ML-Assisted Hate Crime Classification and Estimation in Sweden
Authors: Holli Sargeant; Hannes Waldetoft; Måns Magnusson
Affiliations: University of Cambridge; Uppsala University; Uppsala University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Hate crimes, driven by biases against specific demographic groups, harm not only individuals but undermine the security, trust, and cohesion of entire communities.Accurately identifying such crimes remains a significant challenge due to under-reporting, limited training, and the complexity of determining bias motivations.In this paper, we analyze the results of a text classification model developed to improve the precision of hate crime statistics and identification in Sweden.Empirical results indicate the model outperforms traditional manual police classification of hate crimes, achieving higher precision across various crime types and regions.We further disaggregate performance to pinpoint persistent challenges and highlight categories where both human and machine decision-makers struggle.While the model focuses on statistical estimation rather than direct case-level decision-making, we discuss the broader implications of algorithmic transparency, accountability, and explainability.Ultimately, this research illustrates how transformer-based neural networks can responsibly bolster the detection and understanding of hate crimes, informing policies to better protect vulnerable communities.
Title: "It's not a representation of me": Examining Accent Bias and Digital Exclusion in Synthetic AI Voice Services
Authors: Michel Sala; Sufi Kaur; Sarah Elizabeth Gillespie; J. M. Gleason; Christo Wilson; Avijit Ghosh
Affiliations: Northeastern University; Northeastern University; Northeastern University; Northeastern University; Northeastern University; Boston University
Year: 2025
Domain: LLMs & NLP

Abstract: Recent advances in artificial intelligence (AI) speech generation and voice cloning technologies have produced naturalistic speech and accurate voice replication, yet their influence on sociotechnical systems across diverse accents and linguistic traits is not fully understood. This study evaluates two synthetic AI voice services (Speechify and ElevenLabs) through a mixed methods approach using surveys and interviews to assess technical performance and uncover how users lived experiences influence their perceptions of accent variations in these speech technologies. Our findings reveal technical performance disparities across five regional, English-language accents and demonstrate how current speech generation technologies may inadvertently reinforce linguistic privilege and accent-based discrimination, potentially creating new forms of digital exclusion. Overall, our study highlights the need for inclusive design and regulation by providing actionable insights for developers, policymakers, and organizations to ensure equitable and socially responsible AI speech technologies.
Title: The World Wide Recipe: A Community-Centred Framework for Fine-Grained Data Collection and Regional Bias Operationalisation
Authors: Jabez Magomere; Shu Ishida; Tejumade Afonja; Aya Salama; Daniel Kochin; Foutse Yuehgoh; Imane Hamzaoui; Raesetje Sefala; Aisha Alaagib; Samantha Dalal; Beatrice Marchegiani; Elizaveta Semenova; Lauren Crais; Siobhan Mackenzie Hall
Affiliations: University of Oxford; Autodesk (United Kingdom); Helmholtz Center for Information Security; ; University of Oxford; ; École Nationale Supérieure d'Informatique; McGill University; ; University of Colorado Boulder; University of Oxford; Imperial College London; University of Oxford; University of Oxford
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: We introduce the World Wide recipe, which sets forth a framework for culturally aware and participatory data collection, and the resultant regionally diverse World Wide Dishes evaluation dataset. We also analyse bias operationalisation to highlight how current systems underperform across several dimensions: inaccuracy, misrepresentation, and cultural insensitivity, with evidence from qualitative community-based observations and quantitative automated tools. We find that these T2I models generally do not produce quality outputs of dishes specific to various regions. This is true even for the US, which is typically considered more well-resourced in training data. The models demonstrate the propensity to produce inaccurate and culturally misrepresentative, flattening, and insensitive outputs. These representational biases have the potential to further reinforce stereotypes and disproportionately contribute to erasure based on region.
Title: African Data Ethics: A Discursive Framework for Black Decolonial AI
Authors: Teanna Barrett; Chinasa T. Okolo; B. Biira; Eman Sherif; Amy X. Zhang; Leilani Battle
Affiliations: University of Washington; Brookings Institution; University of Washington; University of Washington; University of Washington; University of Washington
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The shift towards pluralism in global data ethics acknowledges the importance of including perspectives from the Global Majority to develop responsible data science practices that mitigate systemic harms in the current data science ecosystem. Sub-Saharan African (SSA) practitioners, in particular, are disseminating progressive data ethics principles and best practices for identifying and navigating anti-blackness and data colonialism. To center SSA voices in the global data ethics discourse, we present a framework for African data ethics informed by the thematic analysis of an interdisciplinary corpus of 47 documents. Our framework features six major principles: Challenge Power Asymmetries, Assert Data Self-Determination, Invest in Local Data Institutions and Infrastructures, Utilize Communalist Practices, Center Communities on the Margins, and Uphold Common Good.
Title: Position is Power: System Prompts as a Mechanism of Bias in Large Language Models
Authors: Anna Neumann; Elisabeth Kirsten; Muhammad Bilal Zafar; Jatinder Singh
Affiliations: University of Duisburg-Essen; Ruhr University Bochum; Ruhr University Bochum; University of Cambridge
Year: 2025
Domain: LLMs & NLP

Abstract: System prompts in Large Language Models (LLMs) are predefined directives that guide model behaviour, taking precedence over user inputs in text processing and generation. LLM deployers increasingly use them to ensure consistent responses across contexts. While model providers set a foundation of system prompts, deployers and third-party developers can append additional prompts without visibility into others' additions, while this layered implementation remains entirely hidden from end-users. As system prompts become more complex, they can directly or indirectly introduce unaccounted for side effects. This lack of transparency raises fundamental questions about how the position of information in different directives shapes model outputs. As such, this work examines how the placement of information affects model behaviour. To this end, we compare how models process demographic information in system versus user prompts across six commercially available LLMs and 50 demographic groups. Our analysis reveals significant biases, manifesting in differences in user representation and decision-making scenarios. Since these variations stem from inaccessible and opaque system-level configurations, they risk representational, allocative and potential other biases and downstream harms beyond the user's ability to detect or correct. Our findings draw attention to these critical issues, which have the potential to perpetuate harms if left unexamined. Further, we argue that system prompt analysis must be incorporated into AI auditing processes, particularly as customisable system prompts become increasingly prevalent in commercial AI deployments.