cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Pragmatic Fairness: Evaluating ML Fairness within the Constraints of Industry
Authors: Jessie J. Smith; Michael Madaio; Robin Burke; Casey Fiesler
Affiliations: University of Colorado Boulder; Google (United States); University of Colorado Boulder; University of Colorado Boulder
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning (ML) fairness evaluation in real-world, industry settings presents unique challenges due to business-driven constraints that influence decision-making processes.While prior research has proposed fairness frameworks and evaluation methodologies, these approaches often focus on idealized conditions and may lack consideration for the practical realities faced by industry practitioners.To understand these practical realities, we conducted a semi-structured interview study with 21 experts from academia and industry specializing in ML fairness.Through this study, we explore three constraints of ML fairness evaluation in industrybalancing competing interests, lacking power/access, and getting buy-in-and how these constraints lead to satisficing, seeking satisfactory rather than ideal outcomes.We define the path from these constraints to satisficing as pragmatic fairness.Using recommender systems as a case study, we explore how practitioners navigate these constraints and highlight actionable strategies to improve fairness evaluations within these business-minded boundaries.This paper provides practical insights to guide fairness evaluations in industry while also showcasing how the FAccT community can better align research goals with the operational realities of practitioners.
Title: Social Perception of Faces in a Vision-Language Model
Authors: Carina Ines Hausladen; Manuel Knott; Colin F. Camerer; Pietro Perona
Affiliations: California Institute of Technology; California Institute of Technology; California Institute of Technology; California Institute of Technology
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: We explore social perception of human faces in CLIP, a widely used open-source vision-language model. To this end, we compare the similarity in CLIP embeddings between different textual prompts and a set of face images. Our textual prompts are constructed from well-validated social psychology terms denoting social perception. The face images are synthetic and are systematically and independently varied along six dimensions: the legally protected attributes of age, gender, and race, as well as facial expression, lighting, and pose. Independently and systematically manipulating face attributes allows us to study the effect of each on social perception and avoids confounds that can occur in wild-collected data due to uncontrolled systematic correlations between attributes. Thus, our findings are experimental rather than observational. Our main findings are three. First, while CLIP is trained on the widest variety of images and texts, it is able to make fine-grained human-like social judgments on face images. Second, age, gender, and race do systematically impact CLIP's social perception of faces, suggesting an undesirable bias in CLIP vis-a-vis legally protected attributes. Most strikingly, we find a strong pattern of bias concerning the faces of Black women, where CLIP produces extreme values of social perception across different ages and facial expressions. Third, facial expression impacts social perception more than age and lighting as much as age. The last finding predicts that studies that do not control for unprotected visual attributes may reach the wrong conclusions on bias. Our novel method of investigation, which is founded on the social psychology literature and on the experiments involving the manipulation of individual attributes, yields sharper and more reliable observations than previous observational methods and may be applied to study biases in any vision-language model.
Title: Group Fair Rated Preference Aggregation: Ties Are (Mostly) All You Need
Authors: Kathleen Cachel; Elke A. Rundensteiner
Affiliations: Worcester Polytechnic Institute; Worcester Polytechnic Institute
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Rated preference aggregation is conventionally performed by averaging ratings from multiple evaluators to create a consensus ordering of candidates from highest to lowest average rating. Ideally, the consensus is fair, meaning critical opportunities are not withheld from marginalized groups of candidates, even if group biases may be present in the to-be-combined ratings. Prior work operationalizing fairness in preference aggregation is limited to settings where evaluators provide rankings of candidates. Yet, in practice, many evaluators assign ratings such as Likert scales or categories to each candidate. Ratings convey different information than rankings leading to distinct fairness issues during their aggregation. To address this gap, we define the open fair rated preference aggregation problem and introduce the corresponding Fate methodology. Fate offers the first group fairness metric specifically for rated preference data.
Title: Identities are not Interchangeable: The Problem of Overgeneralization in Fair Machine Learning
Authors: Angelina Wang
Affiliations: Cornell University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: A key value proposition of machine learning is generalizability: the same methods and model architecture should be able to work across different domains and different contexts. While powerful, this generalization can sometimes go too far, and miss the importance of the specifics. In this work, we look at how fair machine learning has often treated as interchangeable the identity axis along which discrimination occurs. In other words, racism is measured and mitigated the same way as sexism, as ableism, as ageism. Disciplines outside of computer science have pointed out both the similarities and differences between these different forms of oppression, and in this work we draw out the implications for fair machine learning. While certainly not all aspects of fair machine learning need to be tailored to the specific form of oppression, there is a pressing need for greater attention to such specificity than is currently evident. Ultimately, context specificity can deepen our understanding of how to build more fair systems, widen our scope to include currently overlooked harms, and, almost paradoxically, also help to narrow our scope and counter the fear of an infinite number of group-specific methods of analysis.
Title: Not Like Us, Hunty: Measuring Perceptions and Behavioral Effects of Minoritized Anthropomorphic Cues in LLMs
Authors: Jeffrey Basoah; Daniel Chechelnitsky; Tao Long; Katharina Reinecke; Chrysoula Zerva; Kaitlyn Zhou; Mark Díaz; Maarten Sap
Affiliations: University of Washington; Carnegie Mellon University; Columbia University; University of Washington; University of Lisbon; Stanford University; Google (United States); Carnegie Mellon University
Year: 2025
Domain: LLMs & NLP

Abstract: As large language models (LLMs) increasingly adapt and personalize to diverse sets of users, there is an increased risk of systems appropriating sociolects, i.e., language styles or dialects that are associated with specific minoritized lived experiences (e.g., African American English, Queer slang). In this work, we examine whether sociolect usage by an LLM agent affects user reliance on its outputs and user perception (satisfaction, frustration, trust, and social presence). We designed and conducted user studies where 498 African American English (AAE) speakers and 487 Queer slang speakers performed a set of question-answering tasks with LLM-based suggestions in either standard American English (SAE) or their self-identified sociolect. Our findings showed that sociolect usage by LLMs influenced both reliance and perceptions, though in some surprising ways. Results suggest that both AAE and Queer slang speakers relied more on the SAE agent, and had more positive perceptions of the SAE agent. Yet, only Queer slang speakers felt more social presence from the Queer slang agent over the SAE one, whereas only AAE speakers preferred and trusted the SAE agent over the AAE one. These findings emphasize the need to test for behavioral outcomes rather than simply assume that personalization would lead to a better and safer reliance outcome. They also highlight the nuanced dynamics of minoritized language in machine interactions, underscoring the need for LLMs to be carefully designed to respect cultural and linguistic boundaries while fostering genuine user engagement and trust.
Title: Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways
Authors: Angelina Wang; Xuechunzi Bai; Solon Barocas; Su Lin Blodgett
Affiliations: Cornell University; University of Chicago; Microsoft (United States); Microsoft (Canada)
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Despite a proliferation of research on the ways that machine learning models can propagate harmful stereotypes, very little of this work is grounded in the psychological experiences of people exposed to such stereotypes.We use a case study of gender stereotypes in image search to examine how people react to machine learning errors.First, we use surveys to show that not all machine learning errors reflect stereotypes nor are equally harmful.Then, in experimental studies we randomly expose participants to stereotypereinforcing, -violating, and -neutral machine learning errors.We find stereotype-reinforcing errors induce more experiential harm, while having minimal impact on participants' cognitive beliefs, attitudes, or behaviors.This experiential harm impacts participants who are women more than those who are men.However, certain stereotype-violating errors are more experientially harmful for men, potentially due to perceived threats to masculinity.We conclude by proposing a more nuanced perspective on the harms of machine learning errors-one that depends on who is experiencing what harm and why.
Title: In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review
Authors: Amelia Jiménez-Sánchez; Natalia-Rozalia Avlona; Sarah de Boer; Víctor M. Campello; Aasa Feragen; Enzo Ferrante; Melanie Ganz; Judy Wawira Gichoya; Camila González; Steff Groefsema; Alessa Hering; Ádám Hulmán; Leo Joskowicz; Dovile Juodelyte; Melih Kandemir; Thijs Kooi; Jorge del Pozo Lérida; Livie Yumeng Li; André G. C. Pacheco; Tim Rädsch; Mauricio Reyes; Théo Sourget; Bram van Ginneken; David Wen; Nina Weng; Jack Junchi Xu; Hubert Dariusz Zając; María A. Zuluaga; Veronika Cheplygina
Affiliations: IT University of Copenhagen; ; Radboud University Nijmegen; ; Danish Geotechnical Society; Institute of Astronomy and Space Physics; Rigshospitalet; Emory University; Stanford University; ; Radboud University Nijmegen; Aarhus University; ; IT University of Copenhagen; Odense Municipality; ; ; Aarhus University; ; Heidelberg University; University of Bern; IT University of Copenhagen; Radboud University Nijmegen; Oxford University Hospitals NHS Trust; Danish Geotechnical Society; Copenhagen University Hospital; ; EURECOM; IT University of Copenhagen
Year: 2025
Domain: Health & Clinical AI

Abstract: Datasets play a critical role in medical imaging research, yet issues such as label quality, shortcuts, and metadata are often overlooked. This lack of attention may harm the generalizability of algorithms and, consequently, negatively impact patient outcomes. While existing medical imaging literature reviews mostly focus on machine learning (ML) methods, with only a few focusing on datasets for specific applications, these reviews remain static-they are published once and not updated thereafter. This fails to account for emerging evidence, such as biases, shortcuts, and additional annotations that other researchers may contribute after the dataset is published. We refer to these newly discovered findings of datasets as research artifacts. To address this gap, we propose a living review that continuously tracks public datasets and their associated research artifacts across multiple medical imaging applications. Our approach includes a framework for the living review to monitor data documentation artifacts, and an SQL database to visualize the citation relationships between research artifact and dataset. Lastly, we discuss key considerations for creating medical imaging datasets, review best practices for data annotation, discuss the significance of shortcuts and demographic diversity, and emphasize the importance of managing datasets throughout their entire lifecycle. Our demo is publicly available at http://inthepicture.itu.dk/.
Title: Dangerous Criminals and Beautiful Prostitutes? Investigating Harmful Representations in Dutch Language Models
Authors: Zilin Lin; Gabriela Trogrlic; Claes H. de Vreese; Natali Helberger
Affiliations: Amsterdam University of the Arts; Amsterdam University of the Arts; Amsterdam University of the Arts; Amsterdam University of the Arts
Year: 2025
Domain: LLMs & NLP

Abstract: While language-based AI is becoming increasingly popular, ensuring that these systems are socially responsible is essential. Despite their growing impact, large language models (LLMs), the engines of many language-driven applications, remain largely in the black box. Concerns about LLMs reinforcing harmful representations are shared by academia, industries, and the public. In professional contexts, researchers rely on LLMs for computational tasks such as text classification and contextual prediction, during which the risk of perpetuating biases cannot be overlooked. In a broader society where LLM-powered tools are widely accessible, interacting with biased models can shape public perceptions and behaviors, potentially reinforcing problematic social issues over time. This study investigates harmful representations in LLMs, focusing on ethnicity and gender in the Dutch context. Through template-based sentence construction and model probing, we identified potentially harmful representations using both automated and manual content analysis at the lexical and sentence levels, combining quantitative measurements with qualitative insights. Our findings have important ethical, legal, and political implications, challenging the acceptability of such harmful representations and emphasizing the need for effective mitigation strategies. <br/><b>Warning</b>: This paper contains examples of language that some people may find offensive or upsetting
Title: Component-Based Fairness in Face Attribute Classification with Bayesian Network-informed Meta Learning
Authors: Yifan Liu; Ruichen Yao; Yaokun Liu; Ruohan Zong; Zelin Li; Yang Zhang; Dong Wang
Affiliations: University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The widespread integration of face recognition technologies into various applications such as access control and personalized advertising necessitates a critical emphasis on fairness. While previous efforts have focused on demographic fairness, the fairness of individual biological face components remains unexplored. In this paper, we focus on face component fairness, a fairness notion defined by biological face features. We propose Bayesian Network-informed Meta Reweighting (BNMR), which incorporates a Bayesian Network calibrator to guide an adaptive meta-learning-based sample reweighting process. Our results show that BNMR is able to consistently outperform recent face bias mitigation baselines and suggest a positive impact of face component fairness on the commonly considered demographic fairness.
Title: Towards Effective Discrimination Testing for Generative AI
Authors: Thomas P. Zollo; Nikita Rajaneesh; Richard S. Zemel; Talia B. Gillis; Emily Black
Affiliations: Columbia University; Columbia University; Columbia University; Columbia University; New York University
Year: 2025
Domain: LLMs & NLP

Abstract: Generative AI (GenAI) models present new challenges in regulating against discriminatory behavior. In this paper, we argue that GenAI fairness research still has not met these challenges; instead, a significant gap remains between existing bias assessment methods and regulatory goals. This leads to ineffective regulation that can allow deployment of reportedly fair, yet actually discriminatory, GenAI systems. Towards remedying this problem, we connect the legal and technical literature around GenAI bias evaluation and identify areas of misalignment. Through four case studies, we demonstrate how this misalignment between fairness testing techniques and regulatory goals can result in discriminatory outcomes in real-world deployments, especially in adaptive or complex environments. We offer practical recommendations for improving discrimination testing to better align with regulatory goals and enhance the reliability of fairness assessments in future deployments.
Title: Who Gets Heard? Calling Out the Hard-to-Reach Myth for Non-WEIRD Populations' Recruitment and Involvement in Research
Authors: Ankolika De; Shaheen Kanthawala; Jessica Maddox
Affiliations: Pennsylvania State University; University of Alabama; University of Alabama
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Despite the growing recognition of the need for inclusive technology design, research often struggles to engage non-WEIRD (Western, Educated, Industrialized, Rich, and Democratic) populations, labeling them as "hard to reach." This paper argues that rigid, Western-centric research methods contribute to this gap, particularly in non-WEIRD contexts.Drawing on two qualitative studies of WhatsApp users in India, we highlight the challenges in the research process when attempting to apply a non-Western perspective while recruiting participants in informal and fragmented communication environments.We emphasize the need to embrace and normalize flexible, context-specific recruitment strategies that move beyond Western norms of rigor.By adopting non-WEIRD methodologies, we aim to make computing research more inclusive and accessible, especially for populations in the Global South.We challenge the notion that these groups are inherently difficult to engage, arguing instead that the difficulty stems from the continued imposition of Western-centric approaches.
Title: Un-Straightening Generative AI: How Queer Artists Surface and Challenge Model Normativity
Authors: Jordan Taylor; Joel Mire; Franchesca Spektor; Alicia DeVrio; Maarten Sap; Haiyi Zhu; Sarah Fox
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2025
Domain: LLMs & NLP

Abstract: Queer people are often discussed as targets of bias, harm, or discrimination in generative AI research.However, the specific ways that queer people engage with generative AI, and thus possible uses that support queer people, have yet to be explored.We conducted a workshop study with 13 queer artists, during which we gave participants access to GPT-4 and DALL-E 3 and facilitated group sensemaking activities.Our participants struggled to use these models FAccT '25, June 23-26, 2025, Athens, Greece
Title: Liberatory Collections and Ethical AI: Reimagining AI Development from Black Community Archives and Datasets
Authors: Payton Croskey; Fabian Offert; Jennifer Jacobs; Kai M. Thaler
Affiliations: University of California, Santa Barbara; University of California, Santa Barbara; University of California, Santa Barbara; University of California, Santa Barbara
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Biases in artificial intelligence (AI) training datasets have led to models that perpetuate harmful racial stereotypes about Black communities while neglecting accurate and affirming representations of Black culture. We build on calls to create transformative technology that transcends harm mitigation, a space we categorize as liberatory technology. Within this space, we identify liberatory collections, community-led repositories that amplify Black voices, as a form of data collection that empowers and benefits communities historically harmed by traditional AI and archival practices. We survey fourteen liberatory collections, uncovering critical cultural perspectives and innovative approaches to obtaining, preserving, sharing, and valuing human information. Our findings emphasize the need for structural changes in AI training datasets and models, moving beyond representational adjustments toward strategies for creating consent-driven training models, funding community-based initiatives, and embedding Black communities multifaceted cultures and histories within AI systems.