cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Popularity bias reduction in multi-criteria recommender systems: the RelPref and PriRelPref approaches
Authors: Nilüfer Ballı; Tuğba Türkoğlu Kaya
Affiliations: Ardahan University; Ardahan University
Year: 2026
Domain: Recommender Systems

Abstract: Popularity bias represents a critical challenge in multi-criteria recommender systems, as collective trends may dominate individual preferences during personalization, thereby complicating the accurate modeling of users’ multi-dimensional preference structures. To address this issue, existing approaches for popularity bias mitigation are reviewed, and two novel methods, Relative Preference Model (RelPref) and Criterion-Aware Relative Preference Model (PriRelPref), are proposed. These methods model multi-criteria user--item interactions in a more precise and balanced manner by explicitly incorporating users’ criterion priorities. RelPref dynamically evaluates user ratings across multiple criteria, whereas PriRelPref further integrates user-specific criterion prioritization to more effectively control the influence of popularity. By aligning users’ most important criteria with the criteria in which items perform strongly, the proposed approaches generate recommendation lists that better reflect individual preferences while maintaining balance across criteria. Experimental results indicate that the proposed methods improve the overall quality of recommendations by enhancing personalization and supporting more balanced item exposure.
Title: Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems
Authors: Jing Nathan Yan; Emma Harvey; Junxiong Wang; Jeffrey M. Rzeszotarski; Allison Koenecke
Affiliations: Cornell University; Cornell University; Cornell University; Loyola University Maryland; Cornell University
Year: 2026
Domain: Recommender Systems

Abstract: Recommender systems (RS), which are widely deployed across high-stakes domains, are susceptible to biases that can cause large-scale societal impacts. Researchers have proposed methods to measure and mitigate such biases—but translating academic theory into practice is inherently challenging. Through a semi-structured interview study (N=11), we map the RS practitioner workflow within large technology companies, focusing on how technical teams consider fairness internally and in collaboration with legal, data, and fairness teams. We identify key challenges to incorporating fairness into existing RS workflows: defining fairness in RS contexts, balancing multi-stakeholder interests, and navigating dynamic environments. We also identify key organization-wide challenges: making time for fairness work and facilitating cross-team communication. Finally, we offer actionable recommendations for the RS community, including practitioners and HCI researchers.
Title: Bias Is a Math Problem, AI Bias Is a Technical Problem: 10-Year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of ‘Bias’, and Academia-Industry Gap
Authors: Sourojit Ghosh; Kyra Wilson
Affiliations: University of Washington; University of Washington
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The rapid development of AI tools and implementation of LLMs within downstream tasks has been paralleled by a surge in research exploring how the outputs of such AI/LLM systems embed biases, a research topic which was already being extensively explored before the era of ChatGPT. Given the high volume of research around the biases within the outputs of AI systems and LLMs, it is imperative to conduct systematic literature reviews to document throughlines within such research. In this paper, we conduct such a review of research covering AI/LLM bias in four premier venues/organizations -- *ACL, FAccT, NeurIPS, and AAAI -- published over the past 10 years. Through a coverage of 189 papers, we uncover patterns of bias research and along what axes of human identity they commonly focus. The first emergent pattern within the corpus was that 155/189 papers did not establish a working definition of 'bias' for their purposes, opting instead to simply state that biases and stereotypes exist that can have harmful downstream effects while establishing only mathematical and technical definition of bias. 94 of these 155 papers have been published in the past 5 years, after literature reviews were published with a similar finding about NLP research and recommendation to consider how such researchers should conceptualize bias, going beyond strictly technical definitions. Furthermore, we find that a large majority of papers -- 151/189 papers -- focus on gender bias (mostly, gender and occupation bias) within the outputs of AI systems and LLMs. By demonstrating a strong focus within the field on gender, race/ethnicity (57/189 papers), age (39/189 papers), religion (36/189 papers) and nationality (25/189 papers) bias, we document how researchers adopt a fairly narrow conception of AI bias by overlooking several non-Western communities in fairness research, as we advocate for a stronger coverage of such populations. Finally, we note that while our corpus contains several examples of innovative debiasing methods across the aforementioned aspects of human identity, only 20/189 papers include recommendations for how to implement their findings or contributions in real-world AI systems or design processes. This indicates a concerning academia-industry gap, especially since many of the biases that our corpus contains several successful mitigation methods that still persist within the outputs of AI systems and LLMs commonly used today. We conclude with recommendations towards future AI/LLM fairness research, with stronger focus on diverse marginalized populations.
Title: Advancing Fairness in Generative AI Through Intrinsic and Extrinsic Bias Evaluation and Mitigation
Authors: Mina Arzaghi
Affiliations: Mila - Quebec Artificial Intelligence Institute
Year: 2025
Domain: LLMs & NLP

Abstract: This research focuses on understanding and mitigating bias in generative AI models, specifically by examining both intrinsic and extrinsic biases. The project aims to develop a unified evaluation framework and bias mitigation strategies to promote fairness across real-world applications, such as finance and healthcare. The goal is to ensure generative AI systems do not propagate harmful societal biases, and the research explores bias detection and mitigation across various deployment stages.
Title: NLP Needs Diversity outside of ‘Diversity’
Authors: Joshua Tint
Affiliations: Arizona State University
Year: 2025
Domain: LLMs & NLP

Abstract: This position paper argues that recent progress with diversity in NLP is disproportionately concentrated on a small number of areas surrounding fairness. We further argue that this is the result of a number of incentives, biases, and barriers which come together to disenfranchise marginalized researchers in non-fairness fields, or to move them into fairness-related fields. We substantiate our claims with an investigation into the demographics of NLP researchers by subfield, using our research to support a number of recommendations for ensuring that all areas within NLP can become more inclusive and equitable. In particular, we highlight the importance of breaking down feedback loops that reinforce disparities, and the need to address geographical and linguistic barriers that hinder participation in NLP research.
Title: Exploring Social Choice Mechanisms for Recommendation Fairness in SCRUF
Authors: Amanda Aird; Cassidy All; Paresha Farastu; Elena Štefancová; Joshua Sun; Nicholas Mattei; Robin Burke
Affiliations: University of Colorado, Boulder; University of Colorado, Boulder; University of Colorado, Boulder; Comenius University Bratislava; Independent Researcher; Tulane University; University of Colorado, Boulder
Year: 2023
Domain: Recommender Systems

Abstract: Fairness problems in recommender systems often have a complexity in practice that is not adequately captured in simplified research formulations. A social choice formulation of the fairness problem, operating within a multi-agent architecture of fairness concerns, offers a flexible and multi-aspect alternative to fairness-aware recommendation approaches. Leveraging social choice allows for increased generality and the possibility of tapping into well-studied social choice algorithms for resolving the tension between multiple, competing fairness concerns. This paper explores a range of options for choice mechanisms in multi-aspect fairness applications using both real and synthetic data and shows that different classes of choice and allocation mechanisms yield different but consistent fairness / accuracy tradeoffs. We also show that a multi-agent formulation offers flexibility in adapting to user population dynamics.
Title: Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties
Authors: Fahim Faisal; Md. Mushfiqur Rahman; Antonios Anastasopoulos
Affiliations: George Mason University; George Mason University; George Mason University
Year: 2024
Domain: LLMs & NLP

Abstract: There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: \url{https://github.com/ffaisal93/dialect_toxicity_llm_judge}
Title: Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages
Authors: Ercong Nie; Sheng Liang; Helmut Schmid; Hinrich Schütze
Affiliations: LMU Munich; LMU Munich; LMU Munich; LMU Munich
Year: 2022
Domain: LLMs & NLP

Abstract: Multilingual Pretrained Language Models (MPLMs) have shown their strong multilinguality in recent empirical cross-lingual transfer studies. In this paper, we propose the Prompts Augmented by Retrieval Crosslingually (PARC) pipeline to improve the zero-shot performance on low-resource languages (LRLs) by augmenting the context with semantically similar sentences retrieved from a high-resource language (HRL) as prompts. PARC improves the zero-shot performance on three downstream tasks (binary sentiment classification, topic categorization and natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in both unlabeled settings (+5.1%) and labeled settings (+16.3%). PARC-labeled also outperforms the finetuning baseline by 3.7%. We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between the high- and low-resource languages as well as the amount of low-resource pretraining data on the other side. A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.
Title: Beyond Demographics: Evaluating News Recommender Systems Fairness Through Behavioural Communities
Authors: Bernhard Steindl; Thomas Elmar Kolb; Julia Neidhardt
Affiliations: TU Wien; TU Wien; TU Wien
Year: 2025
Domain: Recommender Systems

Abstract: Fairness in recommender systems is often framed around demographic attributes.In this work, we explore a novel directionevaluating fairness across latent behavioural communities derived from user interactions on a real-world news platform.Using graphbased community detection (Louvain and Infomap), we identify large user groups and examine how different network modelling choices affect fairness outcomes in both traditional and fairnessaware recommender systems.Experiments on an Austrian news dataset reveal that small changes in graph construction considerably impact community formation and recommendation quality.Notably, fairness-aware algorithms show only marginal improvements over standard approaches, underscoring the complexity of achieving equitable outcomes in real-world systems and raising important questions for future research.
Title: Fairness and Diversity in Recommender Systems: A Survey
Authors: Yuying Zhao; Yu Wang; Yunchao Liu; Xueqi Cheng; Charų C. Aggarwal; Tyler Derr
Affiliations: Vanderbilt University; Vanderbilt University; Vanderbilt University; Vanderbilt University; IBM Research - Thomas J. Watson Research Center; Vanderbilt University
Year: 2024
Domain: Recommender Systems

Abstract: Recommender systems (RS) are effective tools for mitigating information overload and have seen extensive applications across various domains. However, the single focus on utility goals proves to be inadequate in addressing real-world concerns, leading to increasing attention to fairness-aware and diversity-aware RS. While most existing studies explore fairness and diversity independently, we identify strong connections between these two domains. In this survey, we first discuss each of them individually and then dive into their connections. Additionally, motivated by the concepts of user-level and item-level fairness, we broaden the understanding of diversity to encompass not only the item level but also the user level. With this expanded perspective on user and item-level diversity, we re-interpret fairness studies from the viewpoint of diversity. This fresh perspective enhances our understanding of fairness-related work and paves the way for potential future research directions. Articles discussed in this survey along with public code links are available at: https://github.com/YuyingZhao/Awesome-Fairness-and-Diversity-Papers-in-Recommender-Systems
Title: Differential Privacy Has Disparate Impact on Model Accuracy
Authors: Eugene Bagdasaryan; Vitaly Shmatikov
Affiliations: Cornell University; Cornell University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.
Title: Exploring gender biases in ML and AI academic research through systematic literature review
Authors: Sunny Shrestha; Sanchari Das
Affiliations: University of Denver; University of Denver
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Automated systems that implement Machine learning (ML) and Artificial Intelligence (AI) algorithms present promising solutions to a variety of technological and non-technological issues. Although, industry leaders are rapidly adopting these systems for anything from marketing to national defense operations, these systems are not without flaws. Recently, many of these systems are found to inherit and propagate gender and racial biases that disadvantages the minority population. In this paper, we analyze academic publications in the area of gender biases in ML and AI algorithms thus outlining different themes, mitigation and detection methods explored through research in this topic. Through a detailed analysis of N = 120 papers, we map the current research landscape on gender specific biases present in ML and AI assisted automated systems. We further point out the aspects of ML/AI gender biases research that are less explored and require more attention. Mainly we focus on the lack of user studies and inclusivity in this field of study. We also shed some light into the gender bias issue as experienced by the algorithm designers. In conclusion, in this paper we provide a holistic view of the breadth of studies conducted in the field of exploring, detecting and mitigating gender biases in ML and AI systems and, a future direction for the studies to take in order to provide a fair and accessible ML and AI systems to all users.
Title: Artificial intelligence in nursing education: A scoping review of intelligent tutoring systems-evidence and implementation gaps
Authors: Nevena Dicheva; Ikram Ur Rehman; Laden Husamaldin; Sama Aleshaiker
Affiliations: University of West London; University of West London; University of West London; University of West London
Year: 2026
Domain: Health & Clinical AI

Abstract: The concept of ITSs is widely implemented across educational settings, but their adoption in nursing education remains poorly understood. To investigate the current state of evidence on ITS in nursing education, identify research gaps and provide recommendations for future research and practice. A scoping review based on the JBI methodology framework. A systematic literature search was initiated (between June and October 2025) across several databases (IEEE Xplore, PubMed, MEDLINE, ERIC, ProQuest, ScienceDirect, and Google Scholar). Grey literature sources and websites of key organisations (e.g., NMC, NICE, IHI, NHS) were also searched. There were no date restrictions, but only articles in English were considered. The screening and selection process was based on the extended PRISMA-ScR framework. Initially, 881 records were retrieved, but 9 studies were included in our scoping review and analysed. Our results indicate that research on ITS in nursing remains limited, with most studies presenting incomplete ITS architectures that focus primarily on domain models or standalone concepts. Among the 9 studies, there is limited evidence suggesting considerations of user-needs evaluation, theoretical frameworks, empirical validation, or curricular integration. Evidence on ITS in nursing education is limited and inconsistent. Our results led to recommendations for future research and practice, including developing a grounded theory of user perspectives, conducting rigorous empirical evaluations, designing and testing comprehensive ITS architectures aligned with nursing education philosophies, and promoting inclusivity by involving underrepresented populations and low-resource settings. • The first review on Intelligent Tutoring Systems (ITS) in nursing education • Conducted following JBI methodology and checklist, and screening process reported as per PRISMA-ScR • Only 9 studies were eligible for inclusion and analysed • The concept of ITS remains poorly investigated and inconsistent • Recommendations for future research and practices include grounded theory development, complete ITS design and development, establishing frameworks for implementation in nursing curriculum, a diverse and large study population and consideration of low-resource settings.
Title: In Anticipation of Perfect Deepfake: Identity-anchored Artifact-agnostic Detection under Rebalanced Deepfake Detection Protocol
Authors: Wei-Han Wang; Chin-Yuan Yeh; Hsi-Wen Chen; De-Nian Yang; Ming-Syan Chen⋆
Affiliations: National Taiwan University; National Taiwan University; National Taiwan University; Academia Sinica; National Taiwan University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: As deep generative models advance, we anticipate deepfakes achieving "perfection"-generating no discernible artifacts or noise. However, current deepfake detectors, intentionally or inadvertently, rely on such artifacts for detection, as they are exclusive to deepfakes and absent in genuine examples. To bridge this gap, we introduce the Rebalanced Deepfake Detection Protocol (RDDP) to stress-test detectors under balanced scenarios where genuine and forged examples bear similar artifacts. We offer two RDDP variants: RDDP-WHITEHAT uses white-hat deepfake algorithms to create 'self-deepfakes,' genuine portrait videos with the resemblance of the underlying identity, yet carry similar artifacts to deepfake videos; RDDP-SURROGATE employs surrogate functions (e.g., Gaussian noise) to process both genuine and forged examples, introducing equivalent noise, thereby sidestepping the need of deepfake algorithms. Towards detecting perfect deepfake videos that aligns with genuine ones, we present ID-Miner, a detector that identifies the puppeteer behind the disguise by focusing on motion over artifacts or appearances. As an identity-based detector, it authenticates videos by comparing them with reference footage. Equipped with the artifact-agnostic loss at frame-level and the identity-anchored loss at video-level, ID-Miner effectively singles out identity signals amidst distracting variations. Extensive experiments comparing ID-Miner with 12 baseline detectors under both conventional and RDDP evaluations with two deepfake datasets, along with additional qualitative studies, affirm the superiority of our method and the necessity for detectors designed to counter perfect deepfakes.
Title: RepMatch: Quantifying Cross-Instance Similarities in Representation Space
Authors: Mohammad Modarres; Abbasi, Sina; Mohammad Taher Pilehvar
Affiliations: Khatam University; Khatam University; Cardiff University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Advances in dataset analysis techniques have enabled more sophisticated approaches to analyzing and characterizing training data instances, often categorizing data based on attributes such as ``difficulty''. In this work, we introduce RepMatch, a novel method that characterizes data through the lens of similarity. RepMatch quantifies the similarity between subsets of training instances by comparing the knowledge encoded in models trained on them, overcoming the limitations of existing analysis methods that focus solely on individual instances and are restricted to within-dataset analysis. Our framework allows for a broader evaluation, enabling similarity comparisons across arbitrary subsets of instances, supporting both dataset-to-dataset and instance-to-dataset analyses. We validate the effectiveness of RepMatch across multiple NLP tasks, datasets, and models. Through extensive experimentation, we demonstrate that RepMatch can effectively compare datasets, identify more representative subsets of a dataset (that lead to better performance than randomly selected subsets of equivalent size), and uncover heuristics underlying the construction of some challenge datasets.