cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Fairness Testing: Review of the Relevance and Methodologies of Ensuring Software Fairness
Authors: Ilmaan Zia; Ankit Raj; Jonathan Bernabe Dabu
Affiliations: California State University, Long Beach; California State University, Long Beach; California State University, Long Beach
Year: 2026
Domain: General Fairness & Bias Mitigation

Abstract: Generative AI has emerged as a transformative force in health care, driving advancements in medical diagnostics, personalized treatments, and operational efficiencies. As this technology becomes increasingly prevalent, it brings both opportunities and challenges, particularly concerning fairness and bias. This position paper delves into the impact of generative AI on health care by first unveiling the hidden challenges of bias and its implications for equitable medical outcomes. We present a balanced debate on the advantages and disadvantages of bias mitigation strategies, providing a critical analysis of their effectiveness and feasibility. Our exploration includes current recommendations aimed at ensuring fairness in AI applications. Through real-world examples, we illustrate how bias has been successfully addressed and the lessons learned. By integrating these elements, this position paper offers a comprehensive perspective on how generative AI can be harnessed to improve health care while navigating the complexities of fairness and equity.
Title: Aggregating Concepts of Fairness and Accuracy in Prediction Algorithms
Authors: David Kinney
Affiliations: Washington University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: An algorithm that outputs predictions about the state of the world will almost always be designed with the implicit or explicit goal of outputting accurate predictions (i.e., predictions that are likely to be true).In addition, the rise of increasingly powerful predictive algorithms brought about by the recent revolution in artificial intelligence has led to an emphasis on building predictive algorithms that are fair, in the sense that their predictions do not systematically evince bias or bring about harm to certain individuals or groups.This state of affairs presents two conceptual challenges.First, the goals of accuracy and fairness can sometimes be in tension, and there are no obvious normative guidelines for managing the tradeoffs between these two desiderata when they arise.Second, there are many distinct ways of measuring both the accuracy and fairness of a predictive algorithm; here too, there are no obvious guidelines on how to aggregate our preferences for predictive algorithms that satisfy disparate measures of fairness and accuracy to various extents.The goal of this paper is to address these challenges by arguing that there are good reasons for using a linear combination of accuracy and fairness metrics to measure the all-things-considered value of a predictive algorithm for agents who care about both accuracy and fairness.My argument depends crucially on a classic result in the preference aggregation literature due to Harsanyi [13].After making this formal argument, I apply my result to an analysis of accuracy-fairness trade-offs using the COMPAS dataset compiled by Angwin et al. ([1]). CCS Concepts Theory of computation Design and analysis of algorithms.
Title: From AI Bias to AI by Us: A Case Study from MIT Critical Data
Authors: Catherine Bielick; Rodrigo Gameiro; Leo Celi
Affiliations: MIT Critical Data; Beth Israel Deaconess Medical Center; Harvard Medical School; MIT Critical Data; Beth Israel Deaconess Medical Center; Harvard Medical School; MIT Critical Data; Beth Israel Deaconess Medical Center; Harvard Medical School
Year: 2026
Domain: Health & Clinical AI

Abstract: Recent advancements in generative AI, particularly in computer vision and natural language processing, have brought significant innovations and highlighted critical bias and fairness issues. This paper comprehensively reviews bias in generative AI, examining its causes, impacts, and potential solutions from legal, ethical, and technical perspectives. I begin by discussing the current state of bias in generative AI, focusing on racial, gender, and cultural biases in both computer vision and natural language processing. Through case studies, I demonstrate the real-world impacts of these biases. The paper then explores the root causes of bias, including data imbalance and algorithmic design. It discusses the profound social and technical impacts, such as implications for social justice, trust in AI systems, and model performance. I review existing domestic and international policies, industry standards, and practices to mitigate AI bias, highlighting their strengths and limitations. The paper concludes with proposed solutions for improving data diversity, developing fairness-aware algorithms, enhancing regulatory frameworks, and promoting ethical AI education and public awareness. Our study underscores the need for continuous efforts and interdisciplinary collaboration to address bias and ensure fairness in generative AI systems.
Title: Mitigating Algorithmic Bias in Multiclass CNN Classifications Using Causal Modeling
Authors: Min Kwang Byun; Wendy Hui; Wai Kwong Lau
Affiliations: Institute for Infocomm Research; Institute for Infocomm Research; The University of Western Australia
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: This study describes a procedure for applying causal modeling to detect and mitigate algorithmic bias in a multiclass classification problem. The dataset was derived from the FairFace dataset, supplemented with emotional labels generated by the DeepFace pre-trained model. A custom Convolutional Neural Network (CNN) was developed, consisting of four convolutional blocks, followed by fully connected layers and dropout layers to mitigate overfitting. Gender bias was identified in the CNN model’s classifications: Females were more likely to be classified as "happy" or "sad," while males were more likely to be classified as "neutral." To address this, the one-vs-all (OvA) technique was applied. A causal model was constructed for each emotion class to adjust the CNN model’s predicted class probabilities. The adjusted probabilities for the various classes were then aggregated by selecting the class with the highest probability. The resulting debiased classifications demonstrated enhanced gender fairness across all classes, with negligible impact—or even a slight improvement—on overall accuracy. This study highlights that algorithmic fairness and accuracy are not necessarily trade-offs. All data and code for this study are publicly available for download.
Title: Whither bias goes, I will go: An integrative, systematic review of algorithmic bias mitigation
Authors: Louis Hickman; Christopher Huynh; Jessica Gass; Brandon M. Booth; Jason Kuruzovich; Louis Tay
Affiliations: Virginia Tech; Virginia Tech; Virginia Tech; University of Colorado Boulder; Rensselaer Polytechnic Institute; Purdue University West Lafayette
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning (ML) models are increasingly used for personnel assessment and selection (e.g., resume screeners, automatically scored interviews). However, concerns have been raised throughout society that ML assessments may be biased and perpetuate or exacerbate inequality. Although organizational researchers have begun investigating ML assessments from traditional psychometric and legal perspectives, there is a need to understand, clarify, and integrate fairness operationalizations and algorithmic bias mitigation methods from the computer science, data science, and organizational research literature. We present a four-stage model of developing ML assessments and applying bias mitigation methods, including (a) generating the training data, (b) training the model, (c) testing the model, and (d) deploying the model. When introducing the four-stage model, we describe potential sources of bias and unfairness at each stage. Then, we systematically review definitions and operationalizations of algorithmic bias, legal requirements governing personnel selection from the United States and Europe, and research on algorithmic bias mitigation across multiple domains and integrate these findings into our framework. Our review provides insights for both research and practice by elucidating possible mechanisms of algorithmic bias while identifying which bias mitigation methods are legal and effective. This integrative framework also reveals gaps in the knowledge of algorithmic bias mitigation that should be addressed by future collaborative research between organizational researchers, computer scientists, and data scientists. We provide recommendations for developing and deploying ML assessments, as well as recommendations for future research into algorithmic bias and fairness. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
Title: Detecting and Mitigating Algorithmic Bias in Automated Hiring Systems Using Fairness Constraints
Authors: Parina K. Parekh; S. Divakara Shetty
Affiliations: Birla Institute of Technology and Science, Pilani - Dubai Campus; Birla Institute of Technology and Science, Pilani - Dubai Campus
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Automated hiring systems, powered by machine learning, are increasingly adopted by organizations to streamline candidate selection. However, these systems risk perpetuating and amplifying historical biases, particularly concerning gender and race. This study presents FairHire, a fairness-aware hiring framework that detects and mitigates bias in resume screening models. A baseline model (Random Forest) was trained using a curated resume dataset, evaluated, and subjected to bias analysis. Fairness metrics including Demographic Parity Difference and Equalized Odds Difference were computed using Fairlearn. Mitigation strategies were then applied to reduce bias in the sensitive attributes. Results demonstrate that fairness constraints can substantially reduce bias when applied during model training. The study highlights the importance of incorporating fairness into AI-based hiring workflows and provides a practical framework for hiring and other decision-making applications.
Title: Public Perceptions of Algorithmic Bias and Fairness in Cloud-Based Decision Systems
Authors: Amal A. Alhosban; Ritik Gaire; Hassan Ali Al-Ababneh
Affiliations: University of Michigan–Flint; University of Michigan–Flint; Zarqa University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Cloud-based machine learning systems are increasingly used in sectors such as healthcare, finance, and public services, where they influence decisions with significant social consequences. While these technologies offer scalability and efficiency, they raise significant concerns regarding security, privacy, and compliance. One of the central issues is algorithmic bias, which can emerge from data, design choices, or system interactions, and is often amplified when deployed at scale through cloud infrastructures. This study examines the relationship between algorithmic bias, social equity, and cloud-based innovation. Drawing on a survey of public perceptions, we find strong recognition of the risks posed by biased systems, including diminished trust, harm to vulnerable populations, and erosion of fairness. Participants overwhelmingly supported regulatory oversight, developer accountability, and greater transparency in algorithmic decision-making. Building on these findings, this paper proposes measures to integrate fairness auditing, representative datasets, and bias mitigation techniques into cloud security and compliance frameworks. We argue that addressing bias is not only an ethical responsibility but also an essential requirement for safeguarding public trust and meeting evolving legal and regulatory standards.
Title: Towards Ethical AI: Bias Detection and Mitigation in AI Models for Recruitment Systems and Criminal Justice Systems
Authors: Jayshree Ghorpade-Aher; Aardra Patil; Sonali Ghorpade
Affiliations: MIT World Peace University; MIT World Peace University; Savitribai Phule Pune University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Present Digital ERA is an age of Artificial Intelligence (AI) that has influenced the sensitive real-time domains such as recruitment processes and criminal justice. The decision-making potential with intelligent machines implements various techniques those aim to induce efficacy in the subject oriented procedures. The challenges faced depicted the inheritance of the biases embedded in preliminary available data which has affected to unfair treatment of marginalized assemblies. The presented state-of-art research study examines up-to-date approaches for identifying and dropping such biases within AI-powered hiring and legal systems. We addressed the problems and investigated feasible solutions at algorithmic, data-processing, and policy levels. The proposed findings focus on technical strategies as well as the broader ethical concerns. Through a detailed analysis of tools like fairness metrics, adversarial debiasing, and explainable AI (XAI), we have proposed a comprehensive Hybrid framework that seeks to support fairness without affecting overall accuracy performance. The study thoughtfully explores bias in complex, multimodal AI settings, especially in recruitment. It underlines the growing need for transparent oversight and ethical policies to ensure AI technologies stay responsible and fair.
Title: Algorithmic Bias Detection: A Focus on Skin Tone and Gender Fairness in AI Models
Authors: Happy Iguare; Peter Taiwo
Affiliations: Morgan State University; Morgan State University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: This study delves into the detection of algorithmic bias and fairness benchmarking in Artificial Intelligence (AI) and Machine Learning (ML) systems, particularly within Computer Vision applications, with an emphasis on how skin tone variations affect model performance. We analyze gender prediction among leading AI services such as Amazon Rekognition, Deep Face, Fair Face, and the BadgerCake gender-classifier by applying fairness metrics and conducting bias-focused studies across various skin tones. A novel aspect of our methodology involves mapping detected facial attributes and skin tones using a predefined scale, enabling a detailed performance analysis across gender segments. Image preprocessing techniques, including contrast enhancement, normalization, and speckle removal, are applied to improve image quality and enhance the reliability of skin tone detection. Using metrics such as equalized odds, calibration, demographic parity, and accuracy parity, our comparative analysis reveals significant disparities, including high accuracy for AWS Rekognition and pronounced biases in Fair Face and BadgerCake for darker skin tones. These findings highlight the need for data augmentation, algorithmic adjustments, and diverse training datasets to mitigate biases and refine algorithms. This research underscores the critical need for continuous evaluation and improvement of AI systems to achieve fair and accurate outcomes across diverse populations.
Title: Large Language Model Bias Mitigation from the Perspective of Knowledge Editing
Authors: Ruizhe Chen; Yichen Li; Zikai Xiao; Zuozhu Liu
Affiliations: Zhejiang University; Zhejiang University
; Angelalign Technology Inc. ;Zhejiang University

Year: 2024
Domain: LLMs & NLP

Abstract: Existing debiasing methods inevitably make unreasonable or undesired predictions as they are designated and evaluated to achieve parity across different social groups but leave aside individual facts, resulting in modified existing knowledge. In this paper, we first establish a new bias mitigation benchmark BiasKE leveraging existing and additional constructed datasets, which systematically assesses debiasing performance by complementary metrics on fairness, specificity, and generalization. Meanwhile, we propose a novel debiasing method, Fairness Stamp (FAST), which enables editable fairness through fine-grained calibration on individual biased knowledge. Comprehensive experiments demonstrate that FAST surpasses state-of-the-art baselines with remarkable debiasing performance while not hampering overall model capability for knowledge preservation, highlighting the prospect of fine-grained debiasing strategies for editable fairness in LLMs.
Title: Mitigating the Bias of Large Language Model Evaluation
Authors: Hongli Zhou; Hui Huang; Yunfei Long; Bing Xu; Conghui Zhu; Hailong Cao; Muyun Yang; Tiejun Zhao
Affiliations: Harbin Institute of Technology;Harbin Institute of Technology;University of Essex
Year: 2024
Domain: LLMs & NLP

Abstract: Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.
Title: Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation
Authors: Jizhi Zhang; Keqin Bao; Yang Zhang; Wenjie Wang; Fuli Feng; Xiangnan He
Affiliations: University of Science and Technology of China; University of Science and Technology of China; University of Science and Technology of China; National University of Singapore; University of Science and Technology of China; University of Science and Technology of China
Year: 2023
Domain: Recommender Systems

Abstract: The remarkable achievements of Large Language Models (LLMs) have led to the emergence of a novel recommendation paradigm — Recommendation via LLM (RecLLM). Nevertheless, it is important to note that LLMs may contain social prejudices, and therefore, the fairness of recommendations made by RecLLM requires further investigation. To avoid the potential risks of RecLLM, it is imperative to evaluate the fairness of RecLLM with respect to various sensitive attributes on the user side. Due to the differences between the RecLLM paradigm and the traditional recommendation paradigm, it is problematic to directly use the fairness benchmark of traditional recommendation. To address the dilemma, we propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM). This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes1 in two recommendation scenarios: music and movies. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations. Our code and dataset can be found at https://github.com/jizhi-zhang/FaiRLLM.
Title: Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
Authors: Han Zhou; Xingchen Wan; Yinhong Liu; Nigel Collier; Ivan Vulić; Anna Korhonen
Affiliations: University of Cambridge; University of Oxford; University of Cambridge; University of Cambridge; University of Cambridge; University of Cambridge;
Year: 2024
Domain: LLMs & NLP

Abstract: Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language generation quality. In particular, pairwise LLM evaluators, which compare two generated texts and determine the preferred one, have been employed in a wide range of applications. However, LLMs exhibit preference biases and worrying sensitivity to prompt designs. In this work, we first reveal that the predictive preference of LLMs can be highly brittle and skewed, even with semantically equivalent instructions. We find that fairer predictive preferences from LLMs consistently lead to judgments that are better aligned with humans. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO, which aims to produce fairer preference decisions and improve the alignment of LLM evaluators with human judgments. To this end, we propose a zero-shot learning objective based on the preference decision fairness. ZEPO demonstrates substantial performance improvements over state-of-the-art LLM evaluators, without requiring labeled data, on representative meta-evaluation benchmarks. Our findings underscore the critical correlation between preference fairness and human alignment, positioning ZEPO as an efficient prompt optimizer for bridging the gap between LLM evaluators and human judgments.
Title: RRAO: An Ontology for the Representation of Reoffending Risk Assessment Knowledge
Authors: Sotiris Angelis; Joana Pinho; Athanasia Sykiotou; Dimitar Markov; Stamatis Chatzistamatis; Stamatis Spirou; George E. Tsekouras; Konstantinos Kotis
Affiliations: University of the Aegean; Innovative Research (United States); Democritus University of Thrace; Center for the Study of Democracy; University of the Aegean; University of the Aegean; University of the Aegean; University of the Aegean
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Judicial decision making related to parole, sentencing, rehabilitation, reintegration, and public safety is often supported by the assessment of the risk of reoffending. AI prediction systems can introduce bias in the analysis of reoffending riskrelated data. Several studies criticize the fairness of such AI systems. This paper presents the Reoffending Risk Assessment Ontology (RRAO) which aims to provide a comprehensive representation of reoffending risk and recidivism knowledge integrated into ontology-based AI systems. RRAO is engineered following the X-HCOME ontology engineering (OE) methodology, which provides a hybrid bottom-up (data driven), top-down (expert knowledge) OE approach, including tasks designed to assess and mitigate bias at the schema level. By developing a bias-free risk assessment ontology, our objective is to enhance the fairness of AI-driven ontology-based reoffending risk prediction systems, ultimately contributing to more fair and effective criminal justice practices.