cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Mitigating Bias in Large Language Models Through Culturally-Relevant LLMs
Authors: Malur Narayan; John Pasmore; Elton Sampaio; Raghavan Vijay; Sayan Maity; Gabriella Waters; Ayanna Howard
Affiliations: Jimei University; National Latina Institute for Reproductive Health; Solar System Exploration Research Virtual Institute; Integrated Systems Incorporation (United States); Morgan State University; The Ohio State University
Year: 2025
Domain: LLMs & NLP

Abstract: With the proliferation of Large Language Models (LLMs) and its use across sectors that impact our civil liberties, the biases present in these language-based applications can propagate and amplify social inequalities and harmful stereotypes. In particular, racial and cultural biases in LLMs not only undermine the fairness and reliability of these systems, but also pose significant ethical concerns, further affecting marginalized communities. To address these concerns, this paper discusses the development of a LLM for mitigating bias through the integration of diverse data sources and culturally relevant material that has often been overlooked in the construction of mainstream AI models. We compare the performance of ChatGPT 3.5 and the LLM, which was incrementally trained on Black history and culture, when tasked with answering a set of prompts that cover a range of news topics. Using a bias assessment score, we assessed each model's ability to handle the task without introducing racial or cultural biases into the news topic summaries. Through empirical studies, our approach not only demonstrates the feasibility of detecting and measuring racial and cultural biases but also offers a scalable solution for creating more equitable and reliable AI technologies.
Title: Probing into the Fairness of Large Language Models: A Case Study of ChatGPT
Authors: Yunqi Li; Lanjing Zhang; Yongfeng Zhang
Affiliations: Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2024
Domain: LLMs & NLP

Abstract: Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited number of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT’s performance in high-takes fields including education, criminology, finance and healthcare. To conduct a thorough evaluation, we consider both group fairness and individual fairness metrics. We also observe the disparities in ChatGPT’s outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs’ fairness performance, facilitates bias mitigation and fosters the development of responsible AI systems. Code and data are open-sourced on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Title: Leveraging Multiple LLM Evaluators for Scalable and Fair Language Model Assessments
Authors: Vincent Koc; Jamshaid Iqbal Janjua; Karan Alang; Sumeer Basha Peta
Affiliations: University of Kuala Lumpur; University of Engineering and Technology Lahore; Oldham Council; Southern Illinois University Edwardsville
Year: 2025
Domain: LLMs & NLP

Abstract: With the increasing popularity of large language models (LLMs), the assessment of their outputs remains a persistent issue. LLMs are often evaluated by human evaluators, which leads to the middle evaluator bias. Single evaluator approaches produce reliable evaluations and the robustness of evaluations performed relying on a single evaluator is challenged. This paper proposes an automated evaluation method that incorporates multiple LLM evaluators to increase scalability and unjustified bias mitigation of model evaluations. This approached is resulted from combining models from different families to overcome the issues presented by single models evaluators such as bias, excessive computational resources, and generalization. Our approach is based on processing evaluation within a multi-evaluator framework in which score aggregation on a per evaluator basis is done, which utilizes lower cost to achieve higher accuracy and transparency in evaluations. We tested this model using several LLMs with common evaluation tasks and showed that this approach has outcomes applicable in practice.
Title: Emergent Biases in Large Language Models: A Critical Review of Taxonomy and Evaluations
Authors: Shawn Ismail; Ramazan Aygün
Affiliations: Kennesaw State University; Kennesaw State University
Year: 2025
Domain: LLMs & NLP

Abstract: Artificial intelligence (AI) and machine learning (ML) algorithms are widely used throughout our economy in making decisions that have far-reaching impacts on employment, education, access to credit, and other areas. Initially considered neutral and fair, ML algorithms have recently been found increasingly biased, creating and perpetuating structural inequalities in society. With the rising concerns about algorithmic bias, a growing body of literature attempts to understand and resolve the issue of algorithmic bias. In this tutorial, we discuss five important aspects of algorithmic bias. We start with its definition and the notions of fairness policy makers, practitioners, and academic researchers have used and proposed. Next, we note the challenges in identifying and detecting algorithmic bias given the observed decision outcome, and we describe methods for bias detection. We then explain the potential sources of algorithmic bias and review several bias-correction methods. Finally, we discuss how agents’ strategic behavior may lead to biased societal outcomes, even when the algorithm itself is unbiased. We conclude by discussing open questions and future research directions.
Title: Bias Detection and Mitigation in Large Language Models for Code Generation
Authors: Alexander I. Iliev; Deepshikha Singh; Sanjeeth Chittyala
Affiliations: Bulgarian Academy of Sciences; SRH University Berlin; SRH University Berlin
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) are essential tools in modern software development, significantly accelerating coding, streamlining debugging, and enabling broader access to advanced algorithmic features. Their impact extends into emerging domains like the Internet of Things (IoT), where AI-generated code increasingly drives edge device behavior and smart system integration. However, LLMs often inherit and propagate biases present in their training data. These biases can surface in AI-generated code, leading to ethically problematic, algorithmically skewed, or even unsafe outcomes—particularly concerning in safety-critical and socially sensitive IoT environments. This research investigates bias in LLM-generated code and proposes a multi-faceted approach combining Contextual Code Analysis, Counterfactual Prompt Engineering, and Reinforcement Learning with Human Feedback (RLHF) to detect and mitigate such biases. We show how these techniques reveal hidden biases in naming conventions, decision logic, and coding patterns, and demonstrate how RLHF can reduce bias at scale while preserving functionality – achieving a 49% reduction. in bias across evaluated benchmarks. Our findings emphasize the need for rigorous bias evaluation frameworks, careful data curation, and transparent development workflows. The tradeoff between bias reduction and code precision is explored, with implications for the ethical use of AI in IoT systems and other high-stakes software contexts. Further research is encouraged in hybrid debiasing techniques, intersectional bias identification, energy-efficient modeling, and the development of standardized ethical coding practices.
Title: Applying Prompt-Based Mitigation of Gender-Role Bias in Large Language Models for Security
Authors: K. Stein; Brandon Blackwell; Ernest Funue; Matthew Dees; Shari Watkins; Lanier Watkins
Affiliations: Johns Hopkins University; Johns Hopkins University; Johns Hopkins University; Johns Hopkins University; American University; Johns Hopkins University
Year: 2025
Domain: LLMs & NLP

Abstract: As large language models (LLMs) become increasingly integrated into society, concerns over fairness and bias have emerged as critical areas of research. Prior research has demonstrated that LLMs often reflect stereotypes present in their training data, and while this is informative, systematic studies producing rigorous experiments and tangible measurements re-main limited. Furthermore, as LLMs are deployed in security-sensitive domains, such as industrial control systems (ICS), Internet-of- Things (loT) networks, and critical infrastructure, ex-ploring tangible, quantitative bias mitigation techniques becomes essential for real-time deployment. In this work, as a proof of concept, we apply our pipelines to 53 gender-neutral professional resume prompts to generally analyze how LLMs assign gender-specific attributes. Using these prompts, we implement four distinct mitigation pipelines: a Baseline with no intervention, In-Context Example prompting, Self-Evaluation and Regeneration, and Chain-of- Thought (CoT) reasoning. Furthermore, to quantify each pipeline, we introduce the Stereotype Alignment Rate (SAR) to quantify bias, defined as the percentage of outputs conforming to traditional gender stereotypes. Results across each experimental pipeline indicate an effective reduction in bias, lowering the SAR by up to 15% compared to the baseline. This work demonstrates that prompt engineering, particularly through the use of counter-stereotypical examples, can serve as an effective and lightweight method for mitigating bias in LLM outputs.
Title: Behind the Words: Unmasking Bias Lurking in LLMs and Data
Authors: Naina Dhingra; Gargi Kale; Harshada Chavan; Sandhya Arora
Affiliations: MKSSS's Cummins College of Engineering for Women
Year: 2025
Domain: LLMs & NLP

Abstract: This study examines hidden biases in large language models (LLMs). Three popular models-RoBERTa, BERT, and GPT-2-were evaluated using StereoSet, CrowS-Pairs, and Indian-BhED, datasets designed to expose biases. Our comparative analysis reveals clear variations in how these models encode and reproduce bias, underscoring the difficulty of mitigation. The results highlight the need for bias-aware evaluation. By contributing empirical evidence, this work advances the discussion on ethical AI and points to directions for building more responsible language technologies.
Title: Deep Learning Approach for Evaluating Fairness in LLMs
Authors: Laxmi Kumari; Shilpa Serasiya; Santosh Kumar Bharti
Affiliations: Pandit Deendayal Petroleum University
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) are prone to bias that can be introduced in the dataset itself or in the training phase. This can make their output lacking in fairness and hinder certain discriminated groups on the basis of stereotypes that the LLM may have been fed with or learnt. In today’s day and age, LLMs are used to make major decisions and thus it is crucial to identify and remove such bias. In this paper, we have focused on identifying the bias present in medium-sized LLMs such as BERT, RoBERTA, DeBERTa and GPT-1 using bias evaluation metrics such as intrinsic and extrinsic. Both the metrics have been used and bias has been identified in the datasets that the LLMs are being trained on (intrinsic bias) as well as in the outputs of the LLMs for certain queries (extrinsic bias).
Title: Comparative Study of Large Language Model Evaluation Frameworks with a Focus on NLP vs LLM-As-A-Judge Metrics
Authors: Afnan Alabdulwahab; Chloe Japic; Chao Le; Deepika Dubey; Bhadra Trivedi; John Hope; Patrick Stone; Sanjana Srivastava; Adam P. Tashman; Aidong Zhang
Affiliations: University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia; University of Virginia
Year: 2025
Domain: LLMs & NLP

Abstract: As Large Language Models (LLMs) are increasingly deployed across industries, evaluating their performance remains a core challenge. In collaboration with Deloitte, this study compares traditional Natural Language Processing (NLP) metrics with the emerging LLM-as-a-Judge paradigm across tasks, including retrieval, response accuracy, toxicity, bias, hallucination, summarization, tone, and readability. Using models like Claude and both public and synthetic datasets, we conduct structured evaluations via API. Results show that LLM-as-a-Judge methods offer nuanced assessments but face self-referential bias and consistency issues, while traditional metrics remain transparent yet limited. We propose an evaluation playbook outlining tradeoffs and best-use scenarios to support standardized, responsible LLM assessment.
Title: Auditing Demographic Bias in Mistral: An Open-Source LLM's Diagnostic Performance on the MedQA Benchmark
Authors: Hosameldin O. A. Ahmed; Abdul Hamid Sadka
Affiliations: Aston University; Aston University
Year: 2026
Domain: LLMs & NLP

Abstract: The application of large language models (LLMs) within clinical decision-support frameworks is receiving growing research attention, yet their fairness and demographic robustness remain insufficiently understood. This study introduces MedQA-Demog, a purpose-built, label-invariant extension of the MedQA-USMLE benchmark, designed to enable systematic auditing of demographic bias in medical reasoning models. Using a deterministic augmentation framework, we generated 4,659 question-answer items that incorporated counterfactual variations in gender, race/ethnicity, and age, and validated them through automated integrity and balance checks. We evaluated the Mistral 7B-Instruct model under stochastic (temperature = 0.7) and deterministic (temperature = 0.0) inference rules via the Ollama local environment, applying Wilson's 95 % confidence intervals, χ²/<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</i>-tests, McNemar’s paired analysis, and Cohen’s h effect sizes to quantify fairness. Across all demographic variants, diagnostic accuracy remained consistent (Δ < 0.04; p > 0.05), and all performance gaps fell within Minimal or Low Bias thresholds. Confusion-matrix and prediction-balance analyses revealed no systematic over- or under-prediction patterns, while power analysis confirmed that observed fluctuations were below the minimum detectable effect (≈ 0.057). A stratified robustness analysis further confirms that these fairness patterns persist across question difficulty levels and are not an artefact of uniformly limited performance. These findings demonstrate that open-weight, instruction-tuned LLMs can maintain demographic stability in clinical reasoning when evaluated through reproducible, controlled pipelines. This framework provides a practical foundation for bias evaluation in open clinical LLMs, supporting their ethical integration into digital health tools and clinical decision-support systems.
Title: A Vision for Debiasing Confirmation Bias in Software Testing via LLM
Authors: Iflaah Salman; Muhammad Waseem; Vladimir Mandić; Rasanjana Dhanushkha De Alwis
Affiliations: Lappeenranta-Lahti University of Technology; Tampere University; University of Novi Sad; Lappeenranta-Lahti University of Technology
Year: 2025
Domain: LLMs & NLP

Abstract: Background: Large language models (LLM) suffer from various forms of biases due to the biased datasets used to train the models. At the same time, human cognitive biases have an equal propensity to express themselves when using LLMs for software engineering tasks. Software testing is a critical phase of the software development life cycle. Confirmation bias is reported to have deteriorated software testing by designing more specification-consistent test cases compared to specificationinconsistent test cases. However, there is a lack of debiasing (mitigation) strategies in this regard. Aims: In this paper, first, we investigate whether the LLM model suffers from confirmation bias while performing software testing tasks. Second, we propose a vision of debasing confirmation bias in software testing via LLM. Method: We conducted an empirical study to detect confirmation bias by an LLM (ChatGPT4.0) in the design of functional test cases. Based on empirical findings, we used the analytical paradigm to design a multi-agent system. Results: We present a vision for debiasing confirmation bias in functional software testing by leveraging LLMs via a multi-agent approach. Conclusions: The proposed vision may improve the performance of LLMs in terms of reduced confirmation bias and serve as a debiasing technique for functional software testing.
Title: Measuring and Mitigating Gender Entropy Bias in Large Language Models
Authors: Sonal Prabhune; B. Padmanabhan; Kaushik Dutta
Affiliations: University of South Florida; Smith Institute; University of South Florida
Year: 2025
Domain: LLMs & NLP

Abstract: We investigate gender bias in the form of differences in information content in some of the popular LLMs and contribute a new benchmark dataset (released on HuggingFace) developed from real-world questions across four key domains in business and health contexts: education, jobs, personal financial management, and general health. Specifically, we introduce entropy bias, which we define as a discrepancy in the amount of information generated by an LLM in response to real questions users have asked. Grounded in authentic user questions and guided by quantitative metrics, our benchmark enables systematic and reproducible analysis of gender entropy biases in language model outputs, which we test using multiple LLMs. Our analyses (metric-based comparisons and “LLM-as-judge” evaluation) suggest that there is no significant bias in LLM responses for men and women at a category level. However, at a finer granularity (the individual question level) there are substantial differences in LLM responses for men and women in the majority of cases, which “cancel” each other out often due to some responses being better for males and vice versa. To address this, we suggest a simple debiasing approach that iteratively merges the responses for the two genders to produce a final result. Our approach demonstrates that a simple, prompt-based debiasing strategy can effectively debias LLM outputs, thus producing responses with higher information content than both gendered variants more than 77% of the cases, and consistently achieving a balanced integration in the remaining cases.
Title: Spoken Stereoset: on Evaluating Social Bias Toward Speaker in Speech Large Language Models
Authors: Yi-Cheng Lin; Wei-Chih Chen; Hung-yi Lee
Affiliations: National Taiwan University; National Taiwan University; National Taiwan University
Year: 2024
Domain: LLMs & NLP

Abstract: Warning: This paper may contain texts with uncomfortable content.Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
Title: Position Bias Across LLM Model Families
Authors: Salvatore Vella; Salah Sharieh; Alexander Ferworn
Affiliations: Toronto Metropolitan University; Toronto Metropolitan University; Toronto Metropolitan University
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models are being used to solve business problems. Their ability to perform tasks and automate business processes provides value in many industries. Open source models, such as Meta’s Llama models, are becoming popular because they can be used on-premises, which is attractive for regulated industries. They can be embedded as they come in various sizes, from 1 billion to 405 billion parameters. OpenAI This study studies the positional bias of the Meta family of models using sizes of $1,3,8,70$ and 405 billion parameter models. OpenAI has a suite of models that are available on the cloud. We use two multiple-choice question and answer data sets from OpenBookQA and CommonsenseQA. We show that while the larger models exhibit less positional bias than the small models, they nevertheless all exhibit positional bias of some sort. These results have implications for model users to be aware of the bias in the models as they process text with multiple options, as is common with business policies and procedures.
Title: Knowledge Graph-Based Diversified Recommendation
Authors: Liang Chen; Yan Jiang; Jingang Ma
Affiliations: Shenyang University of Technology; Shenyang University of Technology; Shenyang University of Technology
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: With the rapid growth of the information age, traditional graph-based recommendation systems have significantly improved accuracy. However, these systems often rely on user-item bipartite graphs, neglecting complex relational information, which limits recommendation diversity, especially for long-tail items. To address this, we propose KGR, a knowledge graph-enhanced recommendation system that utilizes the semantic richness of knowledge graphs to improve diversity while balancing the trade-off between diversity and accuracy in traditional GNN models. We design three key modules: first, we use knowledge graph category and relational information to select diverse neighbors for aggregation with a submodular function; second, we apply an adaptive layer attention mechanism to adjust layer weights dynamically; and third, we introduce a loss reweighting module to prioritize long-tail item categories, improving recommendations for cold-start and long-tail items. Experimental results validate the KGR model's effectiveness in enhancing both diversity and accuracy.
Title: Bursting the Filter Bubble with Knowledge Graph Inversion
Authors: Fernando Spadea; Oshani Seneviratne
Affiliations: Rensselaer Polytechnic Institute; Rensselaer Polytechnic Institute
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: As recommender systems increasingly mediate our online experiences, ethical concerns arise as they often reinforce filter bubblesnarrow content loops that isolate users from diverse perspectives.In this paper, we propose a novel approach to mitigating filter bubbles by combining personalized knowledge graph (KG) completion with federated learning (FL) and KG edge inversion techniques.User-specific KGs are constructed from private interaction data and remain entirely on-device, preserving privacy while enabling the system to learn fine-grained preferences.A central recommendation model is trained collaboratively via FL, allowing KG completion without exposing sensitive user data.To disrupt filter bubbles, we introduce KG edge inversion, a method that strategically inverts selected relations in the user's KG to simulate alternative viewpoints.The model then produces plausible yet diverse recommendations, effectively guiding users beyond their typical content landscape.Our approach offers a transparent and controllable framework for promoting opinion diversity and combating the isolating effects of algorithmic personalization.