cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Risk Taxonomy Mitigation and Assessment Benchmarks of Large Language Model Systems
Authors: Tianyu Cui; Yanling Wang; Chuanpu Fu; Yong Xiao; Sijia Li; Xinhao Deng; Yunpeng Liu; Qinglin Zhang; Ziyi Qiu; Peiyang Li; Zhixing Tan; Junwu Xiong; Xinyu Kong; Zujie Wen; Ke Xu; Qi Li
Affiliations: Zhongguancun Laboratory; Zhongguancun Laboratory; Tsinghua University; Zhongguancun Laboratory; Institute of Information Engineering, Chinese Academy of Sciences; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Zhongguancun Laboratory; Ant Group; Ant Group; Ant Group; Zhongguancun Laboratory; Zhongguancun Laboratory
Year: 2023
Domain: LLMs & NLP

Abstract: Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta, and Anthropic have also made lots of efforts on responsible LLMs. Therefore, there is a growing need to organize the existing studies and establish comprehensive taxonomies for the community. In this paper, we delve into four essential modules of an LLM system, including an input module for receiving prompts, a language model trained on extensive corpora, a toolchain module for development and deployment, and an output module for exporting LLM-generated content. Based on this, we propose a comprehensive taxonomy, which systematically analyzes potential risks associated with each module of an LLM system and discusses the corresponding mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. We hope that this paper can help LLM participants embrace a systematic perspective to build their responsible LLM systems.
Title: Large Language Models in Healthcare and Medical Domain: A Review
Authors: Zabir Al Nazi; Wei Peng
Affiliations: University of California, Riverside; Arista (United States)
Year: 2024
Domain: LLMs & NLP

Abstract: The deployment of large language models (LLMs) within the healthcare sector has sparked both enthusiasm and apprehension. These models exhibit the remarkable ability to provide proficient responses to free-text queries, demonstrating a nuanced understanding of professional medical knowledge. This comprehensive survey delves into the functionalities of existing LLMs designed for healthcare applications and elucidates the trajectory of their development, starting with traditional Pretrained Language Models (PLMs) and then moving to the present state of LLMs in the healthcare sector. First, we explore the potential of LLMs to amplify the efficiency and effectiveness of diverse healthcare applications, particularly focusing on clinical language understanding tasks. These tasks encompass a wide spectrum, ranging from named entity recognition and relation extraction to natural language inference, multimodal medical applications, document classification, and question-answering. Additionally, we conduct an extensive comparison of the most recent state-of-the-art LLMs in the healthcare domain, while also assessing the utilization of various open-source LLMs and highlighting their significance in healthcare applications. Furthermore, we present the essential performance metrics employed to evaluate LLMs in the biomedical domain, shedding light on their effectiveness and limitations. Finally, we summarize the prominent challenges and constraints faced by large language models in the healthcare sector by offering a holistic perspective on their potential benefits and shortcomings. This review provides a comprehensive exploration of the current landscape of LLMs in healthcare, addressing their role in transforming medical applications and the areas that warrant further research and development.
Title: Understanding Asian Stereotyping and Bias in LLMs
Authors: Flora Huang
Affiliations: Stanford University
Year: 2024
Domain: LLMs & NLP

Abstract: In recent years, the advancements of large language models (LLMs) have helped them become increasingly entrenched into our world. LLM assistance with processing natural language tasks and generating text is now commonplace. However, despite their advancements, LLMs are critically flawed due to their internalization of social biases. As a result, the use of LLMs may often spread or even amplify issues such as racial bias and stereotyping. While racial bias in LLMs have been well documented, there exists a critical knowledge gap in understanding what stereotyping or biases against specific ethnic groups are encoded into LLMs. This paper provides a deep dive into understanding the extent and potential impact of Asian stereotyping in popular LLM systems, through the use of experimentation and surveying. Overall, it was discovered that while many users of LLMs have not reported encountering Asian stereotyping and/or bias in their LLM usage, stereotyping and bias are still prevalent in LLM systems. Qualitative studies of LLM responses found significant examples of Asian stereotyping, with the "model minority" stereotype being most common. Additionally, experiments also found that there exists a negative sentiment polarity gap for the Asian ethnic group compared to the average, suggesting that LLMs have encoded negative opinions/beliefs towards Asians. Finally, it was discovered that LLMs tended to assign occupations with higher income levels to people of Asian descent, potentially at the expense of people from other ethnic groups. While Asian stereotyping may not have been directly detected by most users, it remains important to address this issue, especially as LLMs may amplify biases and wield an increasingly large influence on our society, actions, and habits.
Title: Evaluating Interfaced LLM Bias
Authors: Kai-Ching Yeh; Jou-An Chi; Da-Chen Lian; Shu-Kai Hsieh
Affiliations: National Taiwan University; National Taiwan University; National Taiwan University; National Taiwan University
Year: 2023
Domain: LLMs & NLP

Abstract: In this research, we comprehensively analyze the potential biases inherent in Large Language Models, utilizing meticulously curated input data to ascertain the extent to which such data sway machine-generated responses to yield prejudiced outcomes. Notwithstanding recent strides in mitigating bias in LLM-based NLP, our findings underscore the continued susceptibility of these models to data-driven bias. We have integrated the PTT NTU board as our primary data source for this investigation. Moreover, our study elucidates that, in certain contexts, machines may manifest biases without supplementary prompts. However, they can be guided toward rendering impartial responses when provided with enhanced contextual nuances.
Title: Intentional Biases in LLM Responses
Authors: Nicklaus Badyal; Derek Jacoby; Yvonne Coady
Affiliations: University of Victoria; University of Victoria; University of Victoria
Year: 2023
Domain: LLMs & NLP

Abstract: In this study we intentionally introduce biases into large language model responses in an attempt to create specific personas for interactive media purposes. We explore the differences between open source models such as Falcon-7b and the GPT-4 model from Open AI, and we quantity some differences in responses afforded by the two systems. We find that the guardrails in the GPT-4 mixture of experts models with a supervisor, while useful in assuring AI alignment in general, are detrimental in trying to construct personas with a variety of uncommon viewpoints. This study aims to set the groundwork for future exploration in intentional biases of large language models such that these practices can be applied in the creative field, and new forms of media.
Title: Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models
Authors: Shiran Dudy; Thulasi Tholeti; Resmi Ramachandranpillai; Muhammad Ali; Toby Jia-Jun Li; Ricardo Baeza-Yates
Affiliations: Northeastern University; Northeastern University; Northeastern University; Northeastern University; University of Notre Dame; Northeastern University
Year: 2025
Domain: LLMs & NLP

Abstract: This paper focuses on algorithmic bias of machine learning and artificial intelligence applications in healthcare information systems. Based on the quantitative data and qualitative comments from a survey and interviews with healthcare professionals, who have different job roles (e.g., clinical vs. administrative), this study provides findings about the relationships between algorithmic bias, perceived fairness, and the intended acceptance and adoption of ML algorithms and algorithm generated outcomes. The results suggest that the opinions of healthcare professionals toward the causes of algorithmic bias, the criteria of algorithm assessment, the perceived fairness, and bias mitigation approaches may vary depending on their job roles, perspectives, tasks, and the algorithm characteristics. More research is needed to investigate algorithmic bias to ensure fairness and equality in healthcare.
Title: Bias in data-driven artificial intelligence systems—An introductory survey (2nd entry)
Authors: Eirini Ntoutsi et al.
Affiliations: Leibniz University Hannover; Foundation for Research and Technology Hellas; Leibniz University Hannover; Leibniz University Hannover; Leibniz University Hannover; Technische Informationsbibliothek (TIB); University of Pisa; University of Pisa; Information Technologies Institute; Information Technologies Institute; Information Technologies Institute; GESIS - Leibniz-Institute for the Social Sciences; GESIS - Leibniz-Institute for the Social Sciences; GESIS - Leibniz-Institute for the Social Sciences; The Open University; The Open University; Technische Universität Berlin; Leibniz University Hannover; Leibniz University Hannover; Arbeitsgemeinschaft Gynäkologische Onkologie Studiengruppe; Arbeitsgemeinschaft Gynäkologische Onkologie Studiengruppe; University of Southampton; University of Stuttgart
Year: 2020
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Abstract Artificial Intelligence (AI)‐based systems are widely employed nowadays to make decisions that have far‐reaching impact on individuals and society. Their decisions might affect everyone, everywhere, and anytime, entailing concerns about potential human rights issues. Therefore, it is necessary to move beyond traditional AI algorithms optimized for predictive performance and embed ethical and legal principles in their design, training, and deployment to ensure social good while still benefiting from the huge potential of the AI technology. The goal of this survey is to provide a broad multidisciplinary overview of the area of bias in AI systems, focusing on technical challenges and solutions as well as to suggest new research directions towards approaches well‐grounded in a legal frame. In this survey, we focus on data‐driven AI, as a large part of AI is powered nowadays by (big) data and powerful machine learning algorithms. If otherwise not specified, we use the general term bias to describe problems related to the gathering or processing of data that might result in prejudiced decisions on the bases of demographic features such as race, sex, and so forth. This article is categorized under: Commercial, Legal, and Ethical Issues &gt; Fairness in Data Mining Commercial, Legal, and Ethical Issues &gt; Ethical Considerations Commercial, Legal, and Ethical Issues &gt; Legal Issues
Title: Contextual knowledge graph approach to bias-reduced decision support systems
Authors: Guang-Li Huang; Arkady Zaslavsky
Affiliations: Deakin University; Deakin University
Year: 2024
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Recent research shows that automated decision-making systems based on Artificial Intelligence (AI) may lead to perceived unfairness and bias, especially in some sensitive areas.There is no universal debiasing solution for AI applications and systems.This paper proposes a bias-reducing framework based on contextual knowledge graphs for decision-making systems to help analyse and detect potential bias factors during system operation in near-real time.In particular, the contextual knowledge graph is designed to learn the relations between current tasks and corresponding features and explore the correlation among data, context and tasks.Three bias assessment metrics (i.e., label bias, sampling bias and timeliness bias) are proposed to measure, quantify and qualitatively define the bias level for the pre-and post-modelling.The trained model using debiased datasets combines contextual knowledge to support fairer decision-making.Experimental results show that the proposed method is more effective in supporting fairer decision-making than the existing methods.
Title: Detecting and Mitigating Bias in LLMs through Knowledge Graph-Augmented Training
Authors: Rajeev Kumar; Harishankar Kumar; Kumari Shalini
Affiliations: Althire AI; Althire AI; Althire AI
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Large language models have revolutionized natural language processing with their surprising capability to understand and generate human-like text. However, many of these models inherit and further amplify the biases present in their training data, raising ethical and fairness concerns. The detection and mitigation of such biases are vital to ensuring that LLMs act responsibly and equitably across diverse domains. This work investigates Knowledge Graph-Augmented Training (KGAT) as a novel method to mitigate bias in LLM. Using structured domain-specific knowledge from real-world knowledge graphs, we improve the understanding of the model and reduce biased output. Public datasets for bias assessment include Gender Shades, Bias in Bios, and FairFace, while metrics such as demographic parity and equal opportunity facilitate rigorous detection. We also performed targeted mitigation strategies to correct biased associations, leading to a significant drop in biased output and improved bias metrics. Equipped with real-world datasets and knowledge graphs, our framework is both scalable and effective, paving the way toward responsible deployment in sensitive and high-stakes applications.
Title: StereoKG: Data-Driven Knowledge Graph Construction For Cultural Knowledge and Stereotypes
Authors: Awantee Deshpande; Dana Ruiter; Marius Mosbach; Dietrich Klakow
Affiliations: Saarland University; Saarland University; Saarland University; Saarland University
Year: 2022
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Analyzing ethnic or religious bias is important for improving fairness, accountability, and transparency of natural language processing models. However, many techniques rely on human-compiled lists of bias terms, which are expensive to create and are limited in coverage. In this study, we present a fully datadriven pipeline for generating a knowledge graph (KG) of cultural knowledge and stereotypes. Our resulting KG covers 5 religious groups and 5 nationalities and can easily be extended to include more entities. Our human evaluation shows that the majority (59.2%) of non-singleton entries are coherent and complete stereotypes. We further show that performing intermediate masked language model training on the verbalized KG leads to a higher level of cultural awareness in the model and has the potential to increase classification performance on knowledge-crucial samples on a related task, i.e., hate speech detection.
Title: Visual Affect Around the World
Authors: Brendan Jou; Tao Chen; Nikolaos Pappas; Miriam Redi; Mercan Topkara; Shih-Fu Chang
Affiliations: Columbia University; Columbia University; Idiap Research Institute; Yahoo (United Kingdom); Columbia University
Year: 2015
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Every culture and language is unique. Our work expressly focuses on the uniqueness of culture and language in relation to human affect, specifically sentiment and emotion semantics, and how they manifest in social multimedia. We develop sets of sentiment- and emotion-polarized visual concepts by adapting semantic structures called adjective-noun pairs, originally introduced by Borth et al. (2013), but in a multilingual context. We propose a new language-dependent method for automatic discovery of these adjective-noun constructs. We show how this pipeline can be applied on a social multimedia platform for the creation of a large-scale multilingual visual sentiment concept ontology (MVSO). Unlike the flat structure in Borth et al. (2013), our unified ontology is organized hierarchically by multilingual clusters of visually detectable nouns and subclusters of emotionally biased versions of these nouns. In addition, we present an image-based prediction task to show how generalizable language-specific models are in a multilingual context. A new, publicly available dataset of >15.6K sentiment-biased visual concepts across 12 languages with language-specific detector banks, >7.36M images and their metadata is also released.
Title: Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?
Authors: Manas Gaur; Keyur Faldu; Amit Sheth
Affiliations: University of South Carolina; Embibe; University of South Carolina
Year: 2020
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: The recent series of innovations in deep learning (DL) have shown enormous\npotential to impact individuals and society, both positively and negatively.\nThe DL models utilizing massive computing power and enormous datasets have\nsignificantly outperformed prior historical benchmarks on increasingly\ndifficult, well-defined research tasks across technology domains such as\ncomputer vision, natural language processing, signal processing, and\nhuman-computer interactions. However, the Black-Box nature of DL models and\ntheir over-reliance on massive amounts of data condensed into labels and dense\nrepresentations poses challenges for interpretability and explainability of the\nsystem. Furthermore, DLs have not yet been proven in their ability to\neffectively utilize relevant domain knowledge and experience critical to human\nunderstanding. This aspect is missing in early data-focused approaches and\nnecessitated knowledge-infused learning and other strategies to incorporate\ncomputational knowledge. This article demonstrates how knowledge, provided as a\nknowledge graph, is incorporated into DL methods using knowledge-infused\nlearning, which is one of the strategies. We then discuss how this makes a\nfundamental difference in the interpretability and explainability of current\napproaches, and illustrate it with examples from natural language processing\nfor healthcare and education applications.\n
Title: Leveraging Structural Causal Models for Bias Detection and Feature Adjustment in Machine Learning Predictions
Authors: Bern Igoche Igoche; Olumuyiwa Matthew; Peter Bednár; Alexander Gegov
Affiliations: University of Portsmouth; University of Portsmouth; University of Portsmouth; University of Portsmouth
Year: 2024
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: This research addresses the critical issue of fairness bias in machine learning predictions and explanations around the Nigerian Polytechnic admission process. We propose a novel approach that leverages Structural Causal Models (SCMs) to design an ontological framework for detecting bias and adjusting features in Local Interpretable Model-agnostic Explanations (LIME). The SCM ontology provides a principled approach to identify the features that contribute to fairness bias, and we show the presence of bias in LIME explanations for the Polytechnic admission dataset. To mitigate this bias, we propose an ablation technique for feature adjustment in a fair-LIME framework that uses SCM ontology. Experimental results on the Benpoly admission dataset show a remarkable reduction in fairness bias with high fidelity to the predictions of the black-box model. The fair-LIME explanations provided a valid and unbiased interpretation of the factors driving admission decisions. In this regard, the paper significantly extends the scope of eXplainable AI by developing a principled method for obtaining an unbiased explanability aimed at fostering the growth of trustworthy AI systems. The fair-LIME framework holds strong implications towards making the development of transparent, accountable, and ethical AI systems a reality for high-stakes decision-making processes.
Title: Gender Bias Analysis in Commonsense Knowledge Graph Embeddings
Authors: Khine Myat Thwe; Teeradaj Racharak; Le-Minh Nguyen
Affiliations: Japan Advanced Institute of Science and Technology; Japan Advanced Institute of Science and Technology; Japan Advanced Institute of Science and Technology
Year: 2023
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: In this study, we explore the existence of gender bias in ConceptNet triples by employing knowledge graph embedding models, a crucial aspect that has gained prominence in the artificial intelligence domain. Commonsense knowledge graphs, such as ConceptNet, play a vital role in an array of natural language processing tasks; nonetheless, the potential gender bias embedded within these graphs may considerably influence the fairness of associated applications. To tackle this concern, we introduce a comprehensive gender bias analysis framework tailored for ConceptNet triples, with a focus on the link prediction task utilizing knowledge graph embedding models. By evaluating multiple state-of-the-art knowledge graph embedding models, we assess the degree of inherent gender bias across the models. Furthermore, we investigate mitigation strategies to attain a debiased representation of gender within commonsense knowledge graphs. Our results contribute to a broader comprehension of gender bias in knowledge graphs, and the proposed mitigation strategies lay the groundwork for developing more equitable AI systems that use ConceptNet.
Title: Towards Engineering Fair Ontologies: Unbiasing a Surveillance Ontology
Authors: Evangelos Paparidis; Konstantinos Kotis
Affiliations: University of the Aegean; University of the Aegean
Year: 2021
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Capturing knowledge in ontology-based AI applications may significantly propagate technical/statistical, cultural/social, cognitive/psychological, or other types of bias, to un-fair AI models and to their generated decisions. Biased ontologies (and consequently, knowledge graphs) engineered for intelligent surveillance applications can introduce technical barriers in fair capture of offenders, thus it must be researched as a first priority problem and a constant concern for explicit actions to be taken in the era of a more secure and fair world. In this paper we report preliminary research conducted on the novel topic of engineering fair ontologies and present first experiments with a prototype ontology and knowledge graph in the surveillance domain. Engineering fair ontologies is a quite new research topic, thus, the related work is at early stages. Having said that, in this paper we already highlight a recommended methodological approach for unbiasing ontologies, demonstrated in the surveillance domain, and we identify specific key research issues and challenges for further investigation by the ontology engineering community.