cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: A Survey on Fairness in Large Language Models
Authors: Yingji Li; Mengnan Du; Rui Song; Xin Wang; Ying Wang
Affiliations: Jilin University; New Jersey Institute of Technology; Jilin University; Jilin University; Jilin University
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) have shown powerful performance and development prospects and are widely deployed in the real world. However, LLMs can capture social biases from unprocessed training data and propagate the biases to downstream tasks. Unfair LLM systems have undesirable social impacts and potential harms. In this paper, we provide a comprehensive review of related research on fairness in LLMs. Considering the influence of parameter magnitude and training paradigm on research strategy, we divide existing fairness research into oriented to medium-sized LLMs under pre-training and fine-tuning paradigms and oriented to large-sized LLMs under prompting paradigms. First, for medium-sized LLMs, we introduce evaluation metrics and debiasing methods from the perspectives of intrinsic bias and extrinsic bias, respectively. Then, for large-sized LLMs, we introduce recent fairness research, including fairness evaluation, reasons for bias, and debiasing methods. Finally, we discuss and provide insight on the challenges and future directions for the development of fairness in LLMs.
Title: Bias and Fairness in Large Language Models: A Survey
Authors: Isabel O. Gallegos; Ryan A. Rossi; Joe Barrow; Md Mehrab Tanjim; Sungchul Kim; Franck Dernoncourt; Tong Yu; Ruiyi Zhang; Nesreen K. Ahmed
Affiliations: Stanford University; Adobe Systems (United States); Adobe Systems (United States); Adobe Systems (United States); Adobe Systems (United States); Adobe Systems (United States); Adobe Systems (United States); Intel (United States)
Year: 2024
Domain: LLMs & NLP

Abstract: Abstract Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this article, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely, metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs.
Title: Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception
Authors: Luyang Lin; Lingzhi Wang; Jinsong Guo; Kam-Fai Wong
Affiliations: The Chinese University of Hong Kong; Harbin Institute of Technology; University College London; The Chinese University of Hong Kong
Year: 2024
Domain: LLMs & NLP

Abstract: The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems
Title: Systematic Biases in LLM Simulations of Debates
Authors: Amir Taubenfeld; Yaniv Dover; Roi Reichart; Ariel Goldstein
Affiliations: The Hebrew University of Jerusalem; The Hebrew University of Jerusalem; Technion; The Hebrew University of Jerusalem
Year: 2024
Domain: LLMs & NLP

Abstract: The emergence of Large Language Models (LLMs), has opened exciting possibilities for constructing computational simulations designed to replicate human behavior accurately.Current research suggests that LLM-based agents become increasingly human-like in their performance, sparking interest in using these AI agents as substitutes for human participants in behavioral studies.However, LLMs are complex statistical learners without straightforward deductive rules, making them prone to unexpected behaviors.Hence, it is crucial to study and pinpoint the key behavioral distinctions between humans and LLM-based agents.In this study, we highlight the limitations of LLMs in simulating human interactions, particularly focusing on LLMs' ability to simulate political debates on topics that are important aspects of people's day-to-day lives and decision-making processes.Our findings indicate a tendency for LLM agents to conform to the model's inherent social biases despite being directed to debate from certain political perspectives.This tendency results in behavioral patterns that seem to deviate from well-established social dynamics among humans.We reinforce these observations using an automatic self-fine-tuning method, which enables us to manipulate the biases within the LLM and demonstrate that agents subsequently align with the altered biases.These results underscore the need for further research to develop methods that help agents overcome these biases, a critical step toward creating more realistic simulations.
Title: Kelly is a Warm Person Joseph is a Role Model: Gender Biases in LLM-Generated Reference Letters
Authors: Yixin Wan; George Pu; Jiao Sun; Aparna Garimella; Kai-Wei Chang; Nanyun Peng
Affiliations: University of California, Los Angeles; University of California, Los Angeles; University of Southern California; Adobe Research; University of California, Los Angeles; University of California, Los Angeles
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.
Title: Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Authors: Jiayi Ye; Yanbo Wang; Yue Huang; Dongping Chen; Qihui Zhang; Nuno Moniz; Tianpeng Gao; Werner Geyer; Chao Huang; Pin-Yu Chen; Nitesh V. Chawla; Xiangliang Zhang
Affiliations: University of Notre Dame; University of Notre Dame; University of Notre Dame; University of Washington; Peking University; University of Notre Dame; IBM Research; IBM Research; University of Hong Kong; IBM Research; University of Notre Dame; University of Notre Dame
Year: 2024
Domain: LLMs & NLP

Abstract: LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework-CALM-which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
Title: Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era
Authors: Sunhao Dai; Xu Chen; Shicheng Xu; Liang Pang; Zhenhua Dong; Jun Xu
Affiliations: Renmin University of China; Renmin University of China; Chinese Academy of Sciences; Chinese Academy of Sciences; Huawei Technologies (China); Renmin University of China
Year: 2024
Domain: LLMs & NLP

Abstract: With the rapid advancements of large language models (LLMs), information retrieval (IR) systems, such as search engines and recommender systems, have undergone a significant paradigm shift. This evolution, while heralding new opportunities, introduces emerging challenges, particularly in terms of biases and unfairness, which may threaten the information ecosystem. In this paper, we present a comprehensive survey of existing works on emerging and pressing bias and unfairness issues in IR systems when the integration of LLMs. We first unify bias and unfairness issues as distribution mismatch problems, providing a groundwork for categorizing various mitigation strategies through distribution alignment. Subsequently, we systematically delve into the specific bias and unfairness issues arising from three critical stages of LLMs integration into IR systems: data collection, model development, and result evaluation. In doing so, we meticulously review and analyze recent literature, focusing on the definitions, characteristics, and corresponding mitigation strategies associated with these issues. Finally, we identify and highlight some open problems and challenges for future work, aiming to inspire researchers and stakeholders in the IR field and beyond to better understand and mitigate bias and unfairness issues of IR in this LLM era. We also consistently maintain a GitHub repository for the relevant papers and resources in this rising direction at https://github.com/KID-22/LLM-IR-Bias-Fairness-Survey.
Title: Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation
Authors: Xiangjue Dong; Yibo Wang; Philip S. Yu; James Caverlee
Affiliations: Texas A&M University; University of Illinois Chicago; University of Illinois Chicago; Texas A&M University
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) can generate biased and toxic responses. Yet most prior work on LLM gender bias evaluation requires predefined gender-related phrases or gender stereotypes, which are challenging to be comprehensively collected and are limited to explicit bias evaluation. In addition, we believe that instances devoid of gender-related language or explicit stereotypes in inputs can still induce gender bias in LLMs. Thus, in this work, we propose a conditional text generation mechanism without the need for predefined gender phrases and stereotypes. This approach employs three types of inputs generated through three distinct strategies to probe LLMs, aiming to show evidence of explicit and implicit gender biases in LLMs. We also utilize explicit and implicit evaluation metrics to evaluate gender bias in LLMs under different strategies. Our experiments demonstrate that an increased model size does not consistently lead to enhanced fairness and all tested LLMs exhibit explicit and/or implicit gender bias, even when explicit gender stereotypes are absent in the inputs.
Title: On the Independence of Association Bias and Empirical Fairness in Language Models
Authors: Laura Cabello; Anna Jørgensen; Anders Søgaard
Affiliations: University of Copenhagen; University of Copenhagen; University of Copenhagen
Year: 2023
Domain: LLMs & NLP

Abstract: The societal impact of pre-trained language models has prompted researchers to probe them for strong associations between protected attributes and value-loaded terms, from slur to prestigious job titles. Such work is said to probe models for bias or fairness-or such probes 'into representational biases' are said to be 'motivated by fairness'-suggesting an intimate connection between bias and fairness. We provide conceptual clarity by distinguishing between association biases (Caliskan et al., 2022) and empirical fairness (Shen et al., 2022) and show the two can be independent. Our main contribution, however, is showing why this should not come as a surprise. To this end, we first provide a thought experiment, showing how association bias and empirical fairness can be completely orthogonal. Next, we provide empirical evidence that there is no correlation between bias metrics and fairness metrics across the most widely used language models. Finally, we survey the sociological and psychological literature and show how this literature provides ample support for expecting these metrics to be uncorrelated.
Title: Fairness in Large Language Models: A Taxonomic Survey
Authors: Zhibo Chu; Zichong Wang; Wenbin Zhang
Affiliations: Florida International University; Florida International University; Florida International University
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) have demonstrated remarkable success across various domains. However, despite their promising performance in numerous real-world applications, most of these algorithms lack fairness considerations. Consequently, they may lead to discriminatory outcomes against certain communities, particularly marginalized populations, prompting extensive study in fair LLMs. On the other hand, fairness in LLMs, in contrast to fairness in traditional machine learning, entails exclusive backgrounds, taxonomies, and fulfillment techniques. To this end, this survey presents a comprehensive overview of recent advances in the existing literature concerning fair LLMs. Specifically, a brief introduction to LLMs is provided, followed by an analysis of factors contributing to bias in LLMs. Additionally, the concept of fairness in LLMs is discussed categorically, summarizing metrics for evaluating bias in LLMs and existing algorithms for promoting fairness. Furthermore, resources for evaluating bias in LLMs, including toolkits and datasets, are summarized. Finally, existing research challenges and open questions are discussed.
Title: Survey on Factuality in Large Language Models: Knowledge Retrieval and Domain-Specificity
Authors: Cunxiang Wang; Xiaoze Liu; Yuanhao Yue; Xiangru Tang; Tianhang Zhang; Jay J. Cheng; Yunzhi Yao; Wen-Yang Gao; Xuming Hu; Zehan Qi; Yidong Wang; Linyi Yang; Jindong Wang; Xing Xie; Zheng Zhang; Yue Zhang
Affiliations: Westlake University; Purdue University; Fudan University; Yale University; Shanghai Jiao Tong University; The Hong Kong University of Science and Technology; Zhejiang University; Tsinghua University; Tsinghua University; Westlake University; Westlake University; Microsoft Research; Microsoft Research; Amazon AWS AI; Westlake University; Westlake University
Year: 2024
Domain: LLMs & NLP

Abstract: This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital. We define the Factuality Issue as the probability of LLMs to produce content inconsistent with established facts. We first delve into the implications of these inaccuracies, highlighting the potential consequences and challenges posed by factual errors in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs store and process facts, seeking the primary causes of factual errors. Our discussion then transitions to methodologies for evaluating LLM factuality, emphasizing key metrics, benchmarks, and studies. We further explore strategies for enhancing LLM factuality, including approaches tailored for specific domains. We focus two primary LLM configurations standalone LLMs and Retrieval-Augmented LLMs that utilizes external data, we detail their unique challenges and potential enhancements. Our survey offers a structured guide for researchers aiming to fortify the factual reliability of LLMs.
Title: A Survey of Large Language Models in Medicine: Progress Application and Challenge
Authors: Hongjian Zhou; Fenglin Liu; Boyang Gu; Xinyu Zou; Jinfa Huang; Jinge Wu; Yiru Li; Sam Chen; Peilin Zhou; Junling Liu; Yining Hua; Chengfeng Mao; Chenyu You; Xian Wu; Yefeng Zheng; Lei Clifton; Zheng Li; Jiebo Luo; David Clifton
Affiliations: University of Oxford; University of Oxford; Imperial College London; University of Waterloo; University of Rochester; University College London; Western University; University of Georgia; HKUST (Guangzhou); Alibaba; Harvard University; Massachusetts Institute of Technology; Yale University; Tencent; Tencent; University of Oxford; Amazon; University of Rochester; University of Oxford
Year: 2023
Domain: LLMs & NLP

Abstract: Large language models (LLMs), such as ChatGPT, have received substantial attention due to their capabilities for understanding and generating human language. While there has been a burgeoning trend in research focusing on the employment of LLMs in supporting different medical tasks (e.g., enhancing clinical diagnostics and providing medical education), a review of these efforts, particularly their development, practical applications, and outcomes in medicine, remains scarce. Therefore, this review aims to provide a detailed overview of the development and deployment of LLMs in medicine, including the challenges and opportunities they face. In terms of development, we provide a detailed introduction to the principles of existing medical LLMs, including their basic model structures, number of parameters, and sources and scales of data used for model development. It serves as a guide for practitioners in developing medical LLMs tailored to their specific needs. In terms of deployment, we offer a comparison of the performance of different LLMs across various medical tasks, and further compare them with state-of-the-art lightweight models, aiming to provide an understanding of the advantages and limitations of LLMs in medicine. Overall, in this review, we address the following questions: 1) What are the practices for developing medical LLMs 2) How to measure the medical task performance of LLMs in a medical setting? 3) How have medical LLMs been employed in real-world practice? 4) What challenges arise from the use of medical LLMs? and 5) How to more effectively develop and deploy medical LLMs? By answering these questions, this review aims to provide insights into the opportunities for LLMs in medicine and serve as a practical resource. We also maintain a regularly updated list of practical guides on medical LLMs at https://github.com/AI-in-Health/MedLLMsPracticalGuide
Title: Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI
Authors: Muhammad Aurangzeb Ahmad; Ilker Yaramis; Taposh Dutta Roy
Affiliations: University of Washington Bothell; Kaiser Permanente; Kaiser Permanente
Year: 2023
Domain: LLMs & NLP

Abstract: Large language models have proliferated across multiple domains in as short period of time. There is however hesitation in the medical and healthcare domain towards their adoption because of issues like factuality, coherence, and hallucinations. Give the high stakes nature of healthcare, many researchers have even cautioned against its usage until these issues are resolved. The key to the implementation and deployment of LLMs in healthcare is to make these models trustworthy, transparent (as much possible) and explainable. In this paper we describe the key elements in creating reliable, trustworthy, and unbiased models as a necessary condition for their adoption in healthcare. Specifically we focus on the quantification, validation, and mitigation of hallucinations in the context in healthcare. Lastly, we discuss how the future of LLMs in healthcare may look like.