cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Algorithmic fairness in artificial intelligence for medicine and healthcare
Authors: Richard J. Chen; Judy J. Wang; Drew F. K. Williamson; Tiffany Chen; Jana Lipková; Ming Y. Lu; Sharifa Sahai; Faisal Mahmood
Affiliations: Broad Institute; Brigham and Women's Hospital; Broad Institute; Broad Institute; Broad Institute; Broad Institute; Broad Institute; Broad Institute
Year: 2023
Domain: Health & Clinical AI

Abstract: Large Language Models (LLMs) are increasingly deployed in medical applications. However, these systems can exhibit biases related to gender, race and professional role, raising significant concerns about their impact on healthcare equity. We propose a systematic framework for evaluating social biases in medical question answering using LLMs. By varying only the role information in prompts across three medical focused subsets of the MMLU benchmark, including College Medicine, Medical Genetics and Professional Medicine, we evaluate multiple LLMs’ performance and quantify bias gaps. Our results highlight the necessity of rigorous bias assessment in medical AI and provide a practical framework for measuring disparities across diverse role dimensions prior to clinical deployment.
Title: Algorithmic fairness and bias mitigation for clinical machine learning with deep reinforcement learning
Authors: Jenny Yang; Andrew A. S. Soltan; David W. Eyre; David A. Clifton
Affiliations: University of Oxford; John Radcliffe Hospital; Open Data Institute; University of Oxford
Year: 2023
Domain: Health & Clinical AI

Abstract: As models based on machine learning continue to be developed for healthcare applications, greater effort is needed to ensure that these technologies do not reflect or exacerbate any unwanted or discriminatory biases that may be present in the data. Here we introduce a reinforcement learning framework capable of mitigating biases that may have been acquired during data collection. In particular, we evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments and aimed to mitigate any site (hospital)-specific and ethnicity-based biases present in the data. Using a specialized reward function and training procedure, we show that our method achieves clinically effective screening performances, while significantly improving outcome fairness compared with current benchmarks and state-of-the-art machine learning methods. We performed external validation across three independent hospitals, and additionally tested our method on a patient intensive care unit discharge status task, demonstrating model generalizability.
Title: Fairness of artificial intelligence in healthcare: review and recommendations
Authors: Daiju Ueda; Taichi Kakinuma; Shohei Fujita; Koji Kamagata; Yasutaka Fushimi; Rintaro Ito; Yusuke Matsui; Taiki Nozaki; Takeshi Nakaura; Noriyuki Fujima; Fuminari Tatsugami; Masahiro Yanagawa; Kenji Hirata; Akira Yamada; Takahiro Tsuboyama; Mariko Kawamura; Tomoyuki Fujioka; Shinji Naganawa
Affiliations: Osaka City University Hospital; The University of Tokyo; Juntendo University; Kyoto University; Nagoya University; Okayama University; Keio University; Kumamoto University; Hokkaido University Hospital; Hiroshima University; The University of Osaka; Hokkaido University; Shinshu University; The University of Osaka; Nagoya University; Tokyo Medical and Dental University; Nagoya University
Year: 2023
Domain: Health & Clinical AI

Abstract: In this review, we address the issue of fairness in the clinical integration of artificial intelligence (AI) in the medical field. As the clinical adoption of deep learning algorithms, a subfield of AI, progresses, concerns have arisen regarding the impact of AI biases and discrimination on patient health. This review aims to provide a comprehensive overview of concerns associated with AI fairness; discuss strategies to mitigate AI biases; and emphasize the need for cooperation among physicians, AI researchers, AI developers, policymakers, and patients to ensure equitable AI integration. First, we define and introduce the concept of fairness in AI applications in healthcare and radiology, emphasizing the benefits and challenges of incorporating AI into clinical practice. Next, we delve into concerns regarding fairness in healthcare, addressing the various causes of biases in AI and potential concerns such as misdiagnosis, unequal access to treatment, and ethical considerations. We then outline strategies for addressing fairness, such as the importance of diverse and representative data and algorithm audits. Additionally, we discuss ethical and legal considerations such as data privacy, responsibility, accountability, transparency, and explainability in AI. Finally, we present the Fairness of Artificial Intelligence Recommendations in healthcare (FAIR) statement to offer best practices. Through these efforts, we aim to provide a foundation for discussing the responsible and equitable implementation and deployment of AI in healthcare.
Title: Bias in artificial intelligence algorithms and recommendations for mitigation
Authors: Lama Nazer; Razan Zatarah; Shai Waldrip; Janny Xue Chen Ke; Mira Moukheiber; Ashish K. Khanna; Rachel Hicklen; Lama Moukheiber; Dana Moukheiber; Haobo Ma; Piyush Mathur
Affiliations: King Hussein Cancer Center; King Hussein Cancer Center; Morehouse School of Medicine; Dalhousie University; Massachusetts Institute of Technology; Atrium Health Wake Forest Baptist; The University of Texas MD Anderson Cancer Center; Massachusetts Institute of Technology; Massachusetts Institute of Technology; Beth Israel Deaconess Medical Center; Cleveland Clinic
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The adoption of artificial intelligence (AI) algorithms is rapidly increasing in healthcare. Such algorithms may be shaped by various factors such as social determinants of health that can influence health outcomes. While AI algorithms have been proposed as a tool to expand the reach of quality healthcare to underserved communities and improve health equity, recent literature has raised concerns about the propagation of biases and healthcare disparities through implementation of these algorithms. Thus, it is critical to understand the sources of bias inherent in AI-based algorithms. This review aims to highlight the potential sources of bias within each step of developing AI algorithms in healthcare, starting from framing the problem, data collection, preprocessing, development, and validation, as well as their full implementation. For each of these steps, we also discuss strategies to mitigate the bias and disparities. A checklist was developed with recommendations for reducing bias during the development and implementation stages. It is important for developers and users of AI-based algorithms to keep these important considerations in mind to advance health equity for all populations.
Title: A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics
Authors: Kai He; Rui Mao; Qika Lin; Yucheng Ruan; Lan, Xiang; Mengling Feng; Erik Cambria
Affiliations: National University of Singapore; Nanyang Technological University; National University of Singapore; National University of Singapore; National University of Singapore; National University of Singapore; Nanyang Technological University
Year: 2023
Domain: Health & Clinical AI

Abstract: The utilization of large language models (LLMs) in the Healthcare domain has generated both excitement and concern due to their ability to effectively respond to freetext queries with certain professional knowledge. This survey outlines the capabilities of the currently developed LLMs for Healthcare and explicates their development process, with the aim of providing an overview of the development roadmap from traditional Pretrained Language Models (PLMs) to LLMs. Specifically, we first explore the potential of LLMs to enhance the efficiency and effectiveness of various Healthcare applications highlighting both the strengths and limitations. Secondly, we conduct a comparison between the previous PLMs and the latest LLMs, as well as comparing various LLMs with each other. Then we summarize related Healthcare training data, training methods, optimization strategies, and usage. Finally, the unique concerns associated with deploying LLMs in Healthcare settings are investigated, particularly regarding fairness, accountability, transparency and ethics. Our survey provide a comprehensive investigation from perspectives of both computer science and Healthcare specialty. Besides the discussion about Healthcare concerns, we supports the computer science community by compiling a collection of open source resources, such as accessible datasets, the latest methodologies, code implementations, and evaluation benchmarks in the Github. Summarily, we contend that a significant paradigm shift is underway, transitioning from PLMs to LLMs. This shift encompasses a move from discriminative AI approaches to generative AI approaches, as well as a shift from model-centered methodologies to data-centered methodologies. Also, we determine that the biggest obstacle of using LLMs in Healthcare are fairness, accountability, transparency and ethics.
Title: “Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters
Authors: Yixin Wan; George Pu; Jiao Sun; Aparna Garimella; Kai-Wei Chang; Nanyun Peng
Affiliations: University of California, Los Angeles; University of California, Los Angeles; Southern California University for Professional Studies; Adobe Systems (United States); University of California, Los Angeles; University of California, Los Angeles
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.
Title: Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers
Authors: Catherine A. Gao; Frederick M. Howard; Nikolay S. Markov; Emma Dyer; Siddhi Ramesh; Yuan Luo; Alexander T. Pearson
Affiliations: Northwestern University; University of Chicago; Northwestern University; University of Chicago; University of Chicago; Northwestern University; University of Chicago
Year: 2023
Domain: LLMs & NLP

Abstract: Large language models such as ChatGPT can produce increasingly realistic text, with unknown information on the accuracy and integrity of using these models in scientific writing. We gathered fifth research abstracts from five high-impact factor medical journals and asked ChatGPT to generate research abstracts based on their titles and journals. Most generated abstracts were detected using an AI output detector, 'GPT-2 Output Detector', with % 'fake' scores (higher meaning more likely to be generated) of median [interquartile range] of 99.98% 'fake' [12.73%, 99.98%] compared with median 0.02% [IQR 0.02%, 0.09%] for the original abstracts. The AUROC of the AI output detector was 0.94. Generated abstracts scored lower than original abstracts when run through a plagiarism detector website and iThenticate (higher scores meaning more matching text found). When given a mixture of original and general abstracts, blinded human reviewers correctly identified 68% of generated abstracts as being generated by ChatGPT, but incorrectly identified 14% of original abstracts as being generated. Reviewers indicated that it was surprisingly difficult to differentiate between the two, though abstracts they suspected were generated were vaguer and more formulaic. ChatGPT writes believable scientific abstracts, though with completely generated data. Depending on publisher-specific guidelines, AI output detectors may serve as an editorial tool to help maintain scientific standards. The boundaries of ethical and acceptable use of large language models to help scientific writing are still being discussed, and different journals and conferences are adopting varying policies.
Title: Personalized Prompt Learning for Explainable Recommendation
Authors: Lei Li; Yongfeng Zhang; Li Chen
Affiliations: Hong Kong Baptist University; Rutgers, The State University of New Jersey; Hong Kong Baptist University
Year: 2023
Domain: Recommender Systems

Abstract: Providing user-understandable explanations to justify recommendations could help users better understand the recommended items, increase the system’s ease of use, and gain users’ trust. A typical approach to realize it is natural language generation. However, previous works mostly adopt recurrent neural networks to meet the ends, leaving the potentially more effective pre-trained Transformer models under-explored. In fact, user and item IDs, as important identifiers in recommender systems, are inherently in different semantic space as words that pre-trained models were already trained on. Thus, how to effectively fuse IDs into such models becomes a critical issue. Inspired by recent advancement in prompt learning, we come up with two solutions: find alternative words to represent IDs (called discrete prompt learning) and directly input ID vectors to a pre-trained model (termed continuous prompt learning). In the latter case, ID vectors are randomly initialized but the model is trained in advance on large corpora, so they are actually in different learning stages. To bridge the gap, we further propose two training strategies: sequential tuning and recommendation as regularization. Extensive experiments show that our continuous prompt learning approach equipped with the training strategies consistently outperforms strong baselines on three datasets of explainable recommendation.
Title: Contrastive Self-supervised Learning in Recommender Systems: A Survey
Authors: Mengyuan Jing; Yanmin Zhu; Tianzi Zang; Ke Wang
Affiliations: Shanghai Jiao Tong University; Shanghai Jiao Tong University; Shanghai Jiao Tong University; Shanghai Jiao Tong University
Year: 2023
Domain: Recommender Systems

Abstract: Deep learning-based recommender systems have achieved remarkable success in recent years. However, these methods usually heavily rely on labeled data (i.e., user-item interactions), suffering from problems such as data sparsity and cold-start. Self-supervised learning, an emerging paradigm that extracts information from unlabeled data, provides insights into addressing these problems. Specifically, contrastive self-supervised learning, due to its flexibility and promising performance, has attracted considerable interest and recently become a dominant branch in self-supervised learning-based recommendation methods. In this survey, we provide an up-to-date and comprehensive review of current contrastive self-supervised learning-based recommendation methods. Firstly, we propose a unified framework for these methods. We then introduce a taxonomy based on the key components of the framework, including view generation strategy, contrastive task, and contrastive objective. For each component, we provide detailed descriptions and discussions to guide the choice of the appropriate method. Finally, we outline open issues and promising directions for future research.
Title: ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models
Authors: Yikang Liu; Ziyin Zhang; Wanyang Zhang; Shisen Yue; Xiaojing Zhao; Xinyuan Cheng; Yiwen Zhang; Hai Hu
Affiliations: Shanghai Jiao Tong University; Shanghai Jiao Tong University; Peking University; Shanghai Jiao Tong University; Shanghai Jiao Tong University; Shanghai Jiao Tong University; Amazon; Shanghai Jiao Tong University
Year: 2023
Domain: LLMs & NLP

Abstract: AI generated content (AIGC) presents considerable challenge to educators around the world. Instructors need to be able to detect such text generated by large language models, either with the naked eye or with the help of some tools. There is also growing need to understand the lexical, syntactic and stylistic features of AIGC. To address these challenges in English language teaching, we first present ArguGPT, a balanced corpus of 4,038 argumentative essays generated by 7 GPT models in response to essay prompts from three sources: (1) in-class or homework exercises, (2) TOEFL and (3) GRE writing tasks. Machine-generated texts are paired with roughly equal number of human-written essays with three score levels matched in essay prompts. We then hire English instructors to distinguish machine essays from human ones. Results show that when first exposed to machine-generated essays, the instructors only have an accuracy of 61% in detecting them. But the number rises to 67% after one round of minimal self-training. Next, we perform linguistic analyses of these essays, which show that machines produce sentences with more complex syntactic structures while human essays tend to be lexically more complex. Finally, we test existing AIGC detectors and build our own detectors using SVMs and RoBERTa. Results suggest that a RoBERTa fine-tuned with the training set of ArguGPT achieves above 90% accuracy in both essay- and sentence-level classification. To the best of our knowledge, this is the first comprehensive analysis of argumentative essays produced by generative large language models. Machine-authored essays in ArguGPT and our models will be made publicly available at https://github.com/huhailinguist/ArguGPT
Title: Generative Visual Question Answering
Authors: Ethan Shen; Scotty Singh; Bhavesh Kumar
Affiliations: University of Washington; University of Washington; University of Washington
Year: 2023
Domain: LLMs & NLP

Abstract: Multi-modal tasks involving vision and language in deep learning continue to rise in popularity and are leading to the development of newer models that can generalize beyond the extent of their training data. The current models lack temporal generalization which enables models to adapt to changes in future data. This paper discusses a viable approach to creating an advanced Visual Question Answering (VQA) model which can produce successful results on temporal generalization. We propose a new data set, GenVQA, utilizing images and captions from the VQAv2 and MS-COCO dataset to generate new images through stable diffusion. This augmented dataset is then used to test a combination of seven baseline and cutting edge VQA models. Performance evaluation focuses on questions mirroring the original VQAv2 dataset, with the answers having been adjusted to the new images. This paper's purpose is to investigate the robustness of several successful VQA models to assess their performance on future data distributions. Model architectures are analyzed to identify common stylistic choices that improve generalization under temporal distribution shifts. This research highlights the importance of creating a large-scale future shifted dataset. This data can enhance the robustness of VQA models, allowing their future peers to have improved ability to adapt to temporal distribution shifts.
Title: Survey on Machine Learning Biases and Mitigation Techniques
Authors: Sunzida Siddique; Mohd Ariful Haque; Roy George; Kishor Datta Gupta; Debashis Gupta; Md Jobair Hossain Faruk
Affiliations: Daffodil International University; Clark Atlanta University; Clark Atlanta University; Clark Atlanta University; Wake Forest University; New York Institute of Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning (ML) has become increasingly prevalent in various domains. However, ML algorithms sometimes give unfair outcomes and discrimination against certain groups. Thereby, bias occurs when our results produce a decision that is systematically incorrect. At various phases of the ML pipeline, such as data collection, pre-processing, model selection, and evaluation, these biases appear. Bias reduction methods for ML have been suggested using a variety of techniques. By changing the data or the model itself, adding more fairness constraints, or both, these methods try to lessen bias. The best technique relies on the particular context and application because each technique has advantages and disadvantages. Therefore, in this paper, we present a comprehensive survey of bias mitigation techniques in machine learning (ML) with a focus on in-depth exploration of methods, including adversarial training. We examine the diverse types of bias that can afflict ML systems, elucidate current research trends, and address future challenges. Our discussion encompasses a detailed analysis of pre-processing, in-processing, and post-processing methods, including their respective pros and cons. Moreover, we go beyond qualitative assessments by quantifying the strategies for bias reduction and providing empirical evidence and performance metrics. This paper serves as an invaluable resource for researchers, practitioners, and policymakers seeking to navigate the intricate landscape of bias in ML, offering both a profound understanding of the issue and actionable insights for responsible and effective bias mitigation.
Title: Diversity-Promoting Ensemble for Medical Image Segmentation
Authors: Mariana-Iuliana Georgescu; Radu Tudor Ionescu; Andreea Iuliana Miron
Affiliations: University of Bucharest; University of Bucharest; Clinical Emergency Hospital Bucharest
Year: 2023
Domain: Health & Clinical AI

Abstract: Medical image segmentation is an actively studied task in medical imaging, where the precision of the annotations is of utter importance towards accurate diagnosis and treatment. In recent years, the task has been approached with various deep learning systems, among the most popular models being U-Net. In this work, we propose a novel strategy to generate ensembles of different architectures for medical image segmentation, by leveraging the diversity (decorrelation) of the models forming the ensemble. More specifically, we utilize the Dice score among model pairs to estimate the correlation between the outputs of the two models forming each pair. To promote diversity, we select models with low Dice scores among each other. We carry out gastro-intestinal tract image segmentation experiments to compare our diversity-promoting ensemble (DiPE) with another strategy to create ensembles based on selecting the top scoring U-Net models. Our empirical results show that DiPE surpasses both individual models as well as the ensemble creation strategy based on selecting the top scoring models.