cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Diagnosing bias in data-driven algorithms for healthcare
Authors: Jenna Wiens; W. Nicholson Price; Michael W. Sjoding
Affiliations: University of Michigan; University of Michigan; University of Michigan
Year: 2020
Domain: Health & Clinical AI

Abstract: A recent analysis highlighting the potential for algorithms to perpetuate existing racial biases in healthcare underscores the importance of thinking carefully about the labels used during algorithm development.
Title: Artificial intelligence and algorithmic bias: implications for health systems
Authors: Trishan Panch; Heather Mattie; Rifat Atun
Affiliations: Harvard University; Harvard University; Harvard Global Health Institute
Year: 2019
Domain: Health & Clinical AI

Abstract: Algorithmic bias occurs when the application of an algorithm compounds existing inequities in socioeconomic status, race, ethnic background, religion, gender, disability or sexual orientation to amplify them and adversely impact equity in health systems.
Title: Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications
Authors: Farah Magrabi; Elske Ammenwerth; Jytte Brender McNair; N. F. de Keizer; Hannele Hyppönen; Pirkko Nykänen; Michael Rigby; Philip Scott; Tuulikki Vehko; Zoie Shui-Yee Wong; Andrew Georgiou
Affiliations: Macquarie University; UMIT - Private Universität für Gesundheitswissenschaften, Medizinische Informatik und Technik; Aalborg University; Amsterdam University Medical Centers; Finnish Institute for Health and Welfare; Tampere University; Keele University; University of Portsmouth; Finnish Institute for Health and Welfare; St. Luke's International University; Macquarie University
Year: 2019
Domain: Health & Clinical AI

Abstract: OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.
Title: Artificial intelligence bias in medical system designs: a systematic review
Authors: Ashish Kumar; Vivekanand Aelgani; Rubeena Vohra; Suneet Kumar Gupta; Mrinalini Bhagawati; Sudip Paul; Luca Saba; Neha Suri; Narendra N. Khanna; John R. Laird; Amer M. Johri; Manudeep Kalra; Mostafa M. Fouda; Mostafa Fatemi; Subbaram Naidu; Jasjit S. Suri
Affiliations: Bennett University; CMR University; Bharati Vidyapeeth Deemed University; Bennett University; North Eastern Hill University; North Eastern Hill University; University of Cagliari; ; ; St. Helena Hospital; Queen's University; Massachusetts General Hospital; Idaho State University; Mayo Clinic; University of Minnesota, Duluth;
Year: 2023
Domain: Health & Clinical AI

Abstract: Fairness emerged as an important requirement to guarantee that Machine Learning (ML) predictive systems do not discriminate against specific individuals or entire sub-populations, in particular, minorities. Given the inherent subjectivity of viewing the concept of fairness, several notions of fairness have been introduced in the literature. This paper is a survey that illustrates the subtleties between fairness notions through a large number of examples and scenarios. In addition, unlike other surveys in the literature, it addresses the question of “which notion of fairness is most suited to a given real-world scenario and why?”. Our attempt to answer this question consists in (1) identifying the set of fairness-related characteristics of the real-world scenario at hand, (2) analyzing the behavior of each fairness notion, and then (3) fitting these two elements to recommend the most suitable fairness notion in every specific setup. The results are summarized in a decision diagram that can be used by practitioners and policy makers to navigate the relatively large catalogue of ML fairness notions.
Title: Inherent Bias in Artificial Intelligence-Based Decision Support Systems for Healthcare
Authors: Varadraj P. Gurupur; Thomas T. H. Wan
Affiliations: University of Central Florida; University of Central Florida
Year: 2020
Domain: Health & Clinical AI

Abstract: The objective of this article is to discuss the inherent bias involved with artificial intelligence-based decision support systems for healthcare. In this article, the authors describe some relevant work published in this area. A proposed overview of solutions is also presented. The authors believe that the information presented in this article will enhance the readers' understanding of this inherent bias and add to the discussion on this topic. Finally, the authors discuss an overview of the need to implement transdisciplinary solutions that can be used to mitigate this bias.
Title: Algorithmic bias in data-driven innovation in the age of AI
Authors: Shahriar Akter; Grace McCarthy; Shahriar Sajib; Katina Michael; Yogesh K. Dwivedi; John D'Ambra; Kathy Ning Shen
Affiliations: University of Wollongong; University of Wollongong; University of Technology Sydney; Arizona State University; Swansea University; UNSW Sydney; United Arab Emirates University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Until now, much of the work on machine learning and health has focused on processes inside the hospital or clinic. However, this represents only a narrow set of tasks and challenges related to health; there is greater potential for impact by leveraging machine learning in health tasks more broadly. In this Perspective we aim to highlight potential opportunities and challenges for machine learning within a holistic view of health and its influences. To do so, we build on research in population and public health that focuses on the mechanisms between different cultural, social and environmental factors and their effect on the health of individuals and communities. We present a brief introduction to research in these fields, data sources and types of tasks, and use these to identify settings where machine learning is relevant and can contribute to new knowledge. Given the key foci of health equity and disparities within public and population health, we juxtapose these topics with the machine learning subfield of algorithmic fairness to highlight specific opportunities where machine learning, public and population health may synergize to achieve health equity.
Title: Fairness Constraints: Mechanisms for Fair Classification
Authors: Muhammad Bilal Zafar; Isabel Valera; Manuel Gomez-Rodriguez; Krishna P. Gummadi
Affiliations: Max Planck Institute for Software Systems; Max Planck Institute for Software Systems; Max Planck Institute for Software Systems; Max Planck Institute for Software Systems
Year: 2015
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic decision making systems are ubiquitous across a wide variety of online as well as offline services. These systems rely on complex learning methods and vast amounts of data to optimize the service functionality, satisfaction of the end user and profitability. However, there is a growing concern that these automated decisions can lead, even in the absence of intent, to a lack of fairness, i.e., their outcomes can disproportionately hurt (or, benefit) particular groups of people sharing one or more sensitive attributes (e.g., race, sex). In this paper, we introduce a flexible mechanism to design fair classifiers by leveraging a novel intuitive measure of decision boundary (un)fairness. We instantiate this mechanism with two well-known classifiers, logistic regression and support vector machines, and show on real-world data that our mechanism allows for a fine-grained control on the degree of fairness, often at a small cost in terms of accuracy.
Title: Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey
Authors: Max Hort; Zhenpeng Chen; Jie M. Zhang; Federica Sarro; Mark Harman
Affiliations: Simula Research Laboratory; University College London; King’s College London; University College London; University College London
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: This paper provides a comprehensive survey of bias mitigation methods for achieving fairness in Machine Learning (ML) models. We collect a total of 341 publications concerning bias mitigation for ML classifiers. These methods can be distinguished based on their intervention procedure (i.e., pre-processing, in-processing, post-processing) and the technique they apply. We investigate how existing bias mitigation methods are evaluated in the literature. In particular, we consider datasets, metrics and benchmarking. Based on the gathered insights (e.g., What is the most popular fairness metric? How many datasets are used for evaluating bias mitigation methods?), we hope to support practitioners in making informed choices when developing and evaluating new bias mitigation methods.
Title: Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources Impacts And Mitigation Strategies
Authors: Emilio Ferrara
Affiliations: University of Southern California
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The significant advancements in applying artificial intelligence (AI) to healthcare decision-making, medical diagnosis, and other domains have simultaneously raised concerns about the fairness and bias of AI systems. This is particularly critical in areas like healthcare, employment, criminal justice, credit scoring, and increasingly, in generative AI models (GenAI) that produce synthetic media. Such systems can lead to unfair outcomes and perpetuate existing inequalities, including generative biases that affect the representation of individuals in synthetic data. This survey study offers a succinct, comprehensive overview of fairness and bias in AI, addressing their sources, impacts, and mitigation strategies. We review sources of bias, such as data, algorithm, and human decision biases—highlighting the emergent issue of generative AI bias, where models may reproduce and amplify societal stereotypes. We assess the societal impact of biased AI systems, focusing on perpetuating inequalities and reinforcing harmful stereotypes, especially as generative AI becomes more prevalent in creating content that influences public perception. We explore various proposed mitigation strategies, discuss the ethical considerations of their implementation, and emphasize the need for interdisciplinary collaboration to ensure effectiveness. Through a systematic literature review spanning multiple academic disciplines, we present definitions of AI bias and its different types, including a detailed look at generative AI bias. We discuss the negative impacts of AI bias on individuals and society and provide an overview of current approaches to mitigate AI bias, including data pre-processing, model selection, and post-processing. We emphasize the unique challenges presented by generative AI models and the importance of strategies specifically tailored to address these. Addressing bias in AI requires a holistic approach involving diverse and representative datasets, enhanced transparency and accountability in AI systems, and the exploration of alternative AI paradigms that prioritize fairness and ethical considerations. This survey contributes to the ongoing discussion on developing fair and unbiased AI systems by providing an overview of the sources, impacts, and mitigation strategies related to AI bias, with a particular focus on the emerging field of generative AI.
Title: Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency
Authors: Joachim Baumann; Anikó Hannák; Christoph Heitz
Affiliations: University of Zurich; Zurich University of Applied Sciences; Zurich University of Applied Sciences
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Binary decision making classifiers are not fair by default. Fairness requirements are an additional element to the decision making rationale, which is typically driven by maximizing some utility function. In that sense, algorithmic fairness can be formulated as a constrained optimization problem. This paper contributes to the discussion on how to implement fairness, focusing on the fairness concepts of positive predictive value (PPV) parity, false omission rate (FOR) parity, and sufficiency (which combines the former two). We show that group-specific threshold rules are optimal for PPV parity and FOR parity, similar to well-known results for other group fairness criteria. However, depending on the underlying population distributions and the utility function, we find that sometimes an upper-bound threshold rule for one group is optimal: utility maximization under PPV parity (or FOR parity) might thus lead to selecting the individuals with the smallest utility for one group, instead of selecting the most promising individuals. This result is counter-intuitive and in contrast to the analogous solutions for statistical parity and equality of opportunity. We also provide a solution for the optimal decision rules satisfying the fairness constraint sufficiency. We show that more complex decision rules are required and that this leads to within-group unfairness for all but one of the groups. We illustrate our findings based on simulated and real data.
Title: Semantic Web technologies and bias in artificial intelligence: A systematic literature review
Authors: Paula Reyero Lobo; Enrico Daga; Harith Alani; Miriam Fernández
Affiliations: The Open University; Open Knowledge (UK); Open Knowledge (UK); Open Knowledge (UK)
Year: 2023
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Bias in Artificial Intelligence (AI) is a critical and timely issue due to its sociological, economic and legal impact, as decisions made by biased algorithms could lead to unfair treatment of specific individuals or groups. Multiple surveys have emerged to provide a multidisciplinary view of bias or to review bias in specific areas such as social sciences, business research, criminal justice, or data mining. Given the ability of Semantic Web (SW) technologies to support multiple AI systems, we review the extent to which semantics can be a “tool” to address bias in different algorithmic scenarios. We provide an in-depth categorisation and analysis of bias assessment, representation, and mitigation approaches that use SW technologies. We discuss their potential in dealing with issues such as representing disparities of specific demographics or reducing data drifts, sparsity, and missing values. We find research works on AI bias that apply semantics mainly in information retrieval, recommendation and natural language processing applications and argue through multiple use cases that semantics can help deal with technical, sociological, and psychological challenges.
Title: Bias in data-driven artificial intelligence systems—An introductory survey
Authors: Eirini Ntoutsi; Pavlos Fafalios; Ujwal Gadiraju; Vasileios Iosifidis; Wolfgang Nejdl; María-Esther Vidal; Salvatore Ruggieri; Franco Turini; Symeon Papadopoulos; Emmanouil Krasanakis; Ioannis Kompatsiaris; Katharina Kinder-Kurlanda; Claudia Wagner; Fariba Karimi; Miriam Fernández; Harith Alani; Bettina Berendt; Tina Kruegel; Christian Heinze; Klaus Broelemann; Gjergji Kasneci; Thanassis Tiropanis; Steffen Staab
Affiliations: L3S Research Center & Leibniz University Hannover; Institute of Computer Science, FORTH-ICS; TIB Leibniz Information Centre for Science and Technology; KDDLAB, Dipartimento di Informatica, Università di Pisa; Information Technologies Institute, CERTH; GESIS Leibniz Institute for the Social Sciences; Knowledge Media Institute, The Open University; Faculty of Electrical Engineering and Computer Science, KU Leuven; Institute for Legal Informatics, Leibniz University of Hanover; Innovation Lab, SCHUFA Holding AG; Electronics and Computer Science, University of Southampton; IPVS, Universität Stuttgart
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: AI-based systems are widely employed nowadays to make decisions that have far-reaching impacts on individuals and society. Their decisions might affect everyone, everywhere and anytime, entailing concerns about potential human rights issues. Therefore, it is necessary to move beyond traditional AI algorithms optimized for predictive performance and embed ethical and legal principles in their design, training and deployment to ensure social good while still benefiting from the huge potential of the AI technology. The goal of this survey is to provide a broad multi-disciplinary overview of the area of bias in AI systems, focusing on technical challenges and solutions as well as to suggest new research directions towards approaches well-grounded in a legal frame. In this survey, we focus on data-driven AI, as a large part of AI is powered nowadays by (big) data and powerful Machine Learning (ML) algorithms. If otherwise not specified, we use the general term bias to describe problems related to the gathering or processing of data that might result in prejudiced decisions on the bases of demographic features like race, sex, etc.
Title: Are AI systems biased against the poor? A machine learning analysis using Word2Vec and GloVe embeddings
Authors: Georgina Curto; Mario Jojoa; Flávio Comim; Begonya García-Zapirain
Affiliations: Universitat Ramon Llull; Universidad de Deusto; Universitat Ramon Llull; Universidad de Deusto
Year: 2022
Domain: LLMs & NLP

Abstract: Among the myriad of technical approaches and abstract guidelines proposed to the topic of AI bias, there has been an urgent call to translate the principle of fairness into the operational AI reality with the involvement of social sciences specialists to analyse the context of specific types of bias, since there is not a generalizable solution. This article offers an interdisciplinary contribution to the topic of AI and societal bias, in particular against the poor, providing a conceptual framework of the issue and a tailor-made model from which meaningful data are obtained using Natural Language Processing word vectors in pretrained Google Word2Vec, Twitter and Wikipedia GloVe word embeddings. The results of the study offer the first set of data that evidences the existence of bias against the poor and suggest that Google Word2vec shows a higher degree of bias when the terms are related to beliefs, whereas bias is higher in Twitter GloVe when the terms express behaviour. This article contributes to the body of work on bias, both from and AI and a social sciences perspective, by providing evidence of a transversal aggravating factor for historical types of discrimination. The evidence of bias against the poor also has important consequences in terms of human development, since it often leads to discrimination, which constitutes an obstacle for the effectiveness of poverty reduction policies.
Title: Analysis of Countries in Terms of Artificial Intelligence Technologies: PROMETHEE and GAIA Method Approach
Authors: Gökhan Özkaya; Ayşe Demirhan
Affiliations: Yıldız Technical University; Yıldız Technical University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Artificial intelligence development and research leaders in business, industry, and nations gain a major competitive edge. Additionally, it is clear that nations with a well-established national artificial intelligence policy have an edge over others, both technologically and economically. To further their artificial intelligence capability, nations also seek to develop a strategy, vision, structure, and working environment that encourages collaboration between the public sector, private industry, and educational institutions. Artificial intelligence is thought to be a tool that will help bridge the gap between powerful and developing countries growing in the future. Using data from “The Global AI Index” for 2021, this study aims to examine and analyze the present state of artificial intelligence management in 62 nations in terms of talent, infrastructure, business environment, development and research government policy, and commercial efforts. The research used PROMETHEE, which is widely used in multi-criteria decision-making evaluations, and its geometric representation, the GAIA plane. This study also found that the United States of America is the world leader in artificial intelligence (AI) research and development as well as AI investment. The United Kingdom, China, Israel, Canada, the Netherlands, South Korea, and Germany all rank well. China is rapidly catching up to the USA. At the very bottom of the list are nations such as Armenia, Kenya, Egypt, South Africa, and Pakistan. Turkey’s values are more similar to those of nations towards the bottom of the list than of those in the top half. There is a significant gap between the top three countries and the rest of the world in all parameters included in the survey. Except for the ‘State Strategy’ category, Turkey scores quite low compared to the top-performing countries. Decision makers are expected to address the identified global challenges of the study by creating a more comprehensive national AI strategy, both financially and in terms of content.