cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: A Survey on Bias and Fairness in Machine Learning
Authors: Ninareh Mehrabi; Fred Morstatter; Nripsuta Ani Saxena; Kristina Lerman; Aram Galstyan
Affiliations: USC Information Sciences Institute; USC Information Sciences Institute; USC Information Sciences Institute; USC Information Sciences Institute; USC Information Sciences Institute
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.
Title: A Review on Fairness in Machine Learning
Authors: Dana Pessach; Erez Shmueli
Affiliations: Tel Aviv University; Tel Aviv University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: An increasing number of decisions regarding the daily lives of human beings are being controlled by artificial intelligence and machine learning (ML) algorithms in spheres ranging from healthcare, transportation, and education to college admissions, recruitment, provision of loans, and many more realms. Since they now touch on many aspects of our lives, it is crucial to develop ML algorithms that are not only accurate but also objective and fair. Recent studies have shown that algorithmic decision making may be inherently prone to unfairness, even when there is no intention for it. This article presents an overview of the main concepts of identifying, measuring, and improving algorithmic fairness when using ML algorithms, focusing primarily on classification tasks. The article begins by discussing the causes of algorithmic bias and unfairness and the common definitions and measures for fairness. Fairness-enhancing mechanisms are then reviewed and divided into pre-process, in-process, and post-process mechanisms. A comprehensive comparison of the mechanisms is then conducted, toward a better understanding of which mechanisms should be used in different scenarios. The article ends by reviewing several emerging research sub-fields of algorithmic fairness, beyond classification.
Title: Fairness and Machine Learning: Limitations and Opportunities
Authors: Solon Barocas; Moritz Hardt; Arvind Narayanan
Affiliations: Cornell University; Heinrich-Heine-Universität Düsseldorf; Princeton University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Our success, happiness, and wellbeing are never fully of our own making. Others' decisions can profoundly affect the course of our lives: whether to admit us to a particular school, offer us a job, or grant us a mortgage. Arbitrary, inconsistent, or faulty decision-making thus raises serious concerns because it risks limiting our ability to achieve the goals that we have set for ourselves and access the opportunities for which we are qualified. Machine learning promises to bring greater discipline to decision-making because it offers to uncover factors that are relevant to decision-making that humans might overlook, given the complexity or subtlety of the relationships in historical evidence. But there are serious risks in learning from examples: reliably generalizing from historical examples to future cases requires that we provide the computer with good examples, and evidence-based decision-making is only as reliable as the evidence on which it is based. This is especially true when using machine learning to model human behavior and characteristics, since historical examples of the relevant outcomes will almost always reflect historical prejudices against certain social groups, prevailing cultural stereotypes, and existing demographic inequalities, and finding patterns in these data will often mean replicating these very same dynamics. This book surveys the risks in applications of machine learning to consequential decision-making in domains such as criminal justice, employment, and lending, and provides a critical review of an emerging set of proposed fairness solutions.
Title: Reconciling Modern Machine Learning Practice and the Bias-Variance Trade-Off
Authors: Mikhail Belkin; Daniel Hsu; Siyuan Ma; Soumik Mandal
Affiliations: The Ohio State University; Columbia University; The Ohio State University; The Ohio State University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.
Title: Fairness-Driven Private Collaborative Machine Learning
Authors: Dana Pessach; Tamir Tassa; Erez Shmueli
Affiliations: Tel-Aviv University; The Open University; Tel-Aviv University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: The performance of machine learning algorithms can be considerably improved when trained over larger datasets. In many domains, such as medicine and finance, larger datasets can be obtained if several parties, each having access to limited amounts of data, collaborate and share their data. However, such data sharing introduces significant privacy challenges. While multiple recent studies have investigated methods for private collaborative machine learning, the fairness of such collaborative algorithms was overlooked. In this work we suggest a feasible privacy-preserving pre-process mechanism for enhancing fairness of collaborative machine learning algorithms. Our experimentation with the proposed method shows that it is able to enhance fairness considerably with only a minor compromise in accuracy.
Title: Fairness in Machine Learning
Authors: Zuohui Fu; Yikun Xian; Ruoyuan Gao; Jieyu Zhao; Qiaoying Huang; Yingqiang Ge; Shuyuan Xu; Shijie Geng; Chirag Shah; Yongfeng Zhang; Gerard de Melo
Affiliations: University of Genoa; DeepMind (United Kingdom)
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Anchoring bias is one of the most prevalent biases within forecasting. It distorts managers’ estimations whenever context-driven intervention to the statistical model output is required. Consequences extend beyond a single organization since forecasting affects order quantity decisions and, therefore, the relations among suppliers, potentially generating a bullwhip effect throughout the supply chain. Anchoring bias can have a significant impact, and despite being related to a numerical value, its detection is very complex. Moreover, it tends to be recurrent when the context that caused the distortion is not explored and precisely understood. Current detection approaches are incomplete, as they do not make explicit the directional component of anchors or their meaning to the decision maker’s mental heuristics. In this work, we present Anchorlogy, an ontology devised to explicitly provide the required context to detect and mitigate anchoring bias during a decision-making process, and a metrological approach to measure it while addressing the deficiencies found in other metrics in the current psychological literature. Our proposal was validated by applying it to two case studies in the forecasting domain, and the results show that it effectively prevents the bullwhip effect in real-world scenarios.
Title: Fairness-Aware Explainable Recommendation over Knowledge Graphs
Authors: Zuohui Fu; Yikun Xian; Ruoyuan Gao; Jieyu Zhao; Qiaoying Huang; Yingqiang Ge; Shuyuan Xu; Shijie Geng; Chirag Shah; Yongfeng Zhang; Gerard de Melo
Affiliations: Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; University of California, Los Angeles; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; University of Washington; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2020
Domain: Recommender Systems

Abstract: There has been growing attention on fairness considerations recently, especially in the context of intelligent decision making systems. For example, explainable recommendation systems may suffer from both explanation bias and performance disparity. We show that inactive users may be more susceptible to receiving unsatisfactory recommendations due to their insufficient training data, and that their recommendations may be biased by the training records of active users due to the nature of collaborative filtering, which leads to unfair treatment by the system. In this paper, we analyze different groups of users according to their level of activity, and find that bias exists in recommendation performance between different groups. Empirically, we find that such performance gap is caused by the disparity of data distribution, specifically the knowledge graph path distribution in this work. We propose a fairness constrained approach via heuristic re-ranking to mitigate this unfairness problem in the context of explainable recommendation over knowledge graphs. We experiment on several real-world datasets with state-of-the-art knowledge graph-based explainable recommendation algorithms. The promising results show that our algorithm is not only able to provide high-quality explainable recommendations, but also reduces the recommendation unfairness in several aspects.
Title: Big Data's Disparate Impact
Authors: Solon Barocas; Andrew D. Selbst
Affiliations: Microsoft Research Cornell University; UCLA School of Law
Year: 2016
Domain: General Fairness & Bias Mitigation

Abstract: While artificial intelligence (AI) holds great promise in improving various domains of clinical practice, from administrative tasks to diagnostic tools and clinical decision support, the potential for bias raises significant concerns related to patient harm. This article provides a brief overview of bias in AI models intended for use in health care settings and its clinical impact. In addition, a simple framework is offered for mitigating bias in AI models.
Title: Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus-Level Constraints
Authors: Jieyu Zhao; Tianlu Wang; Mark Yatskar; Vicente Ordóñez; Kai-Wei Chang
Affiliations: University of Virginia; University of Virginia; University of Washington; University of Virginia; University of Virginia
Year: 2017
Domain: LLMs & NLP

Abstract: Language is increasingly being used to define rich visual recognition problems with supporting image collections sourced from the web. Structured prediction models are used in these tasks to take advantage of correlations between co-occurring labels and visual input but risk inadvertently encoding social biases found in web corpora. In this work, we study data and models associated with multilabel object classification and visual semantic role labeling. We find that (a) datasets for these tasks contain significant gender bias and (b) models trained on these datasets further amplify existing bias. For example, the activity cooking is over 33% more likely to involve females than males in a training set, and a trained model further amplifies the disparity to 68% at test time. We propose to inject corpus-level constraints for calibrating existing structured prediction models and design an algorithm based on Lagrangian relaxation for collective inference. Our method results in almost no performance loss for the underlying recognition task but decreases the magnitude of bias amplification by 47.5% and 40.5% for multilabel classification and visual semantic role labeling, respectively.
Title: Certifying and Removing Disparate Impact
Authors: Michael Feldman; Sorelle A. Friedler; John Moeller; Carlos Scheidegger; Suresh Venkatasubramanian
Affiliations: Haverford College; Haverford College; University of Utah; University of Arizona; University of Utah
Year: 2015
Domain: General Fairness & Bias Mitigation

Abstract: What does it mean for an algorithm to be biased? In U.S. law, unintentional bias is encoded via disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. This legal determination hinges on a definition of a protected class (ethnicity, gender) and an explicit description of the process.
Title: Integrating Behavioral Economic and Technical Insights to Address Algorithmic Bias
Authors: Gediminas Adomavičius; Mochen Yang
Affiliations: University of Minnesota; University of Minnesota
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Many important decisions are increasingly being made with the help of information systems that use artificial intelligence and machine learning models. These computational models are designed to discover useful patterns from large amounts of data, which augment human capabilities to make decisions in various application domains. However, there are growing concerns regarding the ethics challenges faced by these automated decision-making (ADM) models, most notably on the issue of algorithmic bias , in which the models systematically produce less favorable (i.e., unfair) decisions for certain groups of people. In this commentary, we argue that algorithmic bias is not just a technical (e.g., computational or statistical) problem, and its successful resolution requires deep insights into individual and organizational behavior, economic incentives, as well as complex dynamics of the sociotechnical systems in which the ADM models are embedded. We discuss a human-centric, fairness-aware ADM framework that highlights the holistic involvement of human decision makers in each step of ADM. We review the emerging literature on fairness-aware machine learning and then discuss various strategic decisions that humans need to make, such as formulating proper fairness objectives, recognizing fairness-induced trade-offs and implications, utilizing machine learning model outputs, and managing/governing the decisions of ADM models. We further illustrate how these strategic decisions are jointly informed by behavioral, economic, and design sciences. Our discussions reveal a number of future research opportunities uniquely suitable for Management Information Systems (MIS) researchers to pursue.
Title: Fairness in Recommendation: Foundations Methods and Applications
Authors: Yunqi Li; Hanxiong Chen; Shuyuan Xu; Yingqiang Ge; Juntao Tan; Shuchang Liu; Yongfeng Zhang
Affiliations: Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2023
Domain: Recommender Systems

Abstract: As one of the most pervasive applications of machine learning, recommender systems are playing an important role on assisting human decision-making. The satisfaction of users and the interests of platforms are closely related to the quality of the generated recommendation results. However, as a highly data-driven system, recommender system could be affected by data or algorithmic bias and thus generate unfair results, which could weaken the reliance of the systems. As a result, it is crucial to address the potential unfairness problems in recommendation settings. Recently, there has been growing attention on fairness considerations in recommender systems with more and more literature on approaches to promote fairness in recommendation. However, the studies are rather fragmented and lack a systematic organization, thus making it difficult to penetrate for new researchers to the domain. This motivates us to provide a systematic survey of existing works on fairness in recommendation. This survey focuses on the foundations for fairness in recommendation literature. It first presents a brief introduction about fairness in basic machine learning tasks such as classification and ranking to provide a general overview of fairness research, as well as introduce the more complex situations and challenges that need to be considered when studying fairness in recommender systems. After that, the survey will introduce fairness in recommendation with a focus on the taxonomies of current fairness definitions, the typical techniques for improving fairness, as well as the datasets for fairness studies in recommendation. The survey also talks about the challenges and opportunities in fairness research with the hope of promoting the fair recommendation research area and beyond.
Title: A Survey on Trustworthy Recommender Systems
Authors: Yingqiang Ge; Shuchang Liu; Zuohui Fu; Juntao Tan; Zelong Li; Shuyuan Xu; Yunqi Li; Yikun Xian; Yongfeng Zhang
Affiliations: Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey; Rutgers, The State University of New Jersey
Year: 2024
Domain: Recommender Systems

Abstract: Recommender systems (RS), serving at the forefront of Human-centered AI, are widely deployed in almost every corner of the web and facilitate the human decision-making process. However, despite their enormous capabilities and potential, RS may also lead to undesired effects on users, items, producers, platforms, or even the society at large, such as compromised user trust due to non-transparency, unfair treatment of different consumers, or producers, privacy concerns due to extensive use of user’s private data for personalization, just to name a few. All of these create an urgent need for Trustworthy Recommender Systems (TRS) so as to mitigate or avoid such adverse impacts and risks. In this survey, we will introduce techniques related to trustworthy recommendation, including but not limited to explainable recommendation, fairness in recommendation, privacy-aware recommendation, robustness in recommendation, user-controllable recommendation, as well as the relationship between these different perspectives in terms of trustworthy recommendation. Through this survey, we hope to deliver readers with a comprehensive view of the research area and raise attention to the community about the importance, existing research achievements, and future research directions on trustworthy recommendation.
Title: A Comprehensive Survey on Trustworthy Recommender Systems
Authors: Wenqi Fan; Xiangyu Zhao; Xiao Chen; Jingran Su; Jingtong Gao; Lin Wang; Qidong Liu; Yiqi Wang; Xu Han; Lei Chen; Qing Li
Affiliations: The Hong Kong Polytechnic University;
City University of Hong Kong; The Hong Kong Polytechnic University; The Hong Kong Polytechnic University; City University of Hong Kong;
The Hong Kong Polytechnic University;City University of Hong Kong; Michigan State University; Michigan State University;The Hong Kong University of Science and Technology;The Hong Kong Polytechnic University
Year: 2022
Domain: Recommender Systems

Abstract: As one of the most successful AI-powered applications, recommender systems aim to help people make appropriate decisions in an effective and efficient way, by providing personalized suggestions in many aspects of our lives, especially for various human-oriented online services such as e-commerce platforms and social media sites. In the past few decades, the rapid developments of recommender systems have significantly benefited human by creating economic value, saving time and effort, and promoting social good. However, recent studies have found that data-driven recommender systems can pose serious threats to users and society, such as spreading fake news to manipulate public opinion in social media sites, amplifying unfairness toward under-represented groups or individuals in job matching services, or inferring privacy information from recommendation results. Therefore, systems' trustworthiness has been attracting increasing attention from various aspects for mitigating negative impacts caused by recommender systems, so as to enhance the public's trust towards recommender systems techniques. In this survey, we provide a comprehensive overview of Trustworthy Recommender systems (TRec) with a specific focus on six of the most important aspects; namely, Safety &amp; Robustness, Nondiscrimination &amp; Fairness, Explainability, Privacy, Environmental Well-being, and Accountability &amp; Auditability. For each aspect, we summarize the recent related technologies and discuss potential research directions to help achieve trustworthy recommender systems in the future.