cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: A Theory of Auditability for Allocation and Social Choice Mechanisms
Authors: Aram Grigoryan; Markus Möller
Affiliations: University of California San Diego; University of Bonn
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In centralized market mechanisms individuals may not fully observe other participants' type reports. Hence, the mechanism designer may deviate from the promised mechanism without the individuals being able to detect these deviations. In this paper, we develop a theory of auditability for allocation and social choice problems. Namely, we measure a mechanism's auditabilty by the smallest number of individuals that can jointly detect any deviation. Our theory reveals stark contrasts between prominent mechanisms' auditabilities in various applications. For priority-based allocation problems, we find that the Immediate Acceptance mechanism is maximally auditable, in a sense that any deviation can always be detected by just two individuals, whereas, on the other extreme, the Deferred Acceptance mechanism is minimally auditable, in a sense that some deviations may go undetected unless some individuals possess full information about everyone's reports. For the auctions setup, we find a similar contrast between the first-price and the second-price auction mechanisms. For voting problems, we characterize the majority voting rule as the unique most auditable anonymous voting mechanism. And finally, for the choice with affirmative action setting, we compare the auditability indices of prominent reserves mechanisms.
Title: ACROCPoLis: A Descriptive Framework for Making Sense of Fairness
Authors: Andrea Aler Tubella; Dimitri Coelho Mollo; Adam Dahlgren Lindström; Hannah Devinney; Virginia Dignum; Petter Ericson; Anna Jönsson; Timotheus Kampik; Tom Lenaerts; Julian Mendez; Juan Carlos Nieves
Affiliations: Umeå University; Umeå University; Umeå University; Umeå University; Umeå University; Umeå University; Umeå University; Umeå University; Université Libre de Bruxelles; Umeå University; Umeå University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Fairness is central to the ethical and responsible development and use of AI systems, with a large number of frameworks and formal notions of algorithmic fairness being available. However, many of the fairness solutions proposed revolve around technical considerations and not the needs of and consequences for the most impacted communities. We therefore want to take the focus away from definitions and allow for the inclusion of societal and relational aspects to represent how the effects of AI systems impact and are experienced by individuals and social groups. In this paper, we do this by means of proposing the ACROCPoLis framework to represent allocation processes with a modeling emphasis on fairness aspects. The framework provides a shared vocabulary in which the factors relevant to fairness assessments for different situations and procedures are made explicit, as well as their interrelationships. This enables us to compare analogous situations, to highlight the differences in dissimilar situations, and to capture differing interpretations of the same situation by different stakeholders.
Title: Addressing Bias and Fairness in Machine Learning: A Practical Guide and Hands-on Tutorial
Authors: Rayid Ghani; Kit T. Rodolfa; Pedro Saleiro; Sergio Jesús
Affiliations: Carnegie Mellon University; Stanford University; Universidade do Porto
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: As data science and machine learning (ML) increasingly shape our society, the importance of developing fair algorithmic decision-making systems becomes paramount. There is a pressing need to train data scientists and practitioners on handling bias and fairness in real-world scenarios, from early stages of a data science project to maintaining ML systems in production. Existing resources are mostly academic and cover the ML training and optimization aspects of bias mitigation, leaving practitioners without comprehensive frameworks for making decisions throughout a real-world project lifecycle. This tutorial aims to bridge the gap between research and practice, providing an in-depth exploration of algorithmic fairness, encompassing metrics and definitions, practical case studies, data bias understanding, bias mitigation and model fairness audits using the Aequitas toolkit. Participants will be equipped to engage in conversations about bias, assist decision-makers in understanding options and trade-offs, evaluate project scoping aspects influencing fairness outcomes, and define actions and interventions based on model predictions. They will also learn to identify cohorts, target variables, evaluation metrics, and establish bias and fairness goals for different groups. Moreover, participants will gain insights into auditing and mitigating model bias, and implementing continuous monitoring to assess retraining needs. The tutorial addresses the current lack of practical training materials, methodologies, and tools for researchers and developers working on real-world algorithmic decision-making systems. By the conclusion of this hands-on tutorial, attendees will be well-versed in navigating bias-related issues, selecting appropriate metrics, and applying bias audit and mitigation frameworks and tools for informed design decisions in real-world data science systems.
Title: Advances in decision support for diagnosis and early management of acute leukaemia
Authors: Amin T. Turki; Merlin Engelke; Marta Sobas
Affiliations: Essen University Hospital; National Institute Of Blood Disease and Bone Marrow Transplantation
Year: 2024
Domain: Health & Clinical AI

Abstract: Acute leukaemia includes several life-threatening malignancies, mainly acute myeloid leukaemia (AML) and acute lymphocytic leukaemia (ALL). Although long-term patient outcomes depend on disease-related genomic factors that translate into treatment resistance and relapse, early mortality remains a substantial concern, especially in people with acute promyelocytic leukaemia (APL). Despite genomic or flow-cytometry diagnostics, traditional morphological examination remains the primary method of diagnosis. AI tools can contribute to diagnosing acute leukaemia, with advancements in cytology-image segmentation and deep learning models pioneering reliable detection of myeloid blasts in blood and marrow.
Title: Algorithm Bias and Perceived Fairness: A Comprehensive Scoping Review
Authors: Amirhossein Hajigholam Saryazdi
Affiliations: Concordia University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Artificial intelligence (AI)-based algorithms are playing an increasingly prominent role in shaping daily life. However, these algorithms can exhibit biases that exacerbate societal injustices. Such biases have a substantial impact on people's perceptions of algorithmic fairness, yet the precise mechanisms and scope of this phenomenon remain relatively understudied. To address this research gap, a comprehensive scoping literature review is conducted, providing an overview of current research in the field. Subsequently, a novel theoretical model is developed that synthesizes key themes, including algorithm bias, algorithm fairness, perceived fairness, individual characteristics, social characteristics, task characteristics, and technology characteristics. The paper contributes proposing a set of propositions that underscore the critical gaps in the existing literature, contribute to a deeper comprehension of the relationships among the identified themes and their constituent elements, and offer a roadmap for future research in the domain.
Title: Algorithmic Pluralism: A Structural Approach To Equal Opportunity
Authors: Shomik Jain; Vinith Suriyakumar; Kathleen Creel; Ashia Wilson
Affiliations: Data & Society Research Institute; Northeastern University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: We present a structural approach toward achieving equal opportunity in systems of algorithmic decision-making called algorithmic pluralism. Algorithmic pluralism describes a state of affairs in which no set of algorithms severely limits access to opportunity, allowing individuals the freedom to pursue a diverse range of life paths. To argue for algorithmic pluralism, we adopt Joseph Fishkin’s theory of bottlenecks, which focuses on the structure of decision-points that determine how opportunities are allocated. The theory contends that each decision-point or “bottleneck’’ limits access to opportunities with some degree of severity and legitimacy. We extend Fishkin’s structural viewpoint and use it to reframe existing systemic concerns about equal opportunity in algorithmic decision-making, such as patterned inequality and algorithmic monoculture. In proposing algorithmic pluralism, we argue for the urgent priority of alleviating severe bottlenecks in algorithmic-decision-making. We contend that there must be a pluralism of opportunity available to many different individuals in order to promote equal opportunity in a systemic way. We further show how this framework has several implications for system design and regulation through current debates about equal opportunity in algorithmic hiring.
Title: Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
Authors: Rongxin Zhu; Jianzhong Qi; Jey Han Lau
Affiliations: University of Melbourne; University of Melbourne; University of Melbourne
Year: 2023
Domain: LLMs & NLP

Abstract: A series of datasets and models have been proposed for summaries generated for well-formatted documents such as news articles. Dialogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT. We define fine-grained factual error detection as a sentence-level multi-label classification problem, and weevaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation.
Title: Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study
Authors: Travis Zack; Eric Lehman; Mirac Süzgün; Jorge A. Rodriguez; Leo Anthony Celi; Judy Wawira Gichoya; Dan Jurafsky; Peter Szolovits; David W. Bates; Raja-Elie E. Abdulnour; Atul J. Butte; Emily Alsentzer
Affiliations: Massachusetts Institute of Technology; Brigham and Women's Hospital; Beth Israel Deaconess Medical Center; Emory University; Massachusetts Institute of Technology
Year: 2023
Domain: LLMs & NLP

Abstract: BACKGROUND: Large language models (LLMs) such as GPT-4 hold great promise as transformative tools in health care, ranging from automating administrative tasks to augmenting clinical decision making. However, these models also pose a danger of perpetuating biases and delivering incorrect medical diagnoses, which can have a direct, harmful impact on medical care. We aimed to assess whether GPT-4 encodes racial and gender biases that impact its use in health care. METHODS: Using the Azure OpenAI application interface, this model evaluation study tested whether GPT-4 encodes racial and gender biases and examined the impact of such biases on four potential applications of LLMs in the clinical domain-namely, medical education, diagnostic reasoning, clinical plan generation, and subjective patient assessment. We conducted experiments with prompts designed to resemble typical use of GPT-4 within clinical and medical education applications. We used clinical vignettes from NEJM Healer and from published research on implicit bias in health care. GPT-4 estimates of the demographic distribution of medical conditions were compared with true US prevalence estimates. Differential diagnosis and treatment planning were evaluated across demographic groups using standard statistical tests for significance between groups. FINDINGS: We found that GPT-4 did not appropriately model the demographic diversity of medical conditions, consistently producing clinical vignettes that stereotype demographic presentations. The differential diagnoses created by GPT-4 for standardised clinical vignettes were more likely to include diagnoses that stereotype certain races, ethnicities, and genders. Assessment and plans created by the model showed significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception. INTERPRETATION: Our findings highlight the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care. We discuss the potential sources of these biases and potential mitigation strategies before clinical implementation. FUNDING: Priscilla Chan and Mark Zuckerberg.
Title: Association Between Unmet Nonmedication Needs After Hospital Discharge and Readmission or Death Among Acute Respiratory Failure Survivors: A Multicenter Prospective Cohort Study*
Authors: Somnath Bose; Danielle Groat; Victor D. Dinglas; Narjes Akhlaghi; Valerie Banner‐Goodspeed; Sarah J. Beesley; Tom Greene; Ramona O. Hopkins; Mustafa Mir-Kasimov; Carla M. Sevin; Alison E. Turnbull; James C. Jackson; Dale M. Needham; Samuel M. Brown; for the Addressing Post-Intensive Care Syndrome (APICS-01) Study Team
Affiliations: Harvard University; Intermountain Medical Center; Yale University; Harvard University; University of Utah; Brigham Young University; University of Utah; Vanderbilt University Medical Center; Johns Hopkins University; Vanderbilt University Medical Center; University of Utah
Year: 2023
Domain: Health & Clinical AI

Abstract: OBJECTIVES: To characterize early unmet nonmedication discharge needs (UDNs), classified as durable medical equipment (DME), home health services (HHS), and follow-up medical appointments (FUAs) and explore their association with 90-day readmission and mortality among survivors of acute respiratory failure (ARF) who were discharged home. DESIGN: Prospective multicenter cohort study. SETTING: Six academic medical centers across United States. PARTICIPANTS: Adult survivors of ARF who required an ICU stay and were discharged home from hospital. INTERVENTIONS: None. Exposure of interest was the proportion of UDN for the following categories: DME, HHS, and FUA ascertained within 7-28 days after hospital discharge. MEASUREMENTS AND MAIN RESULTS: Two hundred eligible patients were recruited between January 2019 and August 2020. One-hundred ninety-five patients were included in the analytic cohort: 118 were prescribed DME, 134 were prescribed HHS, and 189 needed at least one FUA according to discharge plans. 98.4% (192/195) had at least one identified nonmedication need at hospital discharge. Median (interquartile range) proportion of unmet needs across three categories were 0 (0-15%) for DME, 0 (0-50%) for HHS, and 0 (0-25%) for FUA, and overall was 0 (0-20%). Fifty-six patients (29%) had 90-day death or readmission. After adjusting for prespecified covariates, having greater than the median level of unmet needs was not associated with an increased risk of readmission or death within 90 days of discharge (risk ratio, 0.89; 0.51-1.57; p = 0.690). Age, hospital length of stay, Acute Physiology and Chronic Health Evaluation II severity of illness score, and Multidimensional Scale Perceived Social Support score were associated with UDN. CONCLUSIONS: UDN were common among survivors of ARF but not significantly associated a composite outcome of 90-day readmission or death. Our results highlight the substantial magnitude of UDN and identifies areas especially vulnerable to lapses in healthcare coordination.
Title: Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems
Authors: Rock Yuren Pang; Jack Cenatempo; Franklyn J. Graham; Bridgette Kuehn; Maddy Whisenant; Portia Botchway; Katie Stone Perez; Allison Koenecke
Affiliations: University of Washington; Microsoft (United States); Microsoft (United States); Microsoft (United States); Microsoft (United States); Microsoft (United States); Microsoft (United States); Cornell University
Year: 2023
Domain: Recommender Systems

Abstract: Recommendation systems increasingly depend on massive human-labeled datasets; however, the human annotators hired to generate these labels increasingly come from homogeneous backgrounds. This poses an issue when downstream predictive models—based on these labels—are applied globally to a heterogeneous set of users. We study this disconnect with respect to the labels themselves, asking whether they are “consistently conceptualized” across annotators of different demographics. In a case study of video game labels, we conduct a survey on 5,174 gamers, identify a subset of inconsistently conceptualized game labels, perform causal analyses, and suggest both cultural and linguistic reasons for cross-country differences in label annotation. We further demonstrate that predictive models of game annotations perform better on global train sets as opposed to homogeneous (single-country) train sets. Finally, we provide a generalizable framework for practitioners to audit their own data annotation processes for consistent label conceptualization, and encourage practitioners to consider global inclusivity in recommendation systems starting from the early stages of annotator recruitment and data-labeling.
Title: Auditing for Racial Discrimination in the Delivery of Education Ads
Authors: Basileal Imana; Aleksandra Korolova; John Heidemann
Affiliations: Princeton University; Princeton University; University of Southern California
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Digital ads on social-media platforms play an important role in shaping access to economic opportunities. Our work proposes and implements a new third-party auditing method that can evaluate racial bias in the delivery of ads for education opportunities. Third-party auditing is important because it allows external parties to demonstrate presence or absence of bias in social-media algorithms. Education is a domain with legal protections against discrimination and concerns of racial-targeting, but bias induced by ad delivery algorithms has not been previously explored in this domain. Prior audits demonstrated discrimination in platforms’ delivery of ads to users for housing and employment ads. These audit findings supported legal action that prompted Meta to change their ad-delivery algorithms to reduce bias, but only in the domains of housing, employment, and credit. In this work, we propose a new methodology that allows us to measure racial discrimination in a platform’s ad delivery algorithms for education ads. We apply our method to Meta using ads for real schools and observe the results of delivery. We find evidence of racial discrimination in Meta’s algorithmic delivery of ads for education opportunities, posing legal and ethical concerns. Our results extend evidence of algorithmic discrimination to the education domain, showing that current bias mitigation mechanisms are narrow in scope, and suggesting a broader role for third-party auditing of social media in areas where ensuring non-discrimination is important.
Title: Auditing Work: Exploring the New York City algorithmic bias audit regime
Authors: Lara Groves; Jacob Metcalf; Alayna Kennedy; Briana Vecchione; Andrew Strait
Affiliations: Data & Society Research Institute; Film Independent; Data & Society Research Institute
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In July 2023, New York City (NYC) implemented the first attempt to create an algorithm auditing regime for commercial machine-learning systems. Local Law 144 (LL 144), requires NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to be subject to annual bias audits by an independent auditor. In this paper, we analyse what lessons can be learned from LL 144 for other national attempts to create algorithm auditing regimes. Using qualitative interviews with 17 experts and practitioners working within the regime, we find LL 144 has failed to create an effective auditing regime: the law fails to clearly define key aspects like AEDTs and what constitutes an independent auditor, leaving auditors, vendors who create AEDTs, and companies using AEDTs to define the law’s practical implementation in ways that failed to protect job applicants. Several factors contribute to this: first, the law was premised on a faulty transparency-driven theory of change that fails to stop biased AEDTs from being used by employers. Second, industry lobbying led to the definition of what constitutes an AEDT being narrowed to the point where most companies considered their tools exempt. Third, we find auditors face enormous practical and cultural challenges gaining access to data from employers and vendors building these tools. Fourth, we find wide disagreement over what constitutes a legitimate auditor and identify four different kinds of ‘auditor roles’ that serve different functions and offer different kinds of services. We conclude with four recommendations for policymakers seeking to create similar bias auditing regimes that use clearer definitions and metrics and more accountability. By exploring LL 144 through the lens of auditors, our paper advances the evidence base around audit as an accountability mechanism, and can provide guidance for policymakers seeking to create similar regimes.
Title: Bias and Debias in Recommender System: A Survey and Future Directions
Authors: Jiawei Chen; Hande Dong; Xiang Wang; Fuli Feng; Meng Wang; Xiangnan He
Affiliations: University of Science and Technology of China; University of Science and Technology of China; University of Science and Technology of China; University of Science and Technology of China; Hefei University of Technology; University of Science and Technology of China
Year: 2022
Domain: Recommender Systems

Abstract: While recent years have witnessed a rapid growth of research papers on recommender system (RS) , most of the papers focus on inventing machine learning models to better fit user behavior data. However, user behavior data is observational rather than experimental. This makes various biases widely exist in the data, including but not limited to selection bias, position bias, exposure bias, and popularity bias. Blindly fitting the data without considering the inherent biases will result in many serious issues, e.g., the discrepancy between offline evaluation and online metrics, hurting user satisfaction and trust on the recommendation service, and so on. To transform the large volume of research models into practical improvements, it is highly urgent to explore the impacts of the biases and perform debiasing when necessary. When reviewing the papers that consider biases in RS, we find that, to our surprise, the studies are rather fragmented and lack a systematic organization. The terminology “bias” is widely used in the literature, but its definition is usually vague and even inconsistent across papers. This motivates us to provide a systematic survey of existing work on RS biases. In this paper, we first summarize seven types of biases in recommendation, along with their definitions and characteristics. We then provide a taxonomy to position and organize the existing work on recommendation debiasing. Finally, we identify some open challenges and envision some future directions, with the hope of inspiring more research work on this important yet less investigated topic. The summary of debiasing methods reviewed in this survey can be found at https://github.com/jiawei-chen/RecDebiasing .