cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Retrieving Relevant Context to Align Representations for Cross-lingual Event Detection
Authors: Anh Chien Nguyen; Linh Ngo; Thien Huu Nguyen
Affiliations: VinUniversity; Hanoi University of Science and Technology; University of Oregon
Year: 2023
Domain: LLMs & NLP

Abstract: We study the problem of cross-lingual transfer learning for event detection (ED) where models trained on a source language are expected to perform well on data for a new target language. Among a few recent works for this problem, the main approaches involve representation matching (e.g., adversarial training) that aims to eliminate language-specific features from the representations to achieve the language-invariant representations. However, due to the mix of language-specific features with event-discriminative context, representation matching methods might also remove important features for event prediction, thus hindering the performance for ED. To address this issue, we introduce a novel approach for cross-lingual ED where representations are augmented with additional context (i.e., not eliminating) to bridge the gap between languages while enriching the contextual information to facilitate ED. At the core of our method involves a retrieval model that retrieves relevant sentences in the target language for an input sentence to compute augmentation representations. Experiments on three languages demonstrate the state-of-the-art performance of our model for cross-lingual ED.
Title: SAF: Stakeholders’ Agreement on Fairness in the Practice of Machine Learning Development
Authors: Georgina Curto; Flávio Comim
Affiliations: University of Notre Dame; Universitat Ramon Llull
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: This paper clarifies why bias cannot be completely mitigated in Machine Learning (ML) and proposes an end-to-end methodology to translate the ethical principle of justice and fairness into the practice of ML development as an ongoing agreement with stakeholders. The pro-ethical iterative process presented in the paper aims to challenge asymmetric power dynamics in the fairness decision making within ML design and support ML development teams to identify, mitigate and monitor bias at each step of ML systems development. The process also provides guidance on how to explain the always imperfect trade-offs in terms of bias to users.
Title: Should Fairness be a Metric or a Model? A Model-based Framework for Assessing Bias in Machine Learning Pipelines
Authors: John P. Lalor; Ahmed Abbasi; Kezia Oketch; Yi Yang; Nicole Forsgren
Affiliations: University of Notre Dame; University of Notre Dame; University of Notre Dame; Hong Kong University of Science and Technology; Microsoft (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Fairness measurement is crucial for assessing algorithmic bias in various types of machine learning (ML) models, including ones used for search relevance, recommendation, personalization, talent analytics, and natural language processing. However, the fairness measurement paradigm is currently dominated by fairness metrics that examine disparities in allocation and/or prediction error as univariate key performance indicators (KPIs) for a protected attribute or group. Although important and effective in assessing ML bias in certain contexts such as recidivism, existing metrics don’t work well in many real-world applications of ML characterized by imperfect models applied to an array of instances encompassing a multivariate mixture of protected attributes, that are part of a broader process pipeline. Consequently, the upstream representational harm quantified by existing metrics based on how the model represents protected groups doesn’t necessarily relate to allocational harm in the application of such models in downstream policy/decision contexts. We propose FAIR-Frame, a model-based framework for parsimoniously modeling fairness across multiple protected attributes in regard to the representational and allocational harm associated with the upstream design/development and downstream usage of ML models. We evaluate the efficacy of our proposed framework on two testbeds pertaining to text classification using pretrained language models. The upstream testbeds encompass over fifty thousand documents associated with twenty-eight thousand users, seven protected attributes and five different classification tasks. The downstream testbeds span three policy outcomes and over 5.41 million total observations. Results in comparison with several existing metrics show that the upstream representational harm measures produced by FAIR-Frame and other metrics are significantly different from one another, and that FAIR-Frame’s representational fairness measures have the highest percentage alignment and lowest error with allocational harm observed in downstream applications. Our findings have important implications for various ML contexts, including information retrieval, user modeling, digital platforms, and text classification, where responsible and trustworthy AI is becoming an imperative.
Title: SKD-NER: Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning
Authors: Yi Chen; He Liang
Affiliations: Xinjiang University; Xinjiang University
Year: 2023
Domain: LLMs & NLP

Abstract: Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones. However, the current strategies fall short of effectively addressing the catastrophic forgetting of previously learned entity types. To tackle this issue, we propose the SKD-NER model, an efficient continual learning NER model based on the span-based approach, which innovatively incorporates reinforcement learning strategies to enhance the model’s ability against catastrophic forgetting. Specifically, we leverage knowledge distillation (KD) to retain memory and employ reinforcement learning strategies during the KD process to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting during continual learning. This approach effectively prevents or mitigates catastrophic forgetting during continuous learning, allowing the model to retain previously learned knowledge while acquiring new knowledge. Our experiments on two benchmark datasets demonstrate that our model significantly improves the performance of the CL-NER task, outperforming state-of-the-art methods.
Title: Sociodemographic bias in clinical machine learning models: a scoping review of algorithmic bias instances and mechanisms
Authors: Michael Colacci; Yu Qing Huang; Gemma Postill; Pavel Zhelnov; Orna Fennelly; Amol A. Verma; Sharon E. Straus; Andrea C. Tricco
Affiliations: St. Michael's Hospital; St. Michael's Hospital; University of Toronto; St. Michael's Hospital; St. Michael's Hospital; St. Michael's Hospital; St. Michael's Hospital; St. Michael's Hospital
Year: 2024
Domain: Health & Clinical AI

Abstract: BACKGROUND AND OBJECTIVES: Clinical machine learning (ML) technologies can sometimes be biased and their use could exacerbate health disparities. The extent to which bias is present, the groups who most frequently experience bias, and the mechanism through which bias is introduced in clinical ML applications is not well described. The objective of this study was to examine instances of bias in clinical ML models. We identified the sociodemographic subgroups PROGRESS that experienced bias and the reported mechanisms of bias introduction. METHODS: We searched MEDLINE, EMBASE, PsycINFO, and Web of Science for all studies that evaluated bias on sociodemographic factors within ML algorithms created for the purpose of facilitating clinical care. The scoping review was conducted according to the Joanna Briggs Institute guide and reported using the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) extension for scoping reviews. RESULTS: We identified 6448 articles, of which 760 reported on a clinical ML model and 91 (12.0%) completed a bias evaluation and met all inclusion criteria. Most studies evaluated a single sociodemographic factor (n = 56, 61.5%). The most frequently evaluated sociodemographic factor was race (n = 59, 64.8%), followed by sex/gender (n = 41, 45.1%), and age (n = 24, 26.4%), with one study (1.1%) evaluating intersectional factors. Of all studies, 74.7% (n = 68) reported that bias was present, 18.7% (n = 17) reported bias was not present, and 6.6% (n = 6) did not state whether bias was present. When present, 87% of studies reported bias against groups with socioeconomic disadvantage. CONCLUSION: Most ML algorithms that were evaluated for bias demonstrated bias on sociodemographic factors. Furthermore, most bias evaluations concentrated on race, sex/gender, and age, while other sociodemographic factors and their intersection were infrequently assessed. Given potential health equity implications, bias assessments should be completed for all clinical ML models.
Title: The Conflict Between Algorithmic Fairness and Non-Discrimination: An Analysis of Fair Automated Hiring
Authors: Robert Lee Poe; Soumia Zohra El Mestari
Affiliations: Scuola Superiore Sant'Anna; University of Luxembourg
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: AI-based automated hiring systems cover a wide range of tools of varying complexity, from resume parsing tools to candidate selection models. Their close interference in economic and social life faces raising demands and investigations aiming to reduce the potential discrimination they may cause. This article covers the intersection of EU non-discrimination law and algorithmic fairness in the context of automated hiring systems. The paper analyzes the balance between equality of opportunity (formal and substantive) and equality of outcome, critiques the focus on non-conservative group fairness in machine learning, and discusses the legal implications of automated hiring systems under EU law. Additionally, it highlights often committed fallacies in relation to the process of de-biasing and advocates for a broader understanding of fairness in machine learning that aligns with EU legal standards and societal values.
Title: The limits of fair medical imaging AI in real-world generalization
Authors: Yuzhe Yang; Haoran Zhang; Judy Wawira Gichoya; Dina Katabi; Marzyeh Ghassemi
Affiliations: Massachusetts Institute of Technology; Massachusetts Institute of Technology; Emory University; Massachusetts Institute of Technology; Massachusetts Institute of Technology
Year: 2024
Domain: Health & Clinical AI

Abstract: As artificial intelligence (AI) rapidly approaches human-level performance in medical imaging, it is crucial that it does not exacerbate or propagate healthcare disparities. Previous research established AI's capacity to infer demographic data from chest X-rays, leading to a key concern: do models using demographic shortcuts have unfair predictions across subpopulations? In this study, we conducted a thorough investigation into the extent to which medical AI uses demographic encodings, focusing on potential fairness discrepancies within both in-distribution training sets and external test sets. Our analysis covers three key medical imaging disciplines-radiology, dermatology and ophthalmology-and incorporates data from six global chest X-ray datasets. We confirm that medical imaging AI leverages demographic shortcuts in disease classification. Although correcting shortcuts algorithmically effectively addresses fairness gaps to create 'locally optimal' models within the original data distribution, this optimality is not true in new test settings. Surprisingly, we found that models with less encoding of demographic attributes are often most 'globally optimal', exhibiting better fairness during model evaluation in new test environments. Our work establishes best practices for medical imaging models that maintain their performance and fairness in deployments beyond their initial training contexts, underscoring critical considerations for AI clinical deployments across populations and sites.
Title: The Neutrality Fallacy: When Algorithmic Fairness Interventions are (Not) Positive Action
Authors: Hilde Weerts; Raphaële Xenidis; Fabien Tarissan; Henrik Palmer Olsen; Mykola Pechenizkiy
Affiliations: Eindhoven University of Technology; Institut d'Etudes Politiques de Paris; École Normale Supérieure Paris-Saclay; University of Copenhagen; Eindhoven University of Technology
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Various metrics and interventions have been developed to identify and mitigate unfair outputs of machine learning systems. While individuals and organizations have an obligation to avoid discrimination, the use of fairness-aware machine learning interventions has also been described as amounting to ‘algorithmic positive action’ under European Union (EU) non-discrimination law. As the Court of Justice of the European Union has been strict when it comes to assessing the lawfulness of positive action, this would impose a significant legal burden on those wishing to implement fair-ml interventions. In this paper, we propose that algorithmic fairness interventions often should be interpreted as a means to prevent discrimination, rather than a measure of positive action. Specifically, we suggest that this category mistake can often be attributed to neutrality fallacies: faulty assumptions regarding the neutrality of (fairness-aware) algorithmic decision-making. Our findings raise the question of whether a negative obligation to refrain from discrimination is sufficient in the context of algorithmic decision-making. Consequently, we suggest moving away from a duty to ‘not do harm’ towards a positive obligation to actively ‘do no harm’ as a more adequate framework for algorithmic decision-making and fair ml-interventions.
Title: Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment
Authors: Karen Drukker; Weijie Chen; Judy Wawira Gichoya; Nicholas P. Gruszauskas; Jayashree Kalpathy–Cramer; Oluwasanmi Koyejo; Kyle J. Myers; Rui Carlos Sá; Berkman Sahiner; Heather M. Whitney; Zi Zhang; Maryellen L. Giger
Affiliations: University of Chicago; Center for Devices and Radiological Health; Emory University; University of Chicago; University of Colorado Anschutz Medical Campus; Stanford University; Arizona Department of Education; University of California, San Diego; United States Food and Drug Administration; University of Chicago; Thomas Jefferson University Hospital; University of Chicago
Year: 2023
Domain: Health & Clinical AI

Abstract: Purpose: To recognize and address various sources of bias essential for algorithmic fairness and trustworthiness and to contribute to a just and equitable deployment of AI in medical imaging, there is an increasing interest in developing medical imaging-based machine learning methods, also known as medical imaging artificial intelligence (AI), for the detection, diagnosis, prognosis, and risk assessment of disease with the goal of clinical implementation. These tools are intended to help improve traditional human decision-making in medical imaging. However, biases introduced in the steps toward clinical deployment may impede their intended function, potentially exacerbating inequities. Specifically, medical imaging AI can propagate or amplify biases introduced in the many steps from model inception to deployment, resulting in a systematic difference in the treatment of different groups. Approach: Our multi-institutional team included medical physicists, medical imaging artificial intelligence/machine learning (AI/ML) researchers, experts in AI/ML bias, statisticians, physicians, and scientists from regulatory bodies. We identified sources of bias in AI/ML, mitigation strategies for these biases, and developed recommendations for best practices in medical imaging AI/ML development. Results: Five main steps along the roadmap of medical imaging AI/ML were identified: (1) data collection, (2) data preparation and annotation, (3) model development, (4) model evaluation, and (5) model deployment. Within these steps, or bias categories, we identified 29 sources of potential bias, many of which can impact multiple steps, as well as mitigation strategies. Conclusions: Our findings provide a valuable resource to researchers, clinicians, and the public at large.
Title: Towards Fair Graph Neural Networks via Graph Counterfactual
Authors: Zhimeng Guo; Jialiang Li; Teng Xiao; Yao Ma; Suhang Wang
Affiliations: Pennsylvania State University; New Jersey Institute of Technology; Pennsylvania State University; Rensselaer Polytechnic Institute; Pennsylvania State University
Year: 2023
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Graph neural networks have shown great ability in representation (GNNs) learning on graphs, facilitating various tasks. Despite their great performance in modeling graphs, recent works show that GNNs tend to inherit and amplify the bias from training data, causing concerns of the adoption of GNNs in high-stake scenarios. Hence, many efforts have been taken for fairness-aware GNNs. However, most existing fair GNNs learn fair node representations by adopting statistical fairness notions, which may fail to alleviate bias in the presence of statistical anomalies. Motivated by causal theory, there are several attempts utilizing graph counterfactual fairness to mitigate root causes of unfairness. However, these methods suffer from non-realistic counterfactuals obtained by perturbation or generation. In this paper, we take a causal view on fair graph learning problem. Guided by the casual analysis, we propose a novel framework CAF, which can select counterfactuals from training data to avoid non-realistic counterfactuals and adopt selected counterfactuals to learn fair node representations for node classification task. Extensive experiments on synthetic and real-world datasets show the effectiveness of CAF. Our code is available at https://github.com/TimeLovercc/CAF-GNN.
Title: Using the electronic health record to provide audit and feedback in medical student clerkships
Authors: Jacqueline Xu; Matthew Silver; Jung Kim; Lindsay Mazotti
Affiliations: Kaiser Permanente; Kaiser Permanente; Kaiser Permanente; Sutter Health
Year: 2024
Domain: Health & Clinical AI

Abstract: Objectives: This article focuses on the role of the electronic health record (EHR) to generate meaningful formative feedback for medical students in the clinical setting. Despite the scores of clinical data housed within the EHR, medical educators have only just begun to tap into this data to enhance student learning. Literature to-date has focused almost exclusively on resident education. Materials and Methods: Development of EHR auto-logging and triggered notifications are discussed as specific use cases in providing enhanced feedback for medical students. Results: By incorporating predictive and prescriptive analytics into the EHR, there is an opportunity to create powerful educational tools which may also support general clinical activity. Discussion: This article explores the possibilities of EHR as an educational resource. This serves as a call to action for educators and technology developers to work together on creating health record user-centric tools, acknowledging the ongoing work done to improve student-level attribution to patients. Conclusion: EHR analytics and tools present a novel approach to enhancing clinical clerkship education for medical students.
Title: Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Authors: Nicolas Scharowski; Michaela Benk; Swen J. Kühne; Léane Wettstein; Florian Brühlmann
Affiliations: University of Basel; ETH Zurich; ZHAW Zurich University of Applied Sciences; University of Basel; University of Basel
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Auditing plays a pivotal role in the development of trustworthy AI. However, current research primarily focuses on creating auditable AI documentation, which is intended for regulators and experts rather than end-users affected by AI decisions. How to communicate to members of the public that an AI has been audited and considered trustworthy remains an open challenge. This study empirically investigated certification labels as a promising solution. Through interviews (N = 12) and a census-representative survey (N = 302), we investigated end-users’ attitudes toward certification labels and their effectiveness in communicating trustworthiness in low- and high-stakes AI scenarios. Based on the survey results, we demonstrate that labels can significantly increase end-users’ trust and willingness to use AI in both low- and high-stakes scenarios. However, end-users’ preferences for certification labels and their effect on trust and willingness to use AI were more pronounced in high-stake scenarios. Qualitative content analysis of the interviews revealed opportunities and limitations of certification labels, as well as facilitators and inhibitors for the effective use of labels in the context of AI. For example, while certification labels can mitigate data-related concerns expressed by end-users (e.g., privacy and data protection), other concerns (e.g., model performance) are more challenging to address. Our study provides valuable insights and recommendations for designing and implementing certification labels as a promising constituent within the trustworthy AI ecosystem.
Title: On the Impact of Explanations on Understanding of Algorithmic Decision-Making
Authors: Timothée Schmude; Laura Koesten; Torsten Möller; Sebastian Tschiatschek
Affiliations: University of Vienna; University of Vienna; University of Vienna; University of Vienna
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Ethical principles for algorithms are gaining importance as more and more stakeholders are affected by "high-risk" algorithmic decision-making (ADM) systems. Understanding how these systems work enables stakeholders to make informed decisions and to assess the systems’ adherence to ethical values. Explanations are a promising way to create understanding, but current explainable artificial intelligence (XAI) research does not always consider existent theories on how understanding is formed and evaluated. In this work, we aim to contribute to a better understanding of understanding by conducting a qualitative task-based study with 30 participants, including users and affected stakeholders. We use three explanation modalities (textual, dialogue, and interactive) to explain a "high-risk" ADM system to participants and analyse their responses both inductively and deductively, using the "six facets of understanding" framework by Wiggins & McTighe [63]. Our findings indicate that the "six facets" framework is a promising approach to analyse participants’ thought processes in understanding, providing categories for both rational and emotional understanding. We further introduce the "dialogue" modality as a valid explanation approach to increase participant engagement and interaction with the "explainer", allowing for more insight into their understanding in the process. Our analysis further suggests that individuality in understanding affects participants’ perceptions of algorithmic fairness, demonstrating the interdependence between understanding and ADM assessment that previous studies have outlined. We posit that drawing from theories on learning and understanding like the "six facets" and leveraging explanation modalities can guide XAI research to better suit explanations to learning processes of individuals and consequently enable their assessment of ethical values of ADM systems.