cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: A Systematic Review of Ethics Disclosures in Predictive Mental Health Research
Authors: Leah Ajmani; Stevie Chancellor; Bijal Mehta; Casey Fiesler; Michael Zimmer; Munmun De Choudhury
Affiliations: University of Minnesota; University of Minnesota; Georgetown University; University of Colorado Boulder; Marquette University; Georgia Institute of Technology
Year: 2023
Domain: Health & Clinical AI

Abstract: Applied machine learning (ML) has not yet coalesced on standard practices for research ethics. For ML that predicts mental illness using social media data, ambiguous ethical standards can impact peoples’ lives because of the area’s sensitivity and material consequences on health. Transparency of current ethics practices in research is important to document decision-making and improve research practice. We present a systematic literature review of 129 studies that predict mental illness using social media data and ML, and the ethics disclosures they make in research publications. Rates of disclosure are going up over time, but this trend is slow moving – it will take another eight years for the average paper to have coverage on 75% of studied ethics categories. Certain practices are more readily adopted, or "stickier", over time, though we found prioritization of data-driven disclosures rather than human-centered. These inconsistently reported ethical considerations indicate a gap between what ML ethicists believe ought to be and what actually is done. We advocate for closing this gap through increased transparency of practice and formal mechanisms to support disclosure.
Title: WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?
Authors: Ali Akbar Septiandri; Marios Constantinides; Mohammad Tahaei; Daniele Quercia
Affiliations: Nokia (United Kingdom); Nokia (United Kingdom); Nokia (United Kingdom); Nokia (United Kingdom)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Studies conducted on Western, Educated, Industrialized, Rich, and Democratic (WEIRD) samples are considered atypical of the world’s population and may not accurately represent human behavior. In this study, we aim to quantify the extent to which the ACM FAccT conference, the leading venue in exploring Artificial Intelligence (AI) systems’ fairness, accountability, and transparency, relies on WEIRD samples. We collected and analyzed 128 papers published between 2018 and 2022, accounting for 30.8% of the overall proceedings published at FAccT in those years (excluding abstracts, tutorials, and papers without human-subject studies or clear country attribution for the participants). We found that 84% of the analyzed papers were exclusively based on participants from Western countries, particularly exclusively from the U.S. (63%). Only researchers who undertook the effort to collect data about local participants through interviews or surveys added diversity to an otherwise U.S.-centric view of science. Therefore, we suggest that researchers collect data from under-represented populations to obtain an inclusive worldview. To achieve this goal, scientific communities should champion data collection from such populations and enforce transparent reporting of data biases.
Title: Diagnosing AI Explanation Methods with Folk Concepts of Behavior
Authors: Alon Jacovi; Jasmijn Bastings; Sebastian Gehrmann; Yoav Goldberg; Katja Filippova
Affiliations: Bar-Ilan University; Google (United States); Bar-Ilan University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: We investigate a formalism for the conditions of a successful explanation of AI. We consider “success” to depend not only on what information the explanation contains, but also on what information the human explainee understands from it. Theory of mind literature discusses the folk concepts that humans use to understand and generalize behavior. We posit that folk concepts of behavior provide us with a “language” that humans understand behavior with. We use these folk concepts as a framework of social attribution by the human explainee—the information constructs that humans are likely to comprehend from explanations—by introducing a blueprint for an explanatory narrative that explains AI behavior with these constructs. We then demonstrate that many XAI methods today can be mapped to folk concepts of behavior in a qualitative evaluation. This allows us to uncover their failure modes that prevent current methods from explaining successfully—i.e., the information constructs that are missing for any given XAI method, and whose inclusion can decrease the likelihood of misunderstanding AI behavior.
Title: How to Explain and Justify Almost Any Decision: Potential Pitfalls for Accountability in AI Decision-Making
Authors: Joyce Zhou; Thorsten Joachims
Affiliations: Cornell University; Cornell University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Discussion of the “right to an explanation” has been increasingly relevant because of its potential utility for auditing automated decision systems, as well as for making objections to such decisions. However, most existing work on explanations focuses on collaborative environments, where designers are motivated to implement good-faith explanations that reveal potential weaknesses of a decision system. This motivation may not hold in an auditing environment. Thus, we ask: how much could explanations be used maliciously to defend a decision system? In this paper, we demonstrate how a black-box explanation system developed to defend a black-box decision system could manipulate decision recipients or auditors into accepting an intentionally discriminatory decision model. In a case-by-case scenario where decision recipients are unable to share their cases and explanations, we find that most individual decision recipients could receive a verifiable justification, even if the decision system is intentionally discriminatory. In a system-wide scenario where every decision is shared, we find that while justifications frequently contradict each other, there is no intuitive threshold to determine if these contradictions are because of malicious justifications or because of simplicity requirements of these justifications conflicting with model behavior. We end with discussion of how system-wide metrics may be more useful than explanation systems for evaluating overall decision fairness, while explanations could be useful outside of fairness auditing.
Title: Explainable AI is Dead, Long Live Explainable AI!
Authors: Tim Miller
Affiliations: The University of Melbourne
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In this paper, we argue for a paradigm shift from the current model of explainable artificial intelligence (XAI), which may be counter-productive to better human decision making. In early decision support systems, we assumed that we could give people recommendations and that they would consider them, and then follow them when required. However, research found that people often ignore recommendations because they do not trust them; or perhaps even worse, people follow them blindly, even when the recommendations are wrong. Explainable artificial intelligence mitigates this by helping people to understand how and why models give certain recommendations. However, recent research shows that people do not always engage with explainability tools enough to help improve decision making. The assumption that people will engage with recommendations and explanations has proven to be unfounded. We argue this is because we have failed to account for two things. First, recommendations (and their explanations) take control from human decision makers, limiting their agency. Second, giving recommendations and explanations does not align with the cognitive processes employed by people making decisions. This position paper proposes a new conceptual framework called Evaluative AI for explainable decision support. This is a machine-in-the-loop paradigm in which decision support tools provide evidence for and against decisions made by people, rather than provide recommendations to accept or reject. We argue that this mitigates issues of over- and under-reliance on decision support tools, and better leverages human expertise in decision making.
Title: Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UK
Authors: Luca Nannini; Agathe Balayn; Adam Leon Smith
Affiliations: Universidade de Santiago de Compostela; Delft University of Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Public attention towards explainability of artificial intelligence (AI) systems has been rising in recent years to offer methodologies for human oversight. This has translated into the proliferation of research outputs, such as from Explainable AI, to enhance transparency and control for system debugging and monitoring, and intelligibility of system process and output for user services. Yet, such outputs are difficult to adopt on a practical level due to a lack of a common regulatory baseline, and the contextual nature of explanations. Governmental policies are now attempting to tackle such exigence, however it remains unclear to what extent published communications, regulations, and standards adopt an informed perspective to support research, industry, and civil interests. In this study, we perform the first thematic and gap analysis of this plethora of policies and standards on explainability in the EU, US, and UK. Through a rigorous survey of policy documents, we first contribute an overview of governmental regulatory trajectories within AI explainability and its sociotechnical impacts. We find that policies are often informed by coarse notions and requirements for explanations. This might be due to the willingness to conciliate explanations foremost as a risk management tool for AI oversight, but also due to the lack of a consensus on what constitutes a valid algorithmic explanation, and how feasible the implementation and deployment of such explanations are across stakeholders of an organization. Informed by AI explainability research, we then conduct a gap analysis of existing policies, which leads us to formulate a set of recommendations on how to address explainability in regulations for AI systems, especially discussing the definition, feasibility, and usability of explanations, as well as allocating accountability to explanation providers.
Title: An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature
Authors: Amina A. Abdu; Irene V. Pasquetto; Abigail Z. Jacobs
Affiliations: University of Michigan; University of Michigan; University of Michigan
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Recent work in algorithmic fairness has highlighted the challenge of defining racial categories for the purposes of anti-discrimination. These challenges are not new but have previously fallen to the state, which enacts race through government statistics, policies, and evidentiary standards in anti-discrimination law. Drawing on the history of state race-making, we examine how longstanding questions about the nature of race and discrimination appear within the algorithmic fairness literature. Through a content analysis of 60 papers published at FAccT between 2018 and 2020, we analyze how race is conceptualized and formalized in algorithmic fairness frameworks. We note that differing notions of race are adopted inconsistently, at times even within a single analysis. We also explore the institutional influences and values associated with these choices. While we find that categories used in algorithmic fairness work often echo legal frameworks, we demonstrate that values from academic computer science play an equally important role in the construction of racial categories. Finally, we examine the reasoning behind different operationalizations of race, finding that few papers explicitly describe their choices and even fewer justify them. We argue that the construction of racial categories is a value-laden process with significant social and political consequences for the project of algorithmic fairness. The widespread lack of justification around the operationalization of race reflects institutional norms that allow these political decisions to remain obscured within the backstage of knowledge production.
Title: How Redundant are Redundant Encodings? Blindness in the Wild and Racial Disparity when Race is Unobserved
Authors: Lingwei Cheng; Isabel Gallegos; Derek Ouyang; Jacob Goldin; Daniel E. Ho
Affiliations: Carnegie Mellon University; Stanford University; Stanford University; University of Chicago; Stanford University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: We address two emerging concerns in algorithmic fairness: (i) redundant encodings of race – the notion that machine learning models encode race with probability nearing one as the feature set grows – which is widely noted in theory, with little empirical evidence; and (ii) the lack of race and ethnicity data in many domains, where state-of-the-art remains (Naive) Bayesian Improved Surname Geocoding (BISG) that relies on name and geographic information. We leverage a novel and highly granular dataset of over 7.7 million patients’ electronic health records to provide one of the first empirical studies of redundant encodings in a realistic health care setting and examine the ability to assess health care disparities when race may be missing. First, we show that machine learning (random forest) applied to name and geographic information can improve on BISG, driven primarily by better performance in identifying minority groups. Second, contrary to theoretical concerns about redundant encodings as undercutting anti-discrimination law’s anti-classification principle, additional electronic health information provides little marginal information about race and ethnicity: race still remains measured with substantial noise. Third, we show how machine learning can enable the disaggregation of racial categories, responding to longstanding critiques of the government race reporting standard. Fourth, we show that an increasing feature set can differentially impact performance on majority and minority groups. Our findings address important questions for fairness in machine learning and algorithmic decision-making, enabling the assessment of disparities, tempering concerns about redundant encodings in one important setting, and demonstrating how bigger data can shape the accuracy of race imputations in nuanced ways.
Title: Bias as Boundary Object: Unpacking The Politics Of An Austerity Algorithm Using Bias Frameworks
Authors: Gabriel Grill; Fabian Fischer; Florian Cech
Affiliations: University of Michigan; University of Applied Arts Vienna; TU Wien
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Whether bias is an appropriate lens for analysis and critique remains a subject of debate among scholars. This paper contributes to this conversation by unpacking the use of bias in a critical analysis of a controversial austerity algorithm introduced by the Austrian public employment service in 2018. It was envisioned to classify the unemployed into three risk categories based on predicted prospects for re-employment. The system promised to increase efficiency and effectivity of counseling while objectifying a new austerity support measure allocation scheme. This approach was intended to cut spending for those deemed at highest risk of long term unemployment. Our in-depth analysis, based on internal documentation not available to the public, systematically traces and categorizes various problematic biases to illustrate harms to job seekers and challenge promises used to justify the adoption of the system. The classification is guided by a long-established bias framework for computer systems developed by Friedman and Nissenbaum, which provides three sensitizing basic categories. We identified in our analysis "technical biases," like issues around measurement, rigidity, and coarseness of variables, "emergent biases," such as disruptive events that change the labor market, and, finally, "preexisting biases," like the use of variables that act as proxies for inequality.
Title: Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets
Authors: Teanna Barrett; Quan Ze Chen; Amy X. Zhang
Affiliations: Howard University; University of Washington; University of Washington
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: To investigate the well-observed racial disparities in computer vision systems that analyze images of humans, researchers have turned to skin tone as a more objective annotation than race metadata for fairness performance evaluations. However, the current state of skin tone annotation procedures is highly varied. For instance, researchers use a range of untested scales and skin tone categories, have unclear annotation procedures, and provide inadequate analyses of uncertainty. In addition, little attention is paid to the positionality of the humans involved in the annotation process—both designers and annotators alike—and the historical and sociological context of skin tone in the United States. Our work is the first to investigate the skin tone annotation process as a sociotechnical project. We surveyed recent skin tone annotation procedures and conducted annotation experiments to examine how subjective understandings of skin tone are embedded in skin tone annotation procedures. Our systematic literature review revealed the uninterrogated association between skin tone and race and the limited effort to analyze annotator uncertainty in current procedures for skin tone annotation in computer vision evaluation. Our experiments demonstrated that design decisions in the annotation procedure such as the order in which the skin tone scale is presented or additional context in the image (i.e., presence of a face) significantly affected the resulting inter-annotator agreement and individual uncertainty of skin tone annotations. We call for greater reflexivity in the design, analysis, and documentation of procedures for evaluation using skin tone.
Title: Attitudes Toward Facial Analysis AI: A Cross-National Study Comparing Argentina, Kenya, Japan, and the USA
Authors: Chiara Ullstein; Severin Engelmann; Orestis Papakyriakopoulos; Yuko Ikkatai; Naira Paola Arnez-Jordan; Rose Caleno; Brian Mboya; Shuichiro Higuma; Tilman Hartwig; H. Yokoyama; Jens Großklags
Affiliations: Technical University of Munich; Cornell University; Technical University of Munich; Kanazawa University; Technical University of Munich; Dedan Kimathi University of Technology; The University of Tokyo; German Environment Agency; The University of Tokyo; Technical University of Munich
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Computer vision AI systems present one of the most radical technical transformations of our time. Such systems are given unparalleled epistemic power to impose meaning on visual data, despite their inherent semantic ambiguity. This epistemic power is particularly evident in computer vision AI that interprets the meaning of human faces. The goal of this work is to empirically document laypeople’s perceptions of the epistemic and ethical complexity of computer vision AI through a large-scale qualitative study with participants in Argentina, Japan, Kenya, and the USA (N=4,468). We developed a vignette scenario about a fictitious company that analyzes people’s portraits using computer vision AI to make a variety of inferences about people based on their faces. For each inference that the fictitious company draws (e.g., age, skin color, intelligence), we ask participants from all countries to reason about how they evaluate computer vision AI inference-making. In a series of workshops, we collaborated as a multinational research team to develop a codebook that captures people’s different justifications of facial analysis AI inferences to create a comprehensive justification portfolio. Our study reveals similarities in justification patterns, but also significant intra-country and inter-country diversity in response to different facial inferences. For example, participants from Argentina, Japan, Kenya, and the USA vastly disagree over the reasonableness of AI classifications such as beautiful or skin color. They tend to agree in their opposition to AI-drawn inferences intelligence and trustworthiness. Adding much-needed non-Western perspectives to debates on computer vision ethics, our results suggest that, contrary to popular justifications for facial classification technologies, there is no such thing as a “common sense” facial classification that accords simply with a general, homogeneous “human intuition.”
Title: Learning Fairness from Demonstrations via Inverse Reinforcement Learning
Authors: Jack Blandin; Ian A. Kash
Affiliations: University of Illinois Chicago; University of Illinois Chicago
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Defining fairness in algorithmic contexts is challenging, particularly when adapting to new domains. Our research introduces a novel method for learning and applying group fairness preferences across different classification domains, without the need for manual fine-tuning. Utilizing concepts from inverse reinforcement learning (IRL), our approach enables the extraction and application of fairness preferences from human experts or established algorithms. We propose the first technique for using IRL to recover and adapt group fairness preferences to new domains, offering a low-touch solution for implementing fair classifiers in settings where expert-established fairness tradeoffs are not yet defined.
Title: Meaningful Transparency for Clinicians: Operationalising HCXAI Research with Gynaecologists
Authors: Bianca G. S. Schor; Emma Kallina; Jatinder Singh; Alan F. Blackwell
Affiliations: University of Cambridge; University of Cambridge; University of Cambridge; University of Cambridge
Year: 2024
Domain: Health & Clinical AI

Abstract: AI systems can bring great benefits to our healthcare systems, e.g. by improving patient outcomes. Yet implementing them into clinical practice remains challenging. To bridge the gap between academic research and design implementation, we argue clinicians need transparency about such systems that is meaningful—i.e. contextually appropriate—to them. Towards this, we explore recent HCXAI recommendations for building transparent AI systems for users in a specific domain: gynaecology. By better understanding clinicians’ perspectives on meaningful transparency, our aim is to complement and help operationalise such recommendations. We conduct a co-design workshop and interviews with n=15 gynaecologists in the UK and the Netherlands. We show that HCXAI must better account for clinical teams with different types of gynaecologist users, and that the timeliness and relevance of the information provided about the AI-based tool throughout its design lifecycle—in particular before a tool is implemented into clinical practice—is crucial for transparency to become meaningful. Our contributions include: i) testing recommendations from the latest HCXAI literature with a prospective, real-life AI application in a relatively less-studied clinical domain; ii) describing and visualising gynaecologists’ understanding of meaningful transparency for clinicians; iii) outlining four design recommendations towards realising meaningful transparency for clinicians and opportunities for research; and iv) expanding HCI and AI research in women’s health by directly engaging with gynaecologists as users and co-designers. Exploring such issues is key to facilitate the implementation of AI systems that meet clinicians’ information needs and that they can trust.