cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies
Authors: Vivian Lai; Chacha Chen; Alison Smith; Q. Vera Liao; Chenhao Tan
Affiliations: University of Colorado Boulder; University of Chicago; Dataminr (United States); Microsoft (Canada); University of Chicago
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: AI systems are adopted in numerous domains due to their increasingly strong predictive performance. However, in high-stakes domains such as criminal justice and healthcare, full automation is often not desirable due to safety, ethical, and legal concerns, yet fully manual approaches can be inaccurate and time-consuming. As a result, there is growing interest in the research community to augment human decision making with AI assistance. Besides developing AI technologies for this purpose, the emerging field of human-AI decision making must embrace empirical approaches to form a foundational understanding of how humans interact and work with AI to make decisions. To invite and help structure research efforts towards a science of understanding and improving human-AI decision making, we survey recent literature of empirical human-subject studies on this topic. We summarize the study design choices made in over 100 papers in three important aspects: (1) decision tasks, (2) AI assistance elements, and (3) evaluation metrics. For each aspect, we summarize current trends, discuss gaps in current practices of the field, and make a list of recommendations for future research. Our work highlights the need to develop common frameworks to account for the design and research spaces of human-AI decision making, so that researchers can make rigorous choices in study design, and the research community can build on each other’s work and produce generalizable scientific knowledge. We also hope this work will serve as a bridge for HCI and AI communities to work together to mutually shape the empirical science and computational technologies for human-AI decision making.
Title: Machine Explanations and Human Understanding
Authors: Chacha Chen; Shi Feng; Amit Sharma; Chenhao Tan
Affiliations: University of Chicago; University of Chicago; Microsoft (United States); University of Chicago
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even negative results. An open question, therefore, is under what conditions explanations can improve human understanding and in what way. To address this question, we first identify three core concepts that cover most existing quantitative measures of understanding: task decision boundary, model decision boundary, and model error. Using adapted causal diagrams, we provide a formal characterization of the relationship between these concepts and human approximations (i.e., understanding) of them. The relationship varies by the level of human intuition in different task types, such as emulation and discovery, which are often ignored when building or evaluating explanation methods. Our key result is that human intuitions are necessary for generating and evaluating machine explanations in human-AI decision making: without assumptions about human intuitions, explanations may improve human understanding of model decision boundary, but cannot improve human understanding of task decision boundary or model error. To validate our theoretical claims, we conduct human subject studies to show the importance of human intuitions. Together with our theoretical contributions, we provide a new paradigm for designing behavioral studies towards a rigorous view of the role of machine explanations across different tasks of human-AI decision making.
Title: Humans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application
Authors: Sunnie S. Y. Kim; Elizabeth Anne Watkins; Olga Russakovsky; Ruth Fong; Andrés Monroy‐Hernández
Affiliations: Princeton University; Intelligent Systems Research (United States); Princeton University; Princeton University; Princeton University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Trust is an important factor in people’s interactions with AI systems. However, there is a lack of empirical studies examining how real end-users trust or distrust the AI system they interact with. Most research investigates one aspect of trust in lab settings with hypothetical end-users. In this paper, we provide a holistic and nuanced understanding of trust in AI through a qualitative case study of a real-world computer vision application. We report findings from interviews with 20 end-users of a popular, AI-based bird identification app where we inquired about their trust in the app from many angles. We find participants perceived the app as trustworthy and trusted it, but selectively accepted app outputs after engaging in verification behaviors, and decided against app adoption in certain high-stakes scenarios. We also find domain knowledge and context are important factors for trust-related assessment and decision-making. We discuss the implications of our findings and provide recommendations for future research on trust in AI.
Title: The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice
Authors: Andrew Bell; Lucius Bynum; Nazarii Drushchak; Tetiana Zakharchenko; Lucas Rosenblatt; Julia Stoyanovich
Affiliations: New York University; New York University; Ukrainian Catholic University; Ukrainian Catholic University; New York University; New York University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The “impossibility theorem” — which is considered foundational in algorithmic fairness literature — asserts that there must be trade-offs between common notions of fairness and performance when fitting statistical models, except in two special cases: when the prevalence of the outcome being predicted is equal across groups, or when a perfectly accurate predictor is used. However, theory does not always translate to practice. In this work, we challenge the implications of the impossibility theorem in practical settings. First, we show analytically that, by slightly relaxing the impossibility theorem (to accommodate a practitioner’s perspective of fairness), it becomes possible to identify abundant sets of models that satisfy seemingly incompatible fairness constraints. Second, we demonstrate the existence of these models through extensive experiments on five real-world datasets. We conclude by offering tools and guidance for practitioners to understand when — and to what degree — fairness along multiple criteria can be achieved. This work has an important implication for the community: achieving fairness along multiple metrics for multiple groups (and their intersections) is much more possible than was previously believed.
Title: Maximal fairness
Authors: Marybeth Defrance; Tijl De Bie
Affiliations: University of Ghent; University of Ghent
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Fairness in AI has garnered quite some attention in research, and increasingly also in society. The so-called "Impossibility Theorem" has been one of the more striking research results with both theoretical and practical consequences, as it states that satisfying a certain combination of fairness measures is impossible. To date, this negative result has not yet been complemented with a positive one: a characterization of which combinations of fairness notions are possible. This work aims to fill this gap by identifying maximal sets of commonly used fairness measures that can be simultaneously satisfied. The fairness measures used are demographic parity, equal opportunity, predictive equality, predictive parity, false omission rate parity, overall accuracy equality and treatment equality. We conclude that in total 12 maximal sets of these fairness measures are possible, among which are seven combinations of two measures, and five combinations of three measures. Our work raises interesting questions regarding the practical relevance of each of these 12 maximal fairness notions in various scenarios.
Title: Enhancing AI fairness through impact assessment in the European Union: a legal and computer science perspective
Authors: Alessandra Calvi; Dimitris Kotzinos
Affiliations: Centre National de la Recherche Scientifique; Centre National de la Recherche Scientifique
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: How to protect people from algorithmic harms? A promising solution, although in its infancy, is algorithmic impact assessment (AIA). AIAs are iterative processes used to investigate the possible short and long terms societal impacts of AI systems before their use, but with ongoing monitoring and periodic revisiting even after their implementation. When conducted in a participatory and transparent fashion, they could create bridges across the legal, social and computer science domains, promoting the accountability of the entity performing them as well as public scrutiny. They could enable to re-attach the societal and regulatory context to the mathematical definition of fairness, thus expanding the formalistic approach thereto. Whilst the regulatory framework in the European Union currently lacks the obligation to perform such AIA, some other provisions are expected to play a role in AI development, leading the way towards more widespread adoption of AIA. These include the Data Protection Impact Assessment (DPIA) under the General Data Protection Regulation (GDPR), the risk assessment process under the Digital Services Act (DSA) and the Conformity Assessment (CA) foreseen under the AI Regulation proposal.
Title: Help or Hinder? Evaluating the Impact of Fairness Metrics and Algorithms in Visualizations for Consensus Ranking
Authors: Hilson Shrestha; Kathleen Cachel; Mallak Alkhathlan; Elke A. Rundensteiner; Lane Harrison
Affiliations: Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute; Worcester Polytechnic Institute
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: For applications where multiple stakeholders provide recommendations, a fair consensus ranking must not only ensure that the preferences of rankers are well represented, but must also mitigate disadvantages among socio-demographic groups in the final result. However, there is little empirical guidance on the value or challenges of visualizing and integrating fairness metrics and algorithms into human-in-the-loop systems to aid decision-makers. In this work, we design a study to analyze the effectiveness of integrating such fairness metrics-based visualization and algorithms. We explore this through a task-based crowdsourced experiment comparing an interactive visualization system for constructing consensus rankings, ConsensusFuse, with a similar system that includes visual encodings of fairness metrics and fair-rank generation algorithms, FairFuse. We analyze the measure of fairness, agreement of rankers’ decisions, and user interactions in constructing the fair consensus ranking across these two systems. In our study with 200 participants, results suggest that providing these fairness-oriented support features nudges users to align their decision with the fairness metrics while minimizing the tedious process of manually having to amend the consensus ranking. We discuss the implications of these results for the design of next-generation fairness oriented-systems and along with emerging directions for future research.
Title: Simplicity Bias Leads to Amplified Performance Disparities
Authors: Samuel J. Bell; Levent Sagun
Affiliations: FAIR, Meta AI; FAIR, Meta AI
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Which parts of a dataset will a given model find difficult? Recent work has shown that SGD-trained models have a bias towards simplicity, leading them to prioritize learning a majority class, or to rely upon harmful spurious correlations. Here, we show that the preference for ‘easy’ runs far deeper: A model may prioritize any class or group of the dataset that it finds simple—at the expense of what it finds complex—as measured by performance difference on the test set. When subsets with different levels of complexity align with demographic groups, we term this difficulty disparity, a phenomenon that occurs even with balanced datasets that lack group/label associations. We show how difficulty disparity is a model-dependent quantity, and is further amplified in commonly-used models as selected by typical average performance scores. We quantify an amplification factor across a range of settings in order to compare disparity of different models on a fixed dataset. Finally, we present two real-world examples of difficulty amplification in action, resulting in worse-than-expected performance disparities between groups even when using a balanced dataset. The existence of such disparities in balanced datasets demonstrates that merely balancing sample sizes of groups is not sufficient to ensure unbiased performance. We hope this work presents a step towards measurable understanding of the role of model bias as it interacts with the structure of data, and call for additional model-dependent mitigation methods to be deployed alongside dataset audits.
Title: A Sociotechnical Audit: Assessing Police Use of Facial Recognition
Authors: Evani Radiya-Dixit; Gina Neff
Affiliations: University of Cambridge; University of Cambridge
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic audits are increasingly used to hold people accountable for the algorithms they implement. However, much work remains to integrate ethical and legal evaluations of how algorithms are used into audits. In this paper, we present a sociotechnical audit to help external stakeholders evaluate the ethics and legality of police use of facial recognition technology. We developed this audit for the specific legal context of England and Wales, and to bring attention to broader concerns such as whether police consult affected communities and comply with human rights law. To design this audit, we compiled ethical and legal standards for governing facial recognition, based on existing literature and feedback from academia, government, civil society, and police organizations. We then applied the resulting audit tool to three facial recognition deployments by police forces in the UK and found that all three failed to meet these standards. Developing this audit helps us provide insights to researchers in designing their own sociotechnical audits, specifically how audits shift power, how to make audits context-specific, how audits reveal what is not transparent, and how audits lead to accountability.
Title: Which Stereotypes Are Moderated and Under-Moderated in Search Engine Autocompletion?
Authors: Alina Leidinger; Richard Rogers
Affiliations: University of Amsterdam; University of Amsterdam
Year: 2023
Domain: LLMs & NLP

Abstract: Language technologies that perpetuate stereotypes actively cement social hierarchies. This study enquires into the moderation of stereotypes in autocompletion results by Google, DuckDuckGo and Yahoo! We investigate the moderation of derogatory stereotypes for social groups, examining the content and sentiment of the autocompletions. We thereby demonstrate which categories are highly moderated (i.e., sexual orientation, religious affiliation, political groups and communities or peoples) and which less so (age and gender), both overall and per engine. We found that under-moderated categories contain results with negative sentiment and derogatory stereotypes. We also identify distinctive moderation strategies per engine, with Google and DuckDuckGo moderating greatly and Yahoo! being more permissive. The research has implications for both moderation of stereotypes in commercial autocompletion tools, as well as large language models in NLP, particularly the question of the content deserving of moderation.
Title: AI Regulation Is (not) All You Need
Authors: Laura Lucaj; Patrick van der Smagt; Djalel Benbouzid
Affiliations: Volkswagen Group (Germany); Volkswagen Group (Germany); Volkswagen Group (Germany)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The development of processes and tools for ethical, trustworthy, and legal AI is only beginning. At the same time, legal requirements are emerging in various jurisdictions, following a deluge of ethical guidelines. It is therefore key to explore the necessary practices that must be adopted to ensure the quality of AI systems, mitigate their potential risks and enable legal compliance. Ensuring that the potential negative impacts of AI on individuals, society, and the environment are mitigated will depend on many factors, including the capacity to properly regulate its deployment and to mandate necessary internal best practices along lifecycles. Regulatory frameworks must evolve from abstract requirements to providing concrete operational mandates that enable better oversight mechanisms in the way AI systems operate, how they are developed, and how they are deployed. In view of the above, this paper explores the necessary practices that can be adopted throughout a comprehensive lifecycle audit as a key practice to ensure the quality of AI systems and enable the development of compliance mechanisms. It also discusses novel governance tools that enable bridging the current operational gaps. Such gaps were identified by interviewing experts, analysing adaptable tools and methodologies from the software engineering domain, and by exploring the state of the art of auditing. The results present recommendations for novel tools and oversight mechanisms for governing AI systems.
Title: Examining risks of racial biases in NLP tools for child protective services
Authors: Anjalie Field; Amanda Coston; Nupoor Gandhi; Alexandra Chouldechova; Emily Putnam‐Hornstein; David Steier; Yulia Tsvetkov
Affiliations: Johns Hopkins University; Carnegie Mellon University; Carnegie Mellon University; Microsoft (United States); University of North Carolina at Chapel Hill; Carnegie Mellon University; University of Washington
Year: 2023
Domain: LLMs & NLP

Abstract: Although much literature has established the presence of demographic bias in natural language processing (NLP) models, most work relies on curated bias metrics that may not be reflective of real-world applications. At the same time, practitioners are increasingly using algorithmic tools in high-stakes settings, with particular recent interest in NLP. In this work, we focus on one such setting: child protective services (CPS). CPS workers often write copious free-form text notes about families they are working with, and CPS agencies are actively seeking to deploy NLP models to leverage these data. Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities. We specifically examine word statistics within notes and algorithmic fairness in risk prediction, coreference resolution, and named entity recognition (NER). We document consistent algorithmic unfairness in NER models, possible algorithmic unfairness in coreference resolution models, and little evidence of exacerbated racial bias in risk prediction. While there is existing pronounced criticism of risk prediction, our results expose previously undocumented risks of racial bias in realistic information extraction systems, highlighting potential concerns in deploying them, even though they may appear more benign. Our work serves as a rare realistic examination of NLP algorithmic fairness in a potential deployed setting and a timely investigation of a specific risk associated with deploying NLP in CPS settings.
Title: Interrogating the T in FAccT
Authors: Eric Corbett; Emily Denton
Affiliations: Google (United States); Google (United States)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Fairness, accountability, and transparency are the three conceptual foundations of the FAccT conference. Transparency, however, has yet to be scrutinized to the same degree as accountability and fairness. As a result, we don't know: What does this community mean when it talks about transparency? How are we doing transparency? And to what ends? What commitments does (or should) the T in FAccT signify? This paper interrogates the T in FAccT using perspectives from critical transparency literature. Subsequently, we argue that FAccT might be better off dropping the T from its title for two reasons: (1) transparency can often be counterproductive to FAccT's primary objectives and (2) it is misleading as FAccT is mainly preoccupied with explainability rather than actual transparency. If we want to keep the T, we need to reframe how we think about and do transparency by making transparency contingent, reclaiming it from explainability, and bringing people into transparency processes.
Title: Representation in AI Evaluations
Authors: A. S. Bergman; Lisa Anne Hendricks; Maribeth Rauh; Boxi Wú; William Agnew; Markus Kunesch; Isabella Duan; Iason Gabriel; William Isaac
Affiliations: Google DeepMind (United Kingdom); Google DeepMind (United Kingdom); Google DeepMind (United Kingdom); University of Washington; Google DeepMind (United Kingdom); University of Chicago; Google DeepMind (United Kingdom); Google DeepMind (United Kingdom)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Calls for representation in artificial intelligence (AI) and machine learning (ML) are widespread, with "representation" or "representativeness" generally understood to be both an instrumentally and intrinsically beneficial quality of an AI system, and central to fairness concerns. But what does it mean for an AI system to be "representative"? Each element of the AI lifecycle is geared towards its own goals and effect on the system, therefore requiring its own analyses with regard to what kind of representation is best. In this work we untangle the benefits of representation in AI evaluations to develop a framework to guide an AI practitioner or auditor towards the creation of representative ML evaluations. Representation, however, is not a panacea. We further lay out the limitations and tensions of instrumentally representative datasets, such as the necessity of data existence and access, surveillance vs expectations of privacy, implications for foundation models and power. This work sets the stage for a research agenda on representation in AI, which extends beyond instrumentally valuable representation in evaluations towards refocusing on, and empowering, impacted communities.