cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Fair Clustering via Equitable Group Representations
Authors: Mohsen Abbasi; Aditya Bhaskara; Suresh Venkatasubramanian
Affiliations: University of Utah; University of Utah; University of Utah
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: What does it mean for a clustering to be fair? One popular approach seeks to ensure that each cluster contains groups in (roughly) the same proportion in which they exist in the population. The normative principle at play is balance: any cluster might act as a representative of the data, and thus should reflect its diversity.
Title: Narratives and Counternarratives on Data Sharing in Africa
Authors: Rediet Abebe; Kehinde Aruleba; Abeba Birhane; Sara Kingsley; George Obaido; Sekou L. Remy; Swathi Sadagopan
Affiliations: University of California Berkeley; University of the Witwatersrand; University College Dublin; Carnegie Mellon University; University of the Witwatersrand; IBM Research Africa; Deloitte
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: As machine learning and data science applications grow ever more prevalent, there is an increased focus on data sharing and open data initiatives, particularly in the context of the African continent. Many argue that data sharing can support research and policy design to alleviate poverty, inequality, and derivative effects in Africa. Despite the fact that the datasets in question are often extracted from African communities, conversations around the challenges of accessing and sharing African data are too often driven by non-African stakeholders. These perspectives frequently employ a deficit narratives, often focusing on lack of education, training, and technological resources in the continent as the leading causes of friction in the data ecosystem.
Title: Group Fairness: Independence Revisited
Authors: Tim Räz
Affiliations: University of Bern
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: This paper critically examines arguments against independence, a measure of group fairness also known as statistical parity and as demographic parity. In recent discussions of fairness in computer science, some have maintained that independence is not a suitable measure of group fairness. This position is at least partially based on two influential papers (Dwork et al., 2012, Hardt et al., 2016) that provide arguments against independence. We revisit these arguments, and we find that the case against independence is rather weak. We also give arguments in favor of independence, showing that it plays a distinctive role in considerations of fairness. Finally, we discuss how to balance different fairness considerations.
Title: Taxonomy of Risks posed by Language Models
Authors: Laura Weidinger; Jonathan Uesato; Maribeth Rauh; Conor Griffin; Po-Sen Huang; John Mellor; Amelia Glaese; Myra Cheng; Borja Balle; Atoosa Kasirzadeh; Courtney Biles; Sasha Brown; Zac Kenton; Will Hawkins; Tom Stepleton; Abeba Birhane; Lisa Anne Hendricks; Laura Rimell; William Isaac; Julia Haas; Sean Legassick; Geoffrey Irving; Iason Gabriel
Affiliations: Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; California Institute of Technology; Google DeepMind; University of Toronto; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; University College Dublin; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind; Google DeepMind
Year: 2022
Domain: LLMs & NLP

Abstract: Responsible innovation on large-scale Language Models (LMs) requires foresight into and in-depth understanding of the risks these models may pose. This paper develops a comprehensive taxonomy of ethical and social risks associated with LMs. We identify twenty-one risks, drawing on expertise and literature from computer science, linguistics, and the social sciences. We situate these risks in our taxonomy of six risk areas: I. Discrimination, Hate speech and Exclusion, II. Information Hazards, III. Misinformation Harms, IV. Malicious Uses, V. Human-Computer Interaction Harms, and VI. Environmental and Socioeconomic harms. For risks that have already been observed in LMs, the causal mechanism leading to harm, evidence of the risk, and approaches to risk mitigation are discussed. We further describe and analyse risks that have not yet been observed but are anticipated based on assessments of other language technologies, and situate these in the same taxonomy. We underscore that it is the responsibility of organizations to engage with the mitigations we discuss throughout the paper. We close by highlighting challenges and directions for further research on risk evaluation and mitigation with the goal of ensuring that language models are developed responsibly.
Title: Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Authors: Wesley Hanwen Deng; Manish Nagireddy; Michelle Seng Ah Lee; Jatinder Singh; Zhiwei Steven Wu; Kenneth Holstein; Haiyi Zhu
Affiliations: Carnegie Mellon University; Carnegie Mellon University; University of Cambridge; University of Cambridge; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Recent years have seen the development of many open-source ML fairness toolkits aimed at helping ML practitioners assess and address unfairness in their systems. However, there has been little research investigating how ML practitioners actually use these toolkits in practice. In this paper, we conducted the first in-depth empirical exploration of how industry practitioners (try to) work with existing fairness toolkits. In particular, we conducted think-aloud interviews to understand how participants learn about and use fairness toolkits, and explored the generality of our findings through an anonymous online survey. We identified several opportunities for fairness toolkits to better address practitioner needs and scaffold them in using toolkits effectively and responsibly. Based on these findings, we highlight implications for the design of future open-source fairness toolkits that can support practitioners in better contextualizing, communicating, and collaborating around ML fairness efforts.
Title: Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation
Authors: Angelina Wang; V. Ramaswamy; Olga Russakovsky
Affiliations: Princeton University; Princeton University; Princeton University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Research in machine learning fairness has historically considered a single binary demographic attribute; however, the reality is of course far more complicated. In this work, we grapple with questions that arise along three stages of the machine learning pipeline when incorporating intersectionality as multiple demographic attributes: (1) which demographic attributes to include as dataset labels, (2) how to handle the progressively smaller size of subgroups during model training, and (3) how to move beyond existing evaluation metrics when benchmarking model fairness for more subgroups. For each question, we provide thorough empirical evaluation on tabular datasets derived from the US Census, and present constructive recommendations for the machine learning community. First, we advocate for supplementing domain knowledge with empirical validation when choosing which demographic attribute labels to train on, while always evaluating on the full set of demographic attributes. Second, we warn against using data imbalance techniques without considering their normative implications and suggest an alternative using the structure in the data. Third, we introduce new evaluation metrics which are more appropriate for the intersectional setting. Overall, we provide substantive suggestions on three necessary (albeit not sufficient!) considerations when incorporating intersectionality into machine learning.
Title: Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis
Authors: Maarten Buyl; Christina Cociancig; Cristina Frattone; Nele Roekens
Affiliations: Ghent University Hospital; University of Bremen; Roma Tre University; 
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Tackling algorithmic discrimination against persons with disabilities (PWDs) demands a distinctive approach that is fundamentally different to that applied to other protected characteristics, due to particular ethical, legal, and technical challenges. We address these challenges specifically in the context of artificial intelligence (AI) systems used in hiring processes (or automated hiring systems, AHSs), in which automated assessment procedures are subject to unique ethical and legal considerations and have an undeniable adverse impact on PWDs. In this paper, we discuss concerns and opportunities raised by AI-driven hiring in relation to disability discrimination. Ultimately, we aim to encourage further research into this topic. Hence, we establish some starting points and design a roadmap for ethicists, lawmakers, advocates as well as AI practitioners alike.
Title: The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
Authors: Aparna Balagopalan; Haoran Zhang; Kimia Hamidieh; Thomas Hartvigsen; Frank Rudzicz; Marzyeh Ghassemi
Affiliations: Massachusetts Institute of Technology; Massachusetts Institute of Technology; University of Toronto; Massachusetts Institute of Technology; University of Toronto; Vector Institute
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning models in safety-critical settings like healthcare are often “blackboxes”: they contain a large number of parameters which are not transparent to users. Post-hoc explainability methods where a simple, human-interpretable model imitates the behavior of these blackbox models are often proposed to help users trust model predictions. In this work, we audit the quality of such explanations for different protected subgroups using real data from four settings in finance, healthcare, college admissions, and the US justice system. Across two different blackbox model architectures and four popular explainability methods, we find that the approximation quality of explanation models, also known as the fidelity, differs significantly between subgroups. We also demonstrate that pairing explainability methods with recent advances in robust machine learning can improve explanation fairness in some settings. However, we highlight the importance of communicating details of non-zero fidelity gaps to users, since a single solution might not exist across all settings. Finally, we discuss the implications of unfair explanation models as a challenging and understudied problem facing the machine learning community.
Title: Are "Intersectionally Fair" AI Algorithms Really Fair to Women of Color? A Philosophical Analysis
Authors: Youjin Kong
Affiliations: Oregon State University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: A growing number of studies on fairness in artificial intelligence (AI) use the notion of intersectionality to measure AI fairness. Most of these studies take intersectional fairness to be a matter of statistical parity among intersectional subgroups: an AI algorithm is “intersectionally fair” if the probability of the outcome is roughly the same across all subgroups defined by different combinations of the protected attributes. This paper identifies and examines three fundamental problems with this dominant interpretation of intersectional fairness in AI. First, the dominant approach is so preoccupied with the intersection of attributes/categories (e.g., race, gender) that it fails to address the intersection of oppression (e.g., racism, sexism), which is more central to intersectionality as a critical framework. Second, the dominant approach faces a dilemma between infinite regress and fairness gerrymandering: it either keeps splitting groups into smaller subgroups or arbitrarily selects protected groups. Lastly, the dominant view fails to capture what it really means for AI algorithms to be fair, in terms of both distributive and non-distributive fairness. I distinguish a strong sense of AI fairness from a weak sense that is prevalent in the literature, and conclude by envisioning paths towards strong intersectional fairness in AI.
Title: The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models
Authors: Greg d'Eon; Jason d'Eon; James R. Wright; Kevin Leyton-Brown
Affiliations: University of British Columbia; Dalhousie University; University of Alberta; University of British Columbia
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Supervised learning models often make systematic errors on rare subsets of the data. When these subsets correspond to explicit labels in the data (e.g., gender, race) such poor performance can be identified straightforwardly. This paper introduces a method for discovering systematic errors that do not correspond to such explicitly labelled subgroups. The key idea is that similar inputs tend to have similar representations in the final hidden layer of a neural network. We leverage this structure by “shining a spotlight” on this representation space to find contiguous regions in which the model performs poorly. We show that the Spotlight surfaces semantically meaningful areas of weakness in a wide variety of existing models spanning computer vision, NLP, and recommender systems, and we verify its performance through quantitative experiments.
Title: Language Variation and Algorithmic Bias: Understanding Algorithmic Bias in British English Automatic Speech Recognition
Authors: Nina Markl
Affiliations: University of Edinburgh
Year: 2022
Domain: LLMs & NLP

Abstract: All language is characterised by variation which language users employ to construct complex social identities and express social meaning. Like other machine learning technologies, speech and language technologies (re)produce structural oppression when they perform worse for marginalised language communities. Using knowledge and theories from sociolinguistics, I explore why commercial automatic speech recognition systems and other language technologies perform significantly worse for already marginalised populations, such as second-language speakers and speakers of stigmatised varieties of English in the British Isles. Situating language technologies within the broader scholarship around algorithmic bias, consider the allocative and representational harms they can cause even (and perhaps especially) in systems which do not exhibit predictive bias, narrowly defined as differential performance between groups. This raises the question whether addressing or “fixing” this “bias” is actually always equivalent to mitigating the harms algorithmic systems can cause, in particular to marginalised communities.
Title: Robots Enact Malignant Stereotypes
Authors: Andrew Hundt; William S. Agnew; Vicky Zeng; Severin Kacianka; Matthew Gombolay
Affiliations: Georgia Institute of Technology; University of Washington; Johns Hopkins University; Technical University of Munich; Georgia Institute of Technology
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Stereotypes, bias, and discrimination have been extensively documented in Machine Learning (ML) methods such as Computer Vision (CV) [18, 80], Natural Language Processing (NLP) [6], or both, in the case of large image and caption models such as OpenAI CLIP [14]. In this paper, we evaluate how ML bias manifests in robots that physically and autonomously act within the world. We audit one of several recently published CLIP-powered robotic manipulation methods, presenting it with objects that have pictures of human faces on the surface which vary across race and gender, alongside task descriptions that contain terms associated with common stereotypes. Our experiments definitively show robots acting out toxic stereotypes with respect to gender, race, and scientifically-discredited physiognomy, at scale. Furthermore, the audited methods are less likely to recognize Women and People of Color. Our interdisciplinary sociotechnical analysis synthesizes across fields and applications such as Science Technology and Society (STS), Critical Studies, History, Safety, Robotics, and AI. We find that robots powered by large datasets and Dissolution Models (sometimes called “foundation models”, e.g. CLIP) that contain humans risk physically amplifying malignant stereotypes in general; and that merely correcting disparities will be insufficient for the complexity and scale of the problem. Instead, we recommend that robot learning methods that physically manifest stereotypes or other harmful outcomes be paused, reworked, or even wound down when appropriate, until outcomes can be proven safe, effective, and just. Finally, we discuss comprehensive policy changes and the potential of new interdisciplinary research on topics like Identity Safety Assessment Frameworks and Design Justice to better understand and address these harms.
Title: Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness
Authors: McKane Andrus; Sarah Villeneuve
Affiliations: Partnership on AI; Partnership on AI
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Most proposed algorithmic fairness techniques require access to demographic data in order to make performance comparisons and standardizations across groups, however this data is largely unavailable in practice, hindering the widespread adoption of algorithmic fairness. Through this paper, we consider calls to collect more data on demographics to enable algorithmic fairness and challenge the notion that discrimination can be overcome with smart enough technical methods and sufficient data. We show how these techniques largely ignore broader questions of data governance and systemic oppression when categorizing individuals for the purpose of fairer algorithmic processing. In this work, we explore under what conditions demographic data should be collected and used to enable algorithmic fairness methods by characterizing a range of social risks to individuals and communities. For the risks to individuals we consider the unique privacy risks of sensitive attributes, the possible harms of miscategorization and misrepresentation, and the use of sensitive data beyond data subjects’ expectations. Looking more broadly, the risks to entire groups and communities include the expansion of surveillance infrastructure in the name of fairness, misrepresenting and mischaracterizing what it means to be part of a demographic group, and ceding the ability to define what constitutes biased or unfair treatment. We argue that, by confronting these questions before and during the collection of demographic data, algorithmic fairness methods are more likely to actually mitigate harmful treatment disparities without reinforcing systems of oppression. Towards this end, we assess privacy-focused methods of data collection and use and participatory data governance structures as proposals for more responsibly collecting demographic data.
Title: Fair Ranking: A Critical Review, Challenges, and Future Directions
Authors: Gourab K Patro; Lorenzo Porcaro; Laura E. Mitchell; Qiuyue Zhang; Meike Zehlike; Nikhil Garg
Affiliations: Indian Institute of Technology Indore; Universitat Pompeu Fabra; ; Accenture; Max Planck Institute for Software Systems; Cornell University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Ranking, recommendation, and retrieval systems are widely used in online platforms and other societal systems, including e-commerce, media-streaming, admissions, gig platforms, and hiring. In the recent past, a large “fair ranking” research literature has been developed around making these systems fair to the individuals, providers, or content that are being ranked. Most of this literature defines fairness for a single instance of retrieval, or as a simple additive notion for multiple instances of retrievals over time. This work provides a critical overview of this literature, detailing the often context-specific concerns that such approaches miss: the gap between high ranking placements and true provider utility, spillovers and compounding effects over time, induced strategic incentives, and the effect of statistical uncertainty. We then provide a path forward for a more holistic and impact-oriented fair ranking research agenda, including methodological lessons from other fields and the role of the broader stakeholder community in overcoming data bottlenecks and designing effective regulatory environments.
Title: Theories of "Gender" in NLP Bias Research
Authors: Hannah Devinney; Jenny Björklund; Henrik Björklund
Affiliations: Umeå University; Uppsala University; Umeå University
Year: 2022
Domain: LLMs & NLP

Abstract: The rise of concern around Natural Language Processing (NLP) technologies containing and perpetuating social biases has led to a rich and rapidly growing area of research. Gender bias is one of the central biases being analyzed, but to date there is no comprehensive analysis of how “gender” is theorized in the field. We survey nearly 200 articles concerning gender bias in NLP to discover how the field conceptualizes gender both explicitly (e.g. through definitions of terms) and implicitly (e.g. through how gender is operationalized in practice). In order to get a better idea of emerging trajectories of thought, we split these articles into two sections by time.
Title: Measuring Representational Harms in Image Captioning
Authors: Angelina Wang; Solon Barocas; Kristen Laird; Hanna Wallach
Affiliations: Princeton University; Microsoft; Microsoft; Microsoft
Year: 2022
Domain: LLMs & NLP

Abstract: Previous work has largely considered the fairness of image captioning systems through the underspecified lens of “bias.” In contrast, we present a set of techniques for measuring five types of representational harms, as well as the resulting measurements obtained for two of the most popular image captioning datasets using a state-of-the-art image captioning system. Our goal was not to audit this image captioning system, but rather to develop normatively grounded measurement techniques, in turn providing an opportunity to reflect on the many challenges involved. We propose multiple measurement techniques for each type of harm. We argue that by doing so, we are better able to capture the multi-faceted nature of each type of harm, in turn improving the (collective) validity of the resulting measurements. Throughout, we discuss the assumptions underlying our measurement approach and point out when they do not hold.