cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions
Authors: Anna P. Meyer; Aws Albarghouthi; Loris D’Antoni
Affiliations: University of Wisconsin–Madison; University of Wisconsin–Madison; University of Wisconsin–Madison
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: We introduce dataset multiplicity, a way to study how inaccuracies, uncertainty, and social bias in training datasets impact test-time predictions. The dataset multiplicity framework asks a counterfactual question of what the set of resultant models (and associated test-time predictions) would be if we could somehow access all hypothetical, unbiased versions of the dataset. We discuss how to use this framework to encapsulate various sources of uncertainty in datasets’ factualness, including systemic social bias, data collection practices, and noisy labels or features. We show how to exactly analyze the impacts of dataset multiplicity for a specific model architecture and type of uncertainty: linear models with label errors. Our empirical analysis shows that real-world datasets, under reasonable assumptions, contain many test samples whose predictions are affected by dataset multiplicity. Furthermore, the choice of domain-specific dataset multiplicity definition determines what samples are affected, and whether different demographic groups are disparately impacted. Finally, we discuss implications of dataset multiplicity for machine learning practice and research, including considerations for when model outcomes should not be trusted.
Title: Runtime Monitoring of Dynamic Fairness Properties
Authors: Thomas A. Henzinger; Mahyar Karimi; Konstantin Kueffner; Kaushik Mallik
Affiliations: Institute of Science and Technology Austria; Institute of Science and Technology Austria; Institute of Science and Technology Austria; Institute of Science and Technology Austria
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: A machine-learned system that is fair in static decision-making tasks may have biased societal impacts in the long-run. This may happen when the system interacts with humans and feedback patterns emerge, reinforcing old biases in the system and creating new biases. While existing works try to identify and mitigate long-run biases through smart system design, we introduce techniques for monitoring fairness in real time. Our goal is to build and deploy a monitor that will continuously observe a long sequence of events generated by the system in the wild, and will output, with each event, a verdict on how fair the system is at the current point in time. The advantages of monitoring are two-fold. Firstly, fairness is evaluated at run-time, which is important because unfair behaviors may not be eliminated a priori, at design-time, due to partial knowledge about the system and the environment, as well as uncertainties and dynamic changes in the system and the environment, such as the unpredictability of human behavior. Secondly, monitors are by design oblivious to how the monitored system is constructed, which makes them suitable to be used as trusted third-party fairness watchdogs. They function as computationally lightweight statistical estimators, and their correctness proofs rely on the rigorous analysis of the stochastic process that models the assumptions about the underlying dynamics of the system. We show, both in theory and experiments, how monitors can warn us (1) if a bank’s credit policy over time has created an unfair distribution of credit scores among the population, and (2) if a resource allocator’s allocation policy over time has made unfair allocations. Our experiments demonstrate that the monitors introduce very low overhead. We believe that runtime monitoring is an important and mathematically rigorous new addition to the fairness toolbox.
Title: Representation Online Matters: Practical End-to-End Diversification in Search and Recommender Systems
Authors: Pedro Silva; Bhawna Juneja; Shloka Desai; Ashudeep Singh; Nadia Fawaz
Affiliations: Pinterest Inc; Pinterest Inc; Pinterest Inc; Pinterest Inc; Pinterest Inc
Year: 2023
Domain: Recommender Systems

Abstract: As the use of online platforms continues to grow across all demographics, users often express a desire to feel represented in the content. To improve representation in search results and recommendations, we introduce end-to-end diversification, ensuring that diverse content flows throughout the various stages of these systems, from retrieval to ranking. We develop, experiment, and deploy scalable diversification mechanisms in multiple production surfaces on the Pinterest platform, including Search, Related Products, and New User Homefeed, to improve the representation of different skin tones in beauty and fashion content. Diversification in production systems includes three components: identifying requests that will trigger diversification, ensuring diverse content is retrieved from the large content corpus during the retrieval stage, and finally, balancing the diversity-utility trade-off in a self-adjusting manner in the ranking stage. Our approaches, which evolved from using Strong-OR logical operator to bucketized retrieval at the retrieval stage and from greedy re-rankers to multi-objective optimization using determinantal point processes for the ranking stage, balances diversity and utility while enabling fast iterations and scalable expansion to diversification over multiple dimensions. Our experiments indicate that these approaches significantly improve diversity metrics, with a neutral to a positive impact on utility metrics and improved user satisfaction, both qualitatively and quantitatively, in production.
Title: The Progression of Disparities within the Criminal Justice System: Differential Enforcement and Risk Assessment Instruments
Authors: Miri Zilka; Riccardo Fogliato; Jiri Hron; Bradley Butcher; Carolyn Ashurst; Adrian Weller
Affiliations: University of Cambridge; Carnegie Mellon University; University of Cambridge; University of Sussex; The Alan Turing Institute; University of Cambridge
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic risk assessment instruments (RAIs) increasingly inform decision-making in criminal justice. RAIs largely rely on arrest records as a proxy for underlying crime. Problematically, the extent to which arrests reflect overall offending can vary with the person’s characteristics. We examine how the disconnect between crime and arrest rates impacts RAIs and their evaluation. Our main contribution is a method for quantifying this bias via estimation of the amount of unobserved offenses associated with particular demographics. These unobserved offenses are then used to augment real-world arrest records to create part real, part synthetic crime records. Using this data, we estimate that four currently deployed RAIs assign 0.5–2.8 percentage points higher risk scores to Black individuals than to White individuals with a similar arrest record, but the gap grows to 4.5–11.0 percentage points when we match on the semi-synthetic crime record. We conclude by discussing the potential risks around the use of RAIs, highlighting how they may exacerbate existing inequalities if the underlying disparities of the criminal justice system are not taken into account. In light of our findings, we provide recommendations to improve the development and evaluation of such tools.
Title: Algorithmic Reproductive Justice
Authors: Jasmine Fledderjohann; Bran Knowles; Esmorie Miller
Affiliations: Lancaster University; Lancaster University; Lancaster University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Reproductive justice is an intersectional feminist framework and movement which argues all people have the right to have a child, to not have a child, to parent in safe and healthy environments, and to own their bodies and control their futures. We identify increasing surveillance, assessing worth, datafication and monetisation, and decimating planetary health as forms of structural violence associated with emerging digital technologies. These trends are implicated in the (re)production of inequities, creating barriers to the realisation of reproductive justice. We call for algorithmic reproductive justice, and highlight the potential for both acts of resistance and industry reform to advance that aim.
Title: Failing Our Youngest: On the Biases, Pitfalls, and Risks in a Decision Support Algorithm Used for Child Protection
Authors: Therese Moreau; Roberta Sinatra; Vedran Sekara
Affiliations: IT University of Copenhagen; Institute for Scientific Interchange; IT University of Copenhagen
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In recent years, Danish child protective services have experienced increasing pressure, prompting the adoption of a decision-support algorithm to aid caseworkers in identifying children at heightened risk of maltreatment, named Decision Support. Despite its critical role, this algorithm has not undergone formal evaluation. Through a freedom of information request, we were able to partially access the algorithm and conduct an audit. We find that the algorithm has significant methodological flaws, suffers from information leakage, relies on inappropriate proxy values for maltreatment assessment, generates inconsistent risk scores, and exhibits age-based discrimination. Given these serious issues, we strongly advise against the use of this kind of algorithms in local government, municipal, and child protection settings, and we call for rigorous evaluation of such tools before implementation and for continual monitoring post-deployment by listing a series of specific recommendations.
Title: A Structured Regression Approach for Evaluating Model Performance Across Intersectional Subgroups
Authors: Christine Herlihy; Kimberly A. Truong; Alexandra Chouldechova; Miroslav Dudı́k
Affiliations: ; Oregon State University; Microsoft (United States); Microsoft (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system’s performance across different subgroups defined by combinations of demographic or other sensitive attributes. The standard approach is to stratify the evaluation data across subgroups and compute performance metrics separately for each group. However, even for moderately-sized evaluation datasets, sample sizes quickly get small once considering intersectional subgroups, which greatly limits the extent to which intersectional groups are included in analysis. In this work, we introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups. We provide corresponding inference strategies for constructing confidence intervals and explore how goodness-of-fit testing can yield insight into the structure of fairness-related harms experienced by intersectional groups. We evaluate our approach on two publicly available datasets, and several variants of semi-synthetic data. The results show that our method is considerably more accurate than the standard approach, especially for small subgroups, and demonstrate how goodness-of-fit testing helps identify the key factors that drive differences in performance.
Title: Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators
Authors: Wiebke Hutiri; Orestis Papakyriakopoulos; Alice Xiang
Affiliations: ; Technical University of Munich; Sony Corporation (United States)
Year: 2024
Domain: LLMs & NLP

Abstract: The rapid and wide-scale adoption of AI to generate human speech poses a range of significant ethical and safety risks to society that need to be addressed. For example, a growing number of speech generation incidents are associated with swatting attacks in the United States, where anonymous perpetrators create synthetic voices that call police officers to close down schools and hospitals, or to violently gain access to innocent citizens’ homes. Incidents like this demonstrate that multimodal generative AI risks and harms do not exist in isolation, but arise from the interactions of multiple stakeholders and technical AI systems. In this paper we analyse speech generation incidents to study how patterns of specific harms arise. We find that specific harms can be categorised according to the exposure of affected individuals, that is to say whether they are a subject of, interact with, suffer due to, or are excluded from speech generation systems. Similarly, specific harms are also a consequence of the motives of the creators and deployers of the systems. Based on these insights we propose a conceptual framework for modelling pathways to ethical and safety harms of AI, which we use to develop a taxonomy of harms of speech generators. Our relational approach captures the complexity of risks and harms in sociotechnical AI systems, and yields a taxonomy that can support appropriate policy interventions and decision making for the responsible development and release of speech generation models.
Title: Operationalizing the Search for Less Discriminatory Alternatives in Fair Lending
Authors: Talia B. Gillis; Vitaly Meursault; Berk Ustun
Affiliations: Columbia University; Federal Reserve Bank of Philadelphia; 
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: The Less Discriminatory Alternative is a key provision of the disparate impact doctrine in the United States. In fair lending, this provision mandates that lenders must adopt models that reduce discrimination when they do not compromise their business interests. In this paper, we develop practical methods to audit for less discriminatory alternatives. Our approach is designed to verify the existence of less discriminatory machine learning models – by returning an alternative model that can reduce discrimination without compromising performance (discovery) or by certifying that an alternative model does not exist (refutation). We develop a method to fit the least discriminatory linear classification model in a specific lending task – by minimizing an exact measure of disparity (e.g., the maximum gap in group FNR) and enforcing hard performance constraints for business necessity (e.g., on FNR and FPR). We apply our method to study the prevalence of less discriminatory alternatives on real-world datasets from consumer finance applications. Our results highlight how models may inadvertently lead to unnecessary discrimination across common deployment regimes, and demonstrate how our approach can support lenders, regulators, and plaintiffs by reliably detecting less discriminatory alternatives in such instances.
Title: Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation
Authors: Jessica Quaye; Alicia Parrish; Oana Inel; Charvi Rastogi; Hannah Rose Kirk; Minsuk Kahng; Erin van Liemt; Max Bartolo; Jess Tsang; Justin P. White; Nathan Clement; Rafael Mosquera; Juan Ciro; Vijay Janapa Reddi; Lora Aroyo
Affiliations: Harvard University Press; Google (United States); University of Zurich; Google (United States); University of Oxford; Google (United States); Google (United States); University College London; Google (United States); Google (United States); Google (United Kingdom); ; ; Harvard University Press; Google (United States)
Year: 2024
Domain: LLMs & NLP

Abstract: With text-to-image (T2I) generative AI models reaching wide audiences, it is critical to evaluate model robustness against non-obvious attacks to mitigate the generation of offensive images. By focusing on “implicitly adversarial” prompts (those that trigger T2I models to generate unsafe images for non-obvious reasons), we isolate a set of difficult safety issues that human creativity is well-suited to uncover. To this end, we built the Adversarial Nibbler Challenge, a red-teaming methodology for crowdsourcing a diverse set of implicitly adversarial prompts. We have assembled a suite of state-of-the-art T2I models, employed a simple user interface to identify and annotate harms, and engaged diverse populations to capture long-tail safety issues that may be overlooked in standard testing. We present an in-depth account of our methodology, a systematic study of novel attack strategies and safety failures, and a visualization tool for easy exploration of the dataset. The first challenge round resulted in over 10k prompt-image pairs with machine annotations for safety. A subset of 1.5k samples contains rich human annotations of harm types and attack styles. Our findings emphasize the necessity of continual auditing and adaptation as new vulnerabilities emerge. This work will enable proactive, iterative safety assessments and promote responsible development of T2I models.
Title: Benchmarking the Fairness of Image Upsampling Methods
Authors: Mike Laszkiewicz; Imant Daunhawer; Julia E. Vogt; Asja Fischer; Johannes Lederer
Affiliations: Ruhr University Bochum; ETH Zurich; ETH Zurich; Ruhr University Bochum; Universität Hamburg
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Recent years have witnessed a rapid development of deep generative models for creating synthetic media, such as images and videos. While the practical applications of these models in everyday tasks are enticing, it is crucial to assess the inherent risks regarding their fairness. In this work, we introduce a comprehensive framework for benchmarking the performance and fairness of conditional generative models. We develop a set of metrics—inspired by their supervised fairness counterparts—to evaluate the models on their fairness and diversity. Focusing on the specific application of image upsampling, we create a benchmark covering a wide variety of modern upsampling methods. As part of the benchmark, we introduce UnfairFace, a subset of FairFace that replicates the racial distribution of common large-scale face datasets. Our empirical study highlights the importance of using an unbiased training set and reveals variations in how the algorithms respond to dataset imbalances. Alarmingly, we find that none of the considered methods produces statistically fair and diverse results. All experiments can be reproduced using our provided repository.1
Title: Analyzing the Relationship Between Difference and Ratio-Based Fairness Metrics
Authors: Min-Hsuan Yeh; Blossom Metevier; Austin Hoag; Philip S. Thomas
Affiliations: ; University of Massachusetts Boston; ; University of Massachusetts Boston
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In research studying the fairness of machine learning algorithms and models, fairness often means that a metric is the same when computed for two different groups of people. For example, one might define fairness to mean that the false positive rate of a classifier is the same for people of different genders, ages, or races. However, it is usually not possible to make this metric identical for all groups. Instead, algorithms ensure that the metric is similar—for example, that the false positive rates are similar. Researchers usually measure this similarity or dissimilarity using either the difference or ratio between the metric values for different groups of people. Although these two approaches are known to be different, there has been little work analyzing their differences and respective benefits. In this paper we examine this relationship analytically and empirically, and conclude that unless there are application-specific reasons to prefer the difference approach, the ratio approach should be preferred.
Title: Tackling Language Modelling Bias in Support of Linguistic Diversity
Authors: Gábor Bella; Paula Helm; Gertraud Koch; Fausto Giunchiglia
Affiliations: Centre National de la Recherche Scientifique; University of Amsterdam; Universität Hamburg; University of Trento
Year: 2024
Domain: LLMs & NLP

Abstract: Current AI-based language technologies—language models, machine translation systems, multilingual dictionaries and corpora—are known to focus on the world’s 2–3% most widely spoken languages. Research efforts of the past decade have attempted to expand this coverage to ‘under-resourced languages.’ The goal of our paper is to bring attention to a corollary phenomenon that we call language modelling bias: multilingual language processing systems often exhibit a hardwired, yet usually involuntary and hidden representational preference towards certain languages. We define language modelling bias as uneven per-language performance under similar test conditions. We show that bias stems not only from technology but also from ethically problematic research and development methodologies that disregard the needs of language communities. Moving towards diversity-aware alternatives, we present an initiative that aims at reducing language modelling bias within lexical resources through both technology design and methodology, based on an eye-level collaboration with local communities.
Title: Diversity of What? On the Different Conceptualizations of Diversity in Recommender Systems
Authors: Sanne Vrijenhoek; Savvina Daniil; Jorden Sandel; Laura Hollink
Affiliations: University of Amsterdam; Centrum Wiskunde & Informatica; University of Amsterdam; Centrum Wiskunde & Informatica
Year: 2024
Domain: Recommender Systems

Abstract: Diversity is a commonly known principle in the design of recommender systems, but also ambiguous in its conceptualization. Through semi-structured interviews we explore how practitioners at three different public service media organizations in the Netherlands conceptualize diversity within the scope of their recommender systems. We provide an overview of the goals that they have with diversity in their systems, which aspects are relevant, and how recommendations should be diversified. We show that even within this limited domain, conceptualization of diversity greatly varies, and argue that it is unlikely that a standardized conceptualization will be achieved. Instead, we should focus on effective communication of what diversity in this particular system means, thus allowing for operationalizations of diversity that are capable of expressing the nuances and requirements of that particular domain.