cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Interventions for Ranking in the Presence of Implicit Bias
Authors: L. Elisa Celis; Anay Mehrotra; Nisheeth K. Vishnoi
Affiliations: Yale University; Indian Institute of Technology Kanpur; Yale University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Implicit bias is the unconscious attribution of particular qualities (or lack thereof) to a member from a particular social group (e.g., defined by gender or race). Studies on implicit bias have shown that these unconscious stereotypes can have adverse outcomes in various social contexts, such as job screening, teaching, or policing. Recently, [34] considered a mathematical model for implicit bias and showed the effectiveness of the Rooney Rule as a constraint to improve the utility of the outcome for certain cases of the subset selection problem. Here we study the problem of designing interventions for the generalization of subset selection - ranking - that requires to output an ordered set and is a central primitive in various social and computational contexts. We present a family of simple and interpretable constraints and show that they can optimally mitigate implicit bias for a generalization of the model studied in [34]. Subsequently, we prove that under natural distributional assumptions on the utilities of items, simple, Rooney Rule-like, constraints can also surprisingly recover almost all the utility lost due to implicit biases. Finally, we augment our theoretical results with empirical findings on real-world distributions from the IIT-JEE (2009) dataset and the Semantic Scholar Research corpus.
Title: Counterfactual Risk Assessments, Evaluation, and Fairness
Authors: Amanda Coston; Alan Mishler; Edward H. Kennedy; Alexandra Chouldechova
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic risk assessments are increasingly used to help humans make decisions in high-stakes settings, such as medicine, criminal justice and education. In each of these cases, the purpose of the risk assessment tool is to inform actions, such as medical treatments or release conditions, often with the aim of reducing the likelihood of an adverse event such as hospital readmission or recidivism. Problematically, most tools are trained and evaluated on historical data in which the outcomes observed depend on the historical decision-making policy. These tools thus reflect risk under the historical policy, rather than under the different decision options that the tool is intended to inform. Even when tools are constructed to predict risk under a specific decision, they are often improperly evaluated as predictors of the target outcome.
Title: Case Study: Predictive Fairness to Reduce Misdemeanor Recidivism Through Social Service Interventions
Authors: Kit T. Rodolfa; Erika Salomon; Lauren Haynes; Iván Higuera Mendieta; Jamie Larson; Rayid Ghani
Affiliations: Carnegie Mellon University; University of Chicago; University of Chicago; University of Chicago; Office of the Attorney General; Carnegie Mellon University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: The criminal justice system is currently ill-equipped to improve outcomes of individuals who cycle in and out of the system with a series of misdemeanor offenses. Often due to constraints of caseload and poor record linkage, prior interactions with an individual may not be considered when an individual comes back into the system, let alone in a proactive manner through the application of diversion programs. The Los Angeles City Attorney's Office recently created a new Recidivism Reduction and Drug Diversion unit (R2D2) tasked with reducing recidivism in this population. Here we describe a collaboration with this new unit as a case study for the incorporation of predictive equity into machine learning based decision making in a resource-constrained setting. The program seeks to improve outcomes by developing individually-tailored social service interventions (i.e., diversions, conditional plea agreements, stayed sentencing, or other favorable case disposition based on appropriate social service linkage rather than traditional sentencing methods) for individuals likely to experience subsequent interactions with the criminal justice system, a time and resource-intensive undertaking that necessitates an ability to focus resources on individuals most likely to be involved in a future case. Seeking to achieve both efficiency (through predictive accuracy) and equity (improving outcomes in traditionally under-served communities and working to mitigate existing disparities in criminal justice outcomes), we discuss the equity outcomes we seek to achieve, describe the corresponding choice of a metric for measuring predictive fairness in this context, and explore a set of options for balancing equity and efficiency when building and selecting machine learning models in an operational public policy setting.
Title: The False Promise of Risk Assessments: Epistemic Reform and the Limits of Fairness
Authors: Ben Green
Affiliations: Harvard University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Risk assessments have proliferated in the United States criminal justice system. The theory of change motivating their adoption involves two key assumptions: first, that risk assessments will reduce human biases by making objective decisions, and second, that risk assessments will promote criminal justice reform. In this paper I interrogate both of these assumptions, concluding that risk assessments are an ill-advised tool for challenging the centrality and legitimacy of incarceration within the criminal justice system. First, risk assessments fail to provide objectivity, as their use creates numerous sites of discretion. Second, risk assessments provide no guarantee of reducing incarceration; instead, they risk legitimizing the criminal justice system's structural racism. I then consider, via an "epistemic reform," the path forward for criminal justice reform. I reinterpret recent results regarding the "impossibility of fairness" as not simply a tension between mathematical metrics but as evidence of a deeper tension between notions of equality. This expanded frame challenges the formalist, colorblind proceduralism at the heart of the criminal justice system and suggests a more structural approach to reform. Together, this analysis highlights how algorithmic fairness narrows the scope of judgments about justice and how "fair" algorithms can reinforce discrimination.
Title: On the Apparent Conflict Between Individual and Group Fairness
Authors: Reuben Binns
Affiliations: University of Oxford
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: A distinction has been drawn in fair machine learning research between 'group' and 'individual' fairness measures. Many technical research papers assume that both are important, but conflicting, and propose ways to minimise the trade-offs between these measures. This paper argues that this apparent conflict is based on a misconception. It draws on discussions from within the fair machine learning research, and from political and legal philosophy, to argue that individual and group fairness are not fundamentally in conflict. First, it outlines accounts of egalitarian fairness which encompass plausible motivations for both group and individual fairness, thereby suggesting that there need be no conflict in principle. Second, it considers the concept of individual justice, from legal philosophy and jurisprudence, which seems similar but actually contradicts the notion of individual fairness as proposed in the fair machine learning literature. The conclusion is that the apparent conflict between individual and group fairness is more of an artefact of the blunt application of fairness measures, rather than a matter of conflicting principles. In practice, this conflict may be resolved by a nuanced consideration of the sources of 'unfairness' in a particular deployment context, and the carefully justified application of measures to mitigate it.
Title: Fairness Is Not Static: Deeper Understanding of Long Term Fairness via Agents and Environments
Authors: Alexander D'Amour; Hansa Srinivasan; James Atwood; Pallavi Baljekar; D. Sculley; Yoni Halpern
Affiliations: Google; Google; Google; Google; Google; Google
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: As machine learning becomes increasingly incorporated within high impact decision ecosystems, there is a growing need to understand the long-term behaviors of deployed ML-based decision systems and their potential consequences. Most approaches to understanding or improving the fairness of these systems have focused on static settings without considering long-term dynamics. This is understandable; long term dynamics are hard to assess, particularly because they do not align with the traditional supervised ML research framework that uses fixed data sets. To address this structural difficulty in the field, we advocate for the use of simulation as a key tool in studying the fairness of algorithms. We explore three toy examples of dynamical systems that have been previously studied in the context of fair decision making for bank loans, college admissions, and allocation of attention. By analyzing how learning agents interact with these systems in simulation, we are able to extend previous work, showing that static or single-step analyses do not give a complete picture of the long-term consequences of an ML-based decision system. We provide an extensible open-source software framework for implementing fairness-focused simulation studies and further reproducible research, available at https://github.com/google/ml-fairness-gym.
Title: Data in New Delhi's Predictive Policing System
Authors: Vidushi Marda; Shivangi Narayan
Affiliations: ; Jawaharlal Nehru University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: In 2015, Delhi Police announced plans for predictive policing. The Crime Mapping, Analytics and Predictive System (CMAPS) would be implemented in India's capital, for live spatial hotspot mapping of crime, criminal behavior patterns and suspect analysis. Four years later, there is little known about the effect of CMAPS due to the lack of public accountability mechanisms and large exceptions for law enforcement under India's Right to Information Act. Through an ethnographic study of Delhi Police's data collection practices, and analysing the institutional and legal reality within which CMAPS will function, this paper presents one of the first accounts of smart policing in India. Through our findings and discussion we show what kinds of biases are present within Delhi Police's data collection practices currently and how they translate and transfer into initiatives like CMAPS. We further discuss what the biases in CMAPS can teach us about future public sector deployment of socio-technical systems in India and other global South geographies. We also offer methodological considerations for studying AI deployments in non-western contexts. We conclude with a set of recommendations for civil society and social justice actors to consider when engaging with opaque systems implemented in the public sector.
Title: Auditing Radicalization Pathways on YouTube
Authors: Manoel Horta Ribeiro; Raphael Ottoni; Robert West; Virgílio Almeida; Wagner Meira
Affiliations: École Polytechnique Fédérale de Lausanne; Universidade Federal de Minas Gerais; École Polytechnique Fédérale de Lausanne; Universidade Federal de Minas Gerais; Universidade Federal de Minas Gerais
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Non-profits, as well as the media, have hypothesized the existence of a radicalization pipeline on YouTube, claiming that users systematically progress towards more extreme content on the platform. Yet, there is to date no substantial quantitative evidence of this alleged pipeline. To close this gap, we conduct a large-scale audit of user radicalization on YouTube. We analyze 330,925 videos posted on 349 channels, which we broadly classified into four types: Media, the Alt-lite, the Intellectual Dark Web (I.D.W.), and the Alt-right. According to the aforementioned radicalization hypothesis, channels in the I.D.W. and the Alt-lite serve as gateways to fringe far-right ideology, here represented by Alt-right channels. Processing 72M+ comments, we show that the three channel types indeed increasingly share the same user base; that users consistently migrate from milder to more extreme content; and that a large percentage of users who consume Alt-right content now consumed Alt-lite and I.D.W. content in the past. We also probe YouTube's recommendation algorithm, looking at more than 2M video and channel recommendations between May/July 2019. We find that Alt-lite content is easily reachable from I.D.W. channels, while Alt-right videos are reachable only through channel recommendations. Overall, we paint a comprehensive picture of user radicalization on YouTube.
Title: Differential Tweetment
Authors: Ari Ball-Burack; Michelle Seng Ah Lee; Jennifer Cobbe; Jatinder Singh
Affiliations: University of Cambridge; University of Cambridge; University of Cambridge; University of Cambridge
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Automated systems for detecting harmful social media content are afflicted by a variety of biases, some of which originate in their training datasets. In particular, some systems have been shown to propagate racial dialect bias: they systematically classify content aligned with the African American English (AAE) dialect as harmful at a higher rate than content aligned with White English (WE). This perpetuates prejudice by silencing the Black community. Towards this problem we adapt and apply two existing bias mitigation approaches: preferential sampling pre-processing and adversarial debiasing in-processing. We analyse the impact of our interventions on model performance and propagated bias. We find that when bias mitigation is employed, a high degree of predictive accuracy is maintained relative to baseline, and in many cases bias against AAE in harmful tweet predictions is reduced. However, the specific effects of these interventions on bias and performance vary widely between dataset contexts. This variation suggests the unpredictability of autonomous harmful content detection outside of its development context. We argue that this, and the low performance of these systems at baseline, raise questions about the reliability and role of such systems in high-impact, real-world settings.
Title: Biases in Generative Art: A Causal Look from the Lens of Art History
Authors: Ramya Srinivasan; Kanji Uchino
Affiliations: Fujitsu Research of America; Fujitsu Research of America
Year: 2021
Domain: LLMs & NLP

Abstract: With rapid progress in artificial intelligence (AI), popularity of generative art has grown substantially. From creating paintings to generating novel art styles, AI based generative art has showcased a variety of applications. However, there has been little focus concerning the ethical impacts of AI based generative art. In this work, we investigate biases in the generative art AI pipeline right from those that can originate due to improper problem formulation to those related to algorithm design. Viewing from the lens of art history, we discuss the socio-cultural impacts of these biases. Leveraging causal models, we highlight how current methods fall short in modeling the process of art creation and thus contribute to various types of biases. We illustrate the same through case studies, in particular those related to style transfer. To the best of our knowledge, this is the first extensive analysis that investigates biases in the generative art AI pipeline from the perspective of art history. We hope our work sparks interdisciplinary discussions related to accountability of generative art.
Title: Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
Authors: Ryan Steed; Aylin Caliskan
Affiliations: Carnegie Mellon University; George Washington University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Recent advances in machine learning leverage massive datasets of unlabeled images from the web to learn general-purpose image representations for tasks from image classification to face recognition. But do unsupervised computer vision models automatically learn implicit patterns and embed social biases that could have harmful downstream effects? We develop a novel method for quantifying biased associations between representations of social concepts and attributes in images. We find that state-of-the-art unsupervised models trained on ImageNet, a popular benchmark image dataset curated from internet images, automatically learn racial, gender, and intersectional biases. We replicate 8 previously documented human biases from social psychology, from the innocuous, as with insects and flowers, to the potentially harmful, as with race and gender. Our results closely match three hypotheses about intersectional bias from social psychology. For the first time in unsupervised computer vision, we also quantify implicit human biases about weight, disabilities, and several ethnicities. When compared with statistical patterns in online image datasets, our findings suggest that machine learning models can automatically learn bias from the way people are stereotypically portrayed on the web.
Title: One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision
Authors: Zaid Khan; Yun Fu
Affiliations: Northeastern University; Northeastern University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Computer vision is widely deployed, has highly visible, society altering applications, and documented problems with bias and representation. Datasets are critical for benchmarking progress in fair computer vision, and often employ broad racial categories as population groups for measuring group fairness. Similarly, diversity is often measured in computer vision datasets by ascribing and counting categorical race labels. However, racial categories are ill-defined, unstable temporally and geographically, and have a problematic history of scientific use. Although the racial categories used across datasets are superficially similar, the complexity of human race perception suggests the racial system encoded by one dataset may be substantially inconsistent with another. Using the insight that a classifier can learn the racial system encoded by a dataset, we conduct an empirical study of computer vision datasets supplying categorical race labels for face images to determine the cross-dataset consistency and generalization of racial categories. We find that each dataset encodes a substantially unique racial system, despite nominally equivalent racial categories, and some racial categories are systemically less consistent than others across datasets. We find evidence that racial categories encode stereotypes, and exclude ethnic groups from categories on the basis of nonconformity to stereotypes. Representing a billion humans under one racial category may obscure disparities and create new ones by encoding stereotypes of racial systems. The difficulty of adequately converting the abstract concept of race into a tool for measuring fairness underscores the need for a method more flexible and culturally aware than racial categories.
Title: BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Authors: Jwala Dhamala; Tony Sun; Varun Kumar; Satyapriya Krishna; Yada Pruksachatkun; Kai-Wei Chang; Rahul Gupta
Affiliations: Amazon; University of California Santa Barbara; Amazon; Amazon; Amazon; Amazon; Amazon
Year: 2021
Domain: LLMs & NLP

Abstract: Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.
Title: Towards Cross-Lingual Generalization of Translation Gender Bias
Authors: Won Ik Cho; Jiwon Kim; Jaeyeong Yang; Nam Soo Kim
Affiliations: Seoul National University; Seoul National University; Seoul National University; Seoul National University
Year: 2021
Domain: LLMs & NLP

Abstract: Cross-lingual generalization issues for less explored languages have been broadly tackled in recent NLP studies. In this study, we apply the philosophy on the problem of translation gender bias, which necessarily involves multilingualism and socio-cultural diversity. Beyond the conventional evaluation criteria for the social bias, we aim to put together various aspects of linguistic viewpoints into the measuring process, to create a template that makes evaluation less tilted to specific types of language pairs. With a manually constructed set of content words and template, we check both the accuracy of gender inference and the fluency of translation, for German, Korean, Portuguese, and Tagalog. Inference accuracy and disparate impact, namely the biasedness factors associated with each other, show that the failure of bias mitigation threatens the delicacy of translation. Furthermore, our analyses on each system and language indicate that the translation fluency and inference accuracy are not necessarily correlated. The results implicitly suggest that the amount of available language resources that boost up the performance might amplify the bias cross-linguistically.