cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Towards Geographic Inclusion in the Evaluation of Text-to-Image Models
Authors: Melissa Hall; Samuel J. Bell; Candace Ross; Adina Williams; Michal Drozdzal; Adriana Romero
Affiliations: ; ; ; ; ; McGill University
Year: 2024
Domain: LLMs & NLP

Abstract: Rapid progress in text-to-image generative models coupled with their deployment for visual content creation has magnified the importance of thoroughly evaluating their performance and identifying potential biases. In pursuit of models that generate images that are realistic, diverse, visually appealing, and consistent with the given prompt, researchers and practitioners often turn to automated metrics to facilitate scalable and cost-effective performance profiling. However, commonly-used metrics often fail to account for the full diversity of human preference; often even in-depth human evaluations face challenges with subjectivity, especially as interpretations of evaluation criteria vary across regions and cultures. In this work, we conduct a large, cross-cultural study to study how much annotators in Africa, Europe, and Southeast Asia vary in their perception of geographic representation, visual appeal, and consistency in real and generated images from state-of-the art public APIs. We collect over 65,000 image annotations and 20 survey responses. We contrast human annotations with common automated metrics, finding that human preferences vary notably across geographic location and that current metrics do not fully account for this diversity. For example, annotators in different locations often disagree on whether exaggerated, stereotypical depictions of a region are considered geographically representative. In addition, the utility of automatic evaluations is dependent on assumptions about their set-up, such as the alignment of feature extractors with human perception of object similarity or the definition of “appeal” captured in reference datasets used to ground evaluations. We recommend steps for improved automatic and human evaluations. This includes collecting annotations from people located inside and outside the region of interest, instructing annotators on whether they should follow specific definitions of evaluation criteria or utilize their own interpretation, and reporting assumptions underlying automatic evaluations.
Title: Algorithmic Fairness in Performative Policy Learning: Escaping the Impossibility of Group Fairness
Authors: Seamus Somerstep; Ya’acov Ritov; Yuekai Sun
Affiliations: University of Michigan; University of Michigan; University of Michigan
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: In many prediction problems, the predictive model affects the distribution of the prediction target. This phenomenon is known as performativity and is often caused by the behavior of individuals with vested interests in the outcome of the predictive model. Although performativity is generally problematic because it manifests as distribution shifts, we develop algorithmic fairness practices that leverage performativity to achieve stronger group fairness guarantees in social classification problems (compared to what is achievable in non-performative settings). In particular, we leverage the policymaker’s ability to steer the population to remedy inequities in the long term. A crucial benefit of this approach is that it is possible to resolve the incompatibilities between conflicting group fairness definitions.
Title: Lazy Data Practices Harm Fairness Research
Authors: Jan Simson; Alessandro Fabris; Christoph Kern
Affiliations: Munich Center for Machine Learning; Max Planck Institute for Security and Privacy; Munich Center for Machine Learning
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Data practices shape research and practice on fairness in machine learning (fair ML). Critical data studies offer important reflections and critiques for the responsible advancement of the field by highlighting shortcomings and proposing recommendations for improvement. In this work, we present a comprehensive analysis of fair ML datasets, demonstrating how unreflective yet common practices hinder the reach and reliability of algorithmic fairness findings. We systematically study protected information encoded in tabular datasets and their usage in 280 experiments across 142 publications.
Title: Auditing GPT's Content Moderation Guardrails: Can ChatGPT Write Your Favorite TV Show?
Authors: Yaaseen Mahomed; Charlie M. Crawford; Sanjana Gautam; Sorelle A. Friedler; Danaë Metaxa
Affiliations: California University of Pennsylvania; Haverford College; Pennsylvania State University; Haverford College; California University of Pennsylvania
Year: 2024
Domain: LLMs & NLP

Abstract: Large language models (LLMs) are increasingly appearing in consumer-facing products. To prevent problematic use, the organizations behind these systems have put content moderation guardrails in place that prevent the models from generating content they consider harmful. However, most of these enforcement standards and processes are opaque. Although they play a major role in the user experience of these tools, automated content moderation tools have received relatively less attention than other aspects of the models. This study undertakes an algorithm audit of OpenAI’s ChatGPT with the goal of better understanding its content moderation guardrails and their potential biases. To evaluate performance on a broad cultural range of content, we generate a dataset of 100 popular United States television shows with one to three synopses for each episode in the first season of each show (3,309 total synopses). We probe GPT’s content moderation endpoint (ME) to identify violating content both in the synopses themselves, and in GPT’s own outputs when asked to generate a script based on each synopsis, also comparing with ME outputs on 81 real scripts from the same TV shows (269,578 total ME outputs). Our findings show that a large number of GPT-generated and real scripts flag as content violations (about 18% of GPT scripts and 69% of real ones). Using metadata, we find that TV maturity ratings, as well as certain genres (Animation, Crime, Fantasy, and others) are statistically significantly related to a script’s likelihood of flagging. We conclude by discussing the implications of LLM self-censorship and directions for future research on their moderation procedures.
Title: Identifying and Improving Disability Bias in GPT-Based Resume Screening
Authors: Kate Glazko; Yusuf Mohammed; Ben Kosa; Venkatesh Potluri; Jennifer Mankoff
Affiliations: University of Washington; University of Washington; University of Washington; University of Washington; University of Washington
Year: 2024
Domain: LLMs & NLP

Abstract: As Generative AI rises in adoption, its use has expanded to include domains such as hiring and recruiting. However, without examining the potential of bias, this may negatively impact marginalized populations, including people with disabilities. To address this important concern, we present a resume audit study, in which we ask ChatGPT (specifically, GPT-4) to rank a resume against the same resume enhanced with an additional leadership award, scholarship, panel presentation, and membership that are disability-related. We find that GPT-4 exhibits prejudice towards these enhanced CVs. Further, we show that this prejudice can be quantifiably reduced by training a custom GPTs on principles of DEI and disability justice. Our study also includes a unique qualitative analysis of the types of direct and indirect ableism GPT-4 uses to justify its biased decisions and suggest directions for additional bias mitigation work. Additionally, since these justifications are presumably drawn from training data containing real-world biased statements made by humans, our analysis suggests additional avenues for understanding and addressing human bias.
Title: Gender Bias Detection in Court Decisions: A Brazilian Case Study
Authors: Raysa M. Benatti; Fabiana Cristina Severi; Sandra Avila; Esther Luna Colombini
Affiliations: Universidade Estadual de Campinas (UNICAMP); Universidade de São Paulo; Universidade Estadual de Campinas (UNICAMP); Universidade Estadual de Campinas (UNICAMP)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Data derived from the realm of the social sciences is often produced in digital text form, which motivates its use as a source for natural language processing methods. Researchers and practitioners have developed and relied on artificial intelligence techniques to collect, process, and analyze documents in the legal field, especially for tasks such as text summarization and classification. While increasing procedural efficiency is often the primary motivation behind natural language processing in the field, several works have proposed solutions for human rights-related issues, such as assessment of public policy and institutional social settings. One such issue is the presence of gender biases in court decisions, which has been largely studied in social sciences fields; biased institutional responses to gender-based violence are a violation of international human rights dispositions since they prevent gender minorities from accessing rights and hamper their dignity. Natural language processing-based approaches can help detect these biases on a larger scale. Still, the development and use of such tools require researchers and practitioners to be mindful of legal and ethical aspects concerning data sharing and use, reproducibility, domain expertise, and value-charged choices. In this work, we (a) present an experimental framework developed to automatically detect gender biases in court decisions issued in Brazilian Portuguese and (b) describe and elaborate on features we identify to be critical in such a technology, given its proposed use as a support tool for research and assessment of court activity.
Title: The Four-fifths Rule is not Disparate Impact: A Woeful Tale of Epistemic Trespassing in Algorithmic Fairness
Authors: Elizabeth Anne Watkins; Jiahao Chen
Affiliations: Intel (United States); 
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Computer scientists are trained in the art of creating abstractions that simplify and generalize. However, a premature abstraction that omits crucial contextual details creates the risk of epistemic trespassing, by falsely asserting its relevance into other contexts. We study how the field of responsible AI has created an imperfect synecdoche by abstracting the four-fifths rule (a.k.a. the <Formula format="inline"><TexMath><?TeX $\nicefrac {4}{5}$?></TexMath><AltText>Math 1</AltText><File name="facct24-53-inline1" type="svg"/></Formula> rule or 80% rule), a single part of disparate impact discrimination law, into the disparate impact metric. This metric incorrectly introduces a new deontic nuance and new potentials for ethical harms that were absent in the original <Formula format="inline"><TexMath><?TeX $\nicefrac {4}{5}$?></TexMath><AltText>Math 2</AltText><File name="facct24-53-inline2" type="svg"/></Formula> rule. We also survey how the field has amplified the potential for harm in codifying the <Formula format="inline"><TexMath><?TeX $\nicefrac {4}{5}$?></TexMath><AltText>Math 3</AltText><File name="facct24-53-inline3" type="svg"/></Formula> rule into popular AI fairness software toolkits. The harmful erasure of legal nuances is a wake-up call for computer scientists to self-critically re-evaluate the abstractions they create and use, particularly in the interdisciplinary field of AI ethics.
Title: Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images
Authors: Hansa Srinivasan; Candice Schumann; Aradhana Sinha; David Madras; Gbolahan O. Olanubi; Alex Beutel; Susanna Ricco; Jilin Chen
Affiliations: Google (United States); Google (United States); Google (United States); Google (United States); Google (United States); OpenAI (United States); Google (United States); Google (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Capturing the diversity of people in images is challenging: recent literature tends to focus on diversifying one or two attributes, requiring expensive attribute labels or building classifiers. We introduce a diverse people image ranking method which more flexibly aligns with human notions of people diversity in a less prescriptive, label-free manner. The Perception-Aligned Text-derived Human representation Space (PATHS) aims to capture all or many relevant features of people-related diversity, and, when used as the representation space in the standard Maximal Marginal Relevance (MMR) ranking algorithm [7], is better able to surface a range of types of people-related diversity (e.g. disability, cultural attire). PATHS is created in two stages. First, a text-guided approach is used to extract a person-diversity representation from a pre-trained image-text model. Then this representation is fine-tuned on perception judgments from human annotators so that it captures the aspects of people-related similarity that humans find most salient. Empirical results show that the PATHS method achieves diversity better than baseline methods, according to side-by-side ratings from human annotators.
Title: Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation
Authors: Jennifer Chien; David Danks
Affiliations: University of California San Diego; University of California San Diego
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic harms are commonly categorized as either allocative or representational. This study specifically addresses the latter, examining current definitions of representational harms to discern what is included and what is not. This analysis motivates our expansion beyond behavioral definitions to encompass harms to cognitive and affective states. The paper outlines high-level requirements for measurement: identifying the necessary expertise to implement this approach and illustrating it through a case study. Our work highlights the unique vulnerabilities of large language models to perpetrating representational harms, particularly when these harms go unmeasured and unmitigated. The work concludes by presenting proposed mitigations and delineating when to employ them. The overarching aim of this research is to establish a framework for broadening the definition of representational harms and to translate insights from fairness research into practical measurement and mitigation praxis.
Title: Gender Representation Across Online Retail Products
Authors: Dana Pessach; Bárbara Poblete
Affiliations: ; Amazon (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: We present a broad characterization of gender representation in a large heterogeneous sample of retail products. In particular, we study online product textual information, such as titles and descriptions. Our goal is to understand from a semantic perspective, differences and similarities in how girls (women) and boys (men) are represented. We perform a comparative analysis of the language used in gendered products (i.e., products that mention exclusively either of these two genders), and additionally compare it to products that are explicitly gender neutral or inclusive. We found that the adjectives, skills, occupations, and values described in gendered products tended to reinforce stereotypes. Some of these stereotypes are aligned with historical findings from research on traditional off-line retail stores, and others are new owing to the up-to-date product dataset our research is based on. By leveraging additional existing resources we were able to gain insight into how certain product descriptions reflect stereotypes that are related to soft-skills and hierarchical occupational information. Conversely, we found that a large segment of products present explicitly as gender neutral or inclusive. We explore whether the language used by gender-inclusive products can be useful to improve stereotypes reflected in gendered product text. Specifically, we study its effect in word embedding fairness through debiasing techniques.
Title: Mitigating Group Bias in Federated Learning for Heterogeneous Devices
Authors: Khotso Selialia; Yasra Chandio; Fatima M. Anwar
Affiliations: University of Massachusetts Amherst; University of Massachusetts Amherst; University of Massachusetts Amherst
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Federated learning is emerging as a privacy-preserving model training approach in distributed edge applications. As such, most edge deployments are heterogeneous in nature, i.e., their sensing capabilities and environments vary across deployments. This edge heterogeneity violates the independence and identical distribution (IID) property of local data across clients. It produces biased global models, i.e., models that contribute to unfair decision-making and discrimination against a particular community or a group. Existing bias mitigation techniques only focus on bias generated from label heterogeneity in non-IID data without accounting for domain variations due to feature heterogeneity.
Title: Impact Charts: A Tool for Identifying Systematic Bias in Social Systems and Data
Authors: Darren Erik Vengroff
Affiliations: Film Independent
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: We introduce impact charts and apply them to problems of systematic bias encoded in three different data sets. Impact charts are highly visual, making the effects they find easy to understand by both domain experts and non-experts. Impact charts are nonlinear and non-parametric, so they are able to identify structural biases whose functional forms are not a priori well understood.
Title: WorldBench: Quantifying Geographic Disparities in LLM Factual Recall
Authors: Mazda Moayeri; Elham Tabassi; Soheil Feizi
Affiliations: ; Michigan State University; 
Year: 2024
Domain: LLMs & NLP

Abstract: As large language models (LLMs) continue to improve and gain popularity, some may use the models to recall facts, despite well documented limitations with LLM factuality. Towards ensuring that models work reliably for all, we seek to uncover if geographic disparities emerge when asking an LLM the same question about different countries. To this end, we present WorldBench, a dynamic and flexible benchmark composed of per-country data from the World Bank. In extensive experiments on state of the art open and closed source models, including GPT-4, Gemini, Llama-2, and Vicuna, to name a few, we find significant biases based on region and income level. For example, error rates are 1.5 times higher for countries from Sub-Saharan Africa compared to North American countries. We observe these disparities to be consistent over 20 LLMs and 11 individual World Bank indicators (i.e. specific statistics, such as population or CO2 emissions). WorldBench also enables automatic detection of citation hallucination, where models cite the World Bank itself while providing false statistics, and a manner to assess when an LLM’s stored facts begin to go out of date. We hope our benchmark will draw attention to geographic disparities in existing LLMs and facilitate the remedying of these biases.
Title: The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models
Authors: Abeba Birhane; Sepehr Dehdashtian; Vinay Uday Prabhu; Vishnu Naresh Boddeti
Affiliations: Trinity College Dublin; Michigan State University; ; Michigan State University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: ‘Scale the model, scale the data, scale the GPU farms’ is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts on model performance remain under-explored. This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the downstream impact of dataset scaling on 14 visio-linguistic models (VLMs) trained on the LAION400-M and LAION-2B datasets by measuring racial and gender bias using the Chicago Face Dataset (CFD) as the probe. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images as offensive non-human classes such as chimpanzee, gorilla, and orangutan decreased, but misclassifying the same images as human offensive classes such as criminal increased. Furthermore, of the 14 Vision Transformer-based VLMs we evaluated, the probability of predicting an image of a Black man and a Latino man as criminal increases by 65% and 69%, respectively, when the dataset is scaled from 400M to 2B samples for the larger ViT-L models. Conversely, for the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples. We ground the model audit results in a qualitative and historical analysis, reflect on our findings and their implications for dataset curation practice, and close with a summary of mitigation mechanisms and ways forward. All the meta-datasets curated in this endeavor and the code used are shared at: https://github.com/SepehrDehdashtian/the-dark-side-of-dataset-scaling. Content warning: This article contains racially dehumanising and offensive descriptions.
Title: Speaking of Accent: A Content Analysis of Accent Misconceptions in ASR Research
Authors: Kerri Prinos; Neal Patwari; Cathleen Power
Affiliations: Washington University in St. Louis; Washington University in St. Louis; 
Year: 2024
Domain: LLMs & NLP

Abstract: Automatic speech recognition (ASR) researchers are working to address the differing transcription performance of ASR by accent or dialect. However, research often has a limited view of accent in ways that reproduce discrimination and limit the scope of potential solutions. In this paper we present a content analysis of 22 papers published in 2022 in top conferences and journals on the topic of accent and ASR. We report on how accent is sometimes mistakenly viewed as something some people don’t have; as having a default; and being an attribute only of the speaker, and not of the listener. We discuss the implications on research and provide recommendations to researchers who hope to reduce ASR biases by accent.