cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: "I'm Fully Who I Am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation
Authors: Anaelia Ovalle; Palash Goyal; Jwala Dhamala; Zachary Jaggers; Kai-Wei Chang; Aram Galstyan; Richard S. Zemel; Rahul Gupta
Affiliations: University of California, Los Angeles; Amazon (United States); Amazon (United States); Amazon (United States); University of California, Los Angeles; Amazon (United States); Amazon (United States); Amazon (United States)
Year: 2023
Domain: LLMs & NLP

Abstract: Warning: This paper contains examples of gender non-affirmative language which could be offensive, upsetting, and/or triggering. Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. Given the recent popularity and adoption of language generation technologies, the potential to further marginalize this population only grows. Although a multitude of NLP fairness literature focuses on illuminating and addressing gender biases, assessing gender harms for TGNB identities requires understanding how such identities uniquely interact with societal gender norms and how they differ from gender binary-centric perspectives. Such measurement frameworks inherently require centering TGNB voices to help guide the alignment between gender-inclusive NLP and whom they are intended to serve. Towards this goal, we ground our work in the TGNB community and existing interdisciplinary literature to assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation (OLG). This social knowledge serves as a guide for evaluating popular large language models (LLMs) on two key aspects: (1) misgendering and (2) harmful responses to gender disclosure. To do this, we introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community. We discover a dominance of binary gender norms reflected by the models; LLMs least misgendered subjects in generated text when triggered by prompts whose subjects used binary pronouns. Meanwhile, misgendering was most prevalent when triggering generation with singular they and neopronouns. When prompted with gender disclosures, TGNB disclosure generated the most stigmatizing language and scored most toxic, on average. Our findings warrant further research on how TGNB harms manifest in LLMs and serve as a broader case study toward concretely grounding the design of gender-inclusive AI in community voices and interdisciplinary literature.
Title: Multi-Dimensional Discrimination in Law and Machine Learning: A Comparative Overview
Authors: Arjun Roy; Jan Horstmann; Eirini Ntoutsi
Affiliations: Universität der Bundeswehr München; ; Universität der Bundeswehr München
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: AI-driven decision-making can lead to discrimination against certain individuals or social groups based on protected characteristics/attributes such as race, gender, or age. The domain of fairness-aware machine learning focuses on methods and algorithms for understanding, mitigating, and accounting for bias in AI/ML models. Still, thus far, the vast majority of the proposed methods assess fairness based on a single protected attribute, e.g. only gender or race. In reality, though, human identities are multi-dimensional, and discrimination can occur based on more than one protected characteristic, leading to the so-called “multi-dimensional discrimination” or “multi-dimensional fairness” problem. While well-elaborated in legal literature, the multi-dimensionality of discrimination is less explored in the machine learning community. Recent approaches in this direction mainly follow the so-called intersectional fairness definition from the legal domain, whereas other notions like additive and sequential discrimination are less studied or not considered thus far. In this work, we overview the different definitions of multi-dimensional discrimination/fairness in the legal domain as well as how they have been transferred/ operationalized (if) in the fairness-aware machine learning domain. By juxtaposing these two domains, we draw the connections, identify the limitations, and point out open research directions.
Title: Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
Authors: Federico Bianchi; Pratyusha Kalluri; Esin Durmus; Faisal Ladhak; Myra Cheng; Debora Nozza; Tatsunori Hashimoto; Dan Jurafsky; James Zou; Aylin Caliskan
Affiliations: Stanford University; Stanford University; Stanford University; Columbia University; Stanford University; Bocconi University; Stanford University; Stanford University; Stanford University; University of Washington
Year: 2023
Domain: LLMs & NLP

Abstract: Machine learning models that convert user-written text descriptions into images are now widely available online and used by millions of users to generate millions of images a day. We investigate the potential for these models to amplify dangerous and complex stereotypes. We find a broad range of ordinary prompts produce stereotypes, including prompts simply mentioning traits, descriptors, occupations, or objects. For example, we find cases of prompting for basic traits or social roles resulting in images reinforcing whiteness as ideal, prompting for occupations resulting in amplification of racial and gender disparities, and prompting for objects resulting in reification of American norms. Stereotypes are present regardless of whether prompts explicitly mention identity and demographic language or avoid such language. Moreover, stereotypes persist despite mitigation strategies; neither user attempts to counter stereotypes by requesting images with specific counter-stereotypes nor institutional attempts to add system “guardrails” have prevented the perpetuation of stereotypes. Our analysis justifies concerns regarding the impacts of today’s models, presenting striking exemplars, and connecting these findings with deep insights into harms drawn from social scientific and humanist disciplines. This work contributes to the effort to shed light on the uniquely complex biases in language-vision models and demonstrates the ways that the mass deployment of text-to-image generation models results in mass dissemination of stereotypes and resulting harms.
Title: Does AI-Assisted Fact-Checking Disproportionately Benefit Majority Groups Online?
Authors: Terrence S. Neumann; Nicholas Wolczynski
Affiliations: The University of Texas at Austin; The University of Texas at Austin
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In recent years, algorithms have been incorporated into fact-checking pipelines. They are used not only to flag previously fact-checked misinformation, but also to provide suggestions about which trending claims should be prioritized for fact-checking - a paradigm called ‘check-worthiness.’ While several studies have examined the accuracy of these algorithms, none have investigated how the benefits from these algorithms (via reduction in exposure to misinformation) are distributed amongst various online communities. In this paper, we investigate how diverse representation across multiple stages of the AI development pipeline affects the distribution of benefits from AI-assisted fact-checking for different online communities. We simulate information propagation through the network using our novel Topic-Aware, Community-Impacted Twitter (TACIT) simulator on a large Twitter followers network, tuned to produce realistic cascades of true and false information across multiple topics. Finally, using simulated data as a test bed, we implement numerous algorithmic fact-checking interventions that explicitly account for notions of diversity. We find that both representative and egalitarian methods for sampling and labeling check-worthiness model training data can lead to network-wide benefit concentrated in majority communities, while incorporating diversity into how fact-checkers use algorithmic recommendations can actively reduce inequalities in benefits between majority and minority communities. These findings contribute to an important conversation around the responsible implementation of AI-assisted fact-checking by social media platforms and fact-checking organizations.
Title: Diverse Perspectives Can Mitigate Political Bias in Crowdsourced Content Moderation
Authors: Jacob Thebault-Spieker; Sukrit Venkatagiri; Naomi Mine; Kurt Luther
Affiliations: University of Wisconsin–Madison; University of Washington; University of Wisconsin–Madison; Virginia Tech
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In recent years, social media companies have grappled with defining and enforcing content moderation policies surrounding political content on their platforms, due in part to concerns about political bias, disinformation, and polarization. These policies have taken many forms, including disallowing political advertising, limiting the reach of political topics, fact-checking political claims, and enabling users to hide political content altogether. However, implementing these policies requires human judgement to label political content, and it is unclear how well human labelers perform at this task, or whether biases affect this process. Therefore, in this study we experimentally evaluate the feasibility and practicality of using crowd workers to identify political content, and we uncover biases that make it difficult to identify this content. Our results problematize crowds composed of seemingly interchangeable workers, and provide preliminary evidence that aggregating judgements from heterogeneous workers may help mitigate political biases. In light of these findings, we identify strategies to achieving fairer labeling outcomes, while also better supporting crowd workers at this task and potentially mitigating biases.
Title: Preventing Discriminatory Decision-Making in Evolving Data Streams
Authors: Zichong Wang; Nripsuta Ani Saxena; Tongjia Yu; Sneha Karki; Tyler Zetty; Israat Haque; Shan Zhou; Dukka B. KC; Ian Stockwell; Xuyu Wang; Albert Bifet; Wenbin Zhang
Affiliations: Michigan Technological University; University of Southern California; Columbia University; Michigan Technological University; Michigan Technological University; Dalhousie University; Michigan Technological University; Michigan Technological University; University System of Maryland; Florida International University; University of Waikato; Michigan Technological University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Bias in machine learning has rightly received significant attention over the past decade. However, most fair machine learning (fair-ML) works to address bias in decision-making systems has focused solely on the offline setting. Despite the wide prevalence of online systems in the real world, work on identifying and correcting bias in the online setting is severely lacking. The unique challenges of the online environment make addressing bias more difficult than in the offline setting. First, Streaming Machine Learning (SML) algorithms must deal with the constantly evolving real-time data stream. Secondly, they need to adapt to changing data distributions (concept drift) to make accurate predictions on new incoming data. Incorporating fairness constraints into this already intricate task is not straightforward. In this work, we focus on the challenges of achieving fairness in biased data streams while accounting for the presence of concept drift, accessing one sample at a time. We present Fair Sampling over Stream (FS2), a novel fair rebalancing approach capable of being integrated with SML classification algorithms. Furthermore, we devise the first unified performance-fairness metric, Fairness Bonded Utility (FBU), to efficiently evaluate and compare the trade-offs between performance and fairness across various bias mitigation methods. FBU simplifies the comparison of fairness-performance trade-offs of multiple techniques through one unified and intuitive evaluation, allowing model designers to easily choose a technique. Overall, extensive evaluations show our measures surpass those of other fair online techniques previously reported in the literature.
Title: Fairer Together: Mitigating Disparate Exposure in Kemeny Rank Aggregation
Authors: Kathleen Cachel; Elke A. Rundensteiner
Affiliations: Worcester Polytechnic Institute; Worcester Polytechnic Institute
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: In social choice, traditional Kemeny rank aggregation combines the preferences of voters, expressed as rankings, into a single consensus ranking without consideration for how this ranking may unfairly affect marginalized groups (i.e., racial or gender). Developing fair rank aggregation methods is critical due to their societal influence in applications prioritizing job applicants, funding proposals, and scheduling medical patients. In this work, we introduce the Fair Exposure Kemeny Aggregation Problem (FairExp-kap) for combining vast and diverse voter preferences into a single ranking that is not only a suitable consensus, but ensures opportunities are not withheld from marginalized groups. In formalizing FairExp-kap, we extend the fairness of exposure notion from information retrieval to the rank aggregation context and present a complimentary metric for voter preference representation. We design algorithms for solving FairExp-kap that explicitly account for position bias, a common ranking-based concern that end-users pay more attention to higher ranked candidates. epik solves FairExp-kap exactly by incorporating non-pairwise fairness of exposure into the pairwise Kemeny optimization; while the approximate epira is a candidate swapping algorithm, that guarantees ranked candidate fairness. Utilizing comprehensive synthetic simulations and six real-world datasets, we show the efficacy of our approach illustrating that we succeed in mitigating disparate group exposure unfairness in consensus rankings, while maximally representing voter preferences.
Title: Detecting Disparities in Police Deployments Using Dashcam Data
Authors: Matt Franchi; J.D. Zamfirescu-Pereira; Wendy Ju; Emma Pierson
Affiliations: Cornell University; University of California, Berkeley; Cornell University; Cornell University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Large-scale policing data is vital for detecting inequity in police behavior and policing algorithms. However, one important type of policing data remains largely unavailable within the United States: aggregated police deployment data capturing which neighborhoods have the heaviest police presences. Here we show that disparities in police deployment levels can be quantified by detecting police vehicles in dashcam images of public street scenes. Using a dataset of 24,803,854 dashcam images from rideshare drivers in New York City, we find that police vehicles can be detected with high accuracy (average precision 0.82, AUC 0.99) and identify 233,596 images which contain police vehicles. There is substantial inequality across neighborhoods in police vehicle deployment levels. The neighborhood with the highest deployment levels has almost 20 times higher levels than the neighborhood with the lowest. Two strikingly different types of areas experience high police vehicle deployments — 1) dense, higher-income, commercial areas and 2) lower-income neighborhoods with higher proportions of Black and Hispanic residents. We discuss the implications of these disparities for policing equity and for algorithms trained on policing data.
Title: Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity
Authors: Joshua Gardner; Renzhe Yu; Quan Nguyen; Christopher Brooks; René F. Kizilcec
Affiliations: University of Washington; Columbia University; University of British Columbia; University of Michigan; Cornell University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We propose a framework and metrics for assessing the utility and fairness of student dropout prediction models that are transferred across institutions. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints, quantifying model biases for intersectional student identities, characterizing potential disparate impact due to these biases, and investigating the impact of various cross-institutional ensembling approaches on fairness and overall model performance. We perform this analysis on data representing over 200,000 enrolled students annually from four universities without sharing training data between institutions.
Title: Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias
Authors: Robert Wolfe; Yiwei Yang; Bill Howe; Aylin Caliskan
Affiliations: University of Washington; University of Washington; University of Washington; University of Washington
Year: 2023
Domain: LLMs & NLP

Abstract: Warning: The content of this paper may be upsetting or triggering.
Title: "I Wouldn't Say Offensive but...": Disability-Centered Perspectives on Large Language Models
Authors: Vinitha Gadiraju; Shaun K. Kane; Sunipa Dev; Alex Taylor; Ding Wang; Emily Denton; Robin Brewer
Affiliations: University of Colorado Boulder; Google (United States); Google (United States); City, University of London; Google (United States); Google (United States); University of Michigan
Year: 2023
Domain: LLMs & NLP

Abstract: Large language models (LLMs) trained on real-world data can inadvertently reflect harmful societal biases, particularly toward historically marginalized communities. While previous work has primarily focused on harms related to age and race, emerging research has shown that biases toward disabled communities exist. This study extends prior work exploring the existence of harms by identifying categories of LLM-perpetuated harms toward the disability community. We conducted 19 focus groups, during which 56 participants with disabilities probed a dialog model about disability and discussed and annotated its responses. Participants rarely characterized model outputs as blatantly offensive or toxic. Instead, participants used nuanced language to detail how the dialog model mirrored subtle yet harmful stereotypes they encountered in their lives and dominant media, e.g., inspiration porn and able-bodied saviors. Participants often implicated training data as a cause for these stereotypes and recommended training the model on diverse identities from disability-positive resources. Our discussion further explores representative data strategies to mitigate harm related to different communities through annotation co-design with ML researchers and developers.
Title: Disparities in Text-to-Image Model Concept Possession Across Languages
Authors: Michael Saxon; William Yang Wang
Affiliations: University of California Santa Barbara; University of California Santa Barbara
Year: 2023
Domain: LLMs & NLP

Abstract: We propose the notion of conceptual possession in generative text-to-image (T2I) systems, wherein a model is considered to possess a concept if it can generate a distinctive, correct, and self-consistent population of images for a simple prompt containing that concept. We use this idea to develop a model benchmark of multilingual parity in conceptual possession across a set of almost 200 tangible nouns across 7 languages: English, Spanish, German, Chinese, Japanese, Hebrew, and Indonesian. This technique allows us to estimate how well-suited a model is to a target language as well as identify model-specific weaknesses, spurious correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of multilinguality, and that despite its simplicity our method captures the necessary conditions for the impressive creative generative abilities users expect from T2I models. Our benchmark will guide future work in reducing disparities across languages, improving accessibility of these technologies.
Title: The Privacy-Bias Tradeoff: Data Minimization and Racial Disparity Assessments in U.S. Government
Authors: Jennifer King; Daniel E. Ho; Arushi Gupta; Victor Wu; Helen Webley-Brown
Affiliations: Stanford University; Stanford University; Stanford University; Stanford University; Massachusetts Institute of Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: An emerging concern in algorithmic fairness is the tension with privacy interests. Data minimization can restrict access to protected attributes, such as race and ethnicity, for bias assessment and mitigation. Less recognized is that for nearly 50 years, the federal government has been engaged in a large-scale experiment in data minimization, limiting (a) data sharing across federal agencies under the Privacy Act of 1974, and (b) data collection under the Paperwork Reduction Act. We document how this “privacy-bias tradeoff” has become an important battleground for fairness assessments in the U.S. government and provides rich lessons for resolving these tradeoffs. President Biden’s 2021 racial justice Executive Order 13,985 mandated that federal agencies conduct equity impact assessments (e.g., for racial disparities) of federal programs. We conduct a comprehensive assessment across high-volume claims agencies that affect many individuals, as well as all agencies filing “equity action plans,” with three findings. First, there is broad agreement in principle that equity impact assessments are important, with few parties raising privacy challenges in theory and many agencies proposing substantial efforts. Second, in practice, major agencies do not collect and may be affirmatively prohibited under the Privacy Act from linking demographic information. This has led to pathological results: until 2022, for instance, the US Dept. of Agriculture imputed race by “visual observation” when race information was not collected. Data minimization has meant that even where agencies want to acquire demographic information in principle, the legal, data infrastructure, and bureaucratic hurdles are severe. Third, we derive policy implications to address these barriers.
Title: On The Impact of Machine Learning Randomness on Group Fairness
Authors: Prakhar Ganesh; Hongyan Chang; Martin Strobel; Reza Shokri
Affiliations: National University of Singapore; National University of Singapore; National University of Singapore; National University of Singapore
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Statistical measures for group fairness in machine learning reflect the gap in performance of algorithms across different groups. These measures, however, exhibit a high variance between different training instances, which makes them unreliable for empirical evaluation of fairness. What causes this high variance? We investigate the impact on group fairness of different sources of randomness in training neural networks. We show that the variance in group fairness measures is rooted in the high volatility of the learning process on under-represented groups. Further, we recognize the dominant source of randomness as the stochasticity of data order during training. Based on these findings, we show how one can control group-level accuracy (i.e., model fairness), with high efficiency and negligible impact on the model’s overall performance, by simply changing the data order for a single epoch.