cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Frameworks, Methods and Shared Tasks: Connecting Participatory AI to Trustworthy AI Through a Systematic Review of Global Projects
Authors: María Isabel Magaña; Katie Shilton
Affiliations: Universidad de La Sabana; University of Maryland, College Park
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: As the discourse on responsible and trustworthy AI intensifies, Participatory AI (PAI) presents a compelling approach to the democratic development of automated technologies. But how should the field think about how and whether participatory methods increase trust in, and the trustworthiness of, AI systems? In response to the recent growth in PAI research, we conducted a systematic examination to understand the landscape of methods and theoretical lenses used in participatory AI projects, and to connect those methods and lenses to trust building. This paper analyzes 95 global PAI projects to understand the participatory landscape of trustworthy AI. Our review explores differences in theoretical frameworks, participation methods, and the details of shared tasks within the AI lifecycle across sectors and geographies. Our findings reveal an evolving definition of PAI, with actors implementing diverse methods and shared tasks. Focusing on shared tasks provides a lens for analyzing how participation can build trust in, and trustworthiness of, AI systems. Considering PAI through the lens of trust creates a framework for mapping PAI activities to both participant and broader public trust and illustrates gaps where PAI processes alone will not be enough to ensure trustworthy AI systems.
Title: Recourse, Repair, Reparation, &amp; Prevention: A Stakeholder Analysis of AI Supply Chains
Authors: Aspen Hopkins; Isabella Struckman; Kevin Klyman; Susan S. Silbey
Affiliations: IIT@MIT; IIT@MIT; IIT@MIT
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The AI industry is exploding in popularity, with increasing attention to potential harms and unwanted consequences. In the current digital ecosystem, AI deployments are often the product of AI supply chains (AISC): networks of outsourced models, data, and tooling through which multiple entities contribute to AI development and distribution. AI supply chains lack the modularity, redundancies, or conventional supply chain practices that enable identification, isolation, and easy correction of failures, exacerbating the already difficult processes of responding to ML-generated harms. As the stakeholders participating in and impacted by AISCs have scaled and diversified, so too have the risks they face. In this stakeholder analysis of AI supply chains, we consider who participates in AISCs, what harms they face, where sources of harm lie, and how market dynamics and power differentials inform the type and probability of remedies. Because AI supply chains are purposely invented and implemented, they may be designed to account for, rather than ignore, the complexities, consequences, and risks of deploying AI systems. To enable responsible design and management of AISCs, we offer a typology of responses to AISC-induced harms: recourse, repair, reparation or prevention. We apply this typology to stakeholders participating in a health-care AISC across three stylized markets $\unicode{x2013}$ vertical integration, horizontal integration, free market $\unicode{x2013}$ to illustrate how stakeholder positioning and power within an AISC may shape responses to an experienced harm.
Title: “You Cannot Sound Like GPT": Signs of language discrimination and resistance in computer science publishing
Authors: Haley Lepp; Daniel Scott Smith
Affiliations: Palo Alto University; Palo Alto University
Year: 2025
Domain: LLMs & NLP

Abstract: LLMs have been celebrated for their potential to help multilingual scientists publish their research.Rather than interpret LLMs as a solution, we hypothesize their adoption can be an indicator of existing linguistic exclusion in scientific writing.Using the case study of ICLR, an influential, international computer science conference, we examine how peer reviewers critique writing clarity.Analyzing almost 80,000 peer reviews, we find significant bias against authors associated with institutions in countries where English is less widely spoken.We see only a muted shift in the expression of this bias after the introduction of ChatGPT in late 2022.To investigate this unexpectedly minor change, we conduct interviews with 14 conference participants from across five continents.Peer reviewers describe associating certain features of writing with people of certain language backgrounds, and such groups in turn with the quality of scientific work.While ChatGPT masks some signs of language background, reviewers explain that they now use ChatGPT "style" and non-linguistic features as indicators of author demographics.Authors, aware of this development, described the ongoing need to remove features which could expose their "non-native" status to reviewers.Our findings offer insight into the role of ChatGPT in the reproduction of scholarly language ideologies which conflate producers of "good English" with producers of "good science." CCS Concepts Social and professional topics
Title: Guardian: A Runtime Framework for LLM-Based UI Exploration
Authors: Dezhi Ran; Hao Wang; Zihe Song; M.S. Wu; Yuan Cao; Ying Zhang; Wei Yang; Tao Xie
Affiliations: Peking University; Peking University; The University of Texas at Dallas; Peking University; Peking University; Peking University; The University of Texas at Dallas; Peking University
Year: 2024
Domain: LLMs & NLP

Abstract: Tests for feature-based UI testing have been indispensable for ensuring the quality of mobile applications (apps for short). The high manual labor costs to create such tests have led to a strong interest in automated feature-based UI testing, where an approach automatically explores the App under Test (AUT) to find correct sequences of UI events achieving the target test objective, given only a high-level test objective description. Given that the task of automated feature-based UI testing resembles conventional AI planning problems, large language models (LLMs), known for their effectiveness in AI planning, could be ideal for this task. However, our study reveals that LLMs struggle with following specific instructions for UI testing and replanning based on new information. This limitation results in reduced effectiveness of LLM-driven solutions for automated feature-based UI testing, despite the use of advanced prompting techniques. Toward addressing the preceding limitation, we propose Guardian, a runtime system framework to improve the effectiveness of automated feature-based UI testing by offloading computational tasks from LLMs with two major strategies. First, Guardian refines UI action space that the LLM can plan over, enforcing the instruction following of the LLM by construction. Second, Guardian deliberately checks whether the gradually enriched information invalidates previous planning by the LLM. Guardian removes the invalidated UI actions from the UI action space that the LLM can plan over, restores the state of the AUT to the state before the execution of the invalidated UI actions, and prompts the LLM to re-plan with the new UI action space. We instantiate Guardian with ChatGPT and construct a benchmark named FestiVal with 58 tasks from 23 highly popular apps. Evaluation results on FestiVal show that Guardian achieves 48.3
Title: Graph-Aware Contrasting for Multivariate Time-Series Classification
Authors: Yucheng Wang; Yuecong Xu; Jianfei Yang; Min Wu; Xiaoli Li; Lihua Xie; Zhenghua Chen
Affiliations: Institute for Infocomm Research; Agency for Science, Technology and Research; Nanyang Technological University; Agency for Science, Technology and Research; Institute for Infocomm Research; Nanyang Technological University; Agency for Science, Technology and Research
Year: 2024
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Contrastive learning, as a self-supervised learning paradigm, becomes popular for Multivariate Time-Series (MTS) classification. It ensures the consistency across different views of unlabeled samples and then learns effective representations for these samples. Existing contrastive learning methods mainly focus on achieving temporal consistency with temporal augmentation and contrasting techniques, aiming to preserve temporal patterns against perturbations for MTS data. However, they overlook spatial consistency that requires the stability of individual sensors and their correlations. As MTS data typically originate from multiple sensors, ensuring spatial consistency becomes essential for the overall performance of contrastive learning on MTS data. Thus, we propose Graph-Aware Contrasting for spatial consistency across MTS data. Specifically, we propose graph augmentations including node and edge augmentations to preserve the stability of sensors and their correlations, followed by graph contrasting with both node- and graph-level contrasting to extract robust sensor- and global-level features. We further introduce multi-window temporal contrasting to ensure temporal consistency in the data for each sensor. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance on various MTS classification tasks. The code is available at https://github.com/Frank-Wang-oss/TS-GAC.
Title: Complete Neural Networks for Complete Euclidean Graphs
Authors: Snir Hordan; Tal Amir; Steven J. Gortler; Nadav Dym
Affiliations: Technion – Israel Institute of Technology; Technion – Israel Institute of Technology; Harvard University Press; Technion – Israel Institute of Technology
Year: 2024
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Neural networks for point clouds, which respect their natural invariance to permutation and rigid motion, have enjoyed recent success in modeling geometric phenomena, from molecular dynamics to recommender systems. Yet, to date, no architecture with polynomial complexity is known to be complete, that is, able to distinguish between any pair of non-isomorphic point clouds. We fill this theoretical gap by showing that point clouds can be completely determined, up to permutation and rigid motion, by applying the 3-WL graph isomorphism test to the point cloud's centralized Gram matrix. Moreover, we formulate an Euclidean variant of the 2-WL test and show that it is also sufficient to achieve completeness. We then show how our complete Euclidean WL tests can be simulated by an Euclidean graph neural network of moderate size and demonstrate their separation capability on highly symmetrical point clouds.
Title: ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers
Authors: Yiming Wang; Jinyu Li
Affiliations: Microsoft Corporation; Microsoft Corporation
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Memory constraint of always-on devices is one of the major concerns when deploying speech processing models on these devices. While larger models trained with sufficiently large amount of data generally perform better, making them fit in the device memory is a demanding challenge. In this paper, we aim to reduce model size by reparameterizing model weights across Transformer encoder layers and assuming a special weight composition and structure. More specifically, inspired by ResNet and the more recent LoRA work, we propose an approach named ResidualTransformer, where each weight matrix in a Transformer layer comprises 1) a shared full-rank component with its adjacent layers, and 2) a unique low-rank component to itself. The low-rank matrices only account for a small amount of model size increase. In addition, we add diagonal weight matrices to improve modeling capacity of the low-rank matrices. Experiments of our 10k-hour speech recognition and speech translation tasks show that the Transformer encoder size can be reduced by ~3X with very slight performance degradation.
Title: Socially Fair k-Means Clustering
Authors: Mehrdad Ghadiri; Samira Samadi; Santosh Vempala
Affiliations: Georgia Institute of Technology; Max Planck Institute for Intelligent Systems; Georgia Institute of Technology
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: We show that the popular k-means clustering algorithm (Lloyd's heuristic), used for a variety of scientific data, can result in outcomes that are unfavorable to subgroups of data (e.g., demographic groups). Such biased clusterings can have deleterious implications for human-centric applications such as resource allocation. We present a fair k-means objective and algorithm to choose cluster centers that provide equitable costs for different groups. The algorithm, Fair-Lloyd, is a modification of Lloyd's heuristic for k-means, inheriting its simplicity, efficiency, and stability. In comparison with standard Lloyd's, we find that on benchmark datasets, Fair-Lloyd exhibits unbiased performance by ensuring that all groups have equal costs in the output k-clustering, while incurring a negligible increase in running time, thus making it a viable fair option wherever k-means is currently used.
Title: Fairness Without Demographics in Repeated Loss Minimization
Authors: Tatsunori Hashimoto; Megha Srivastava; Hongseok Namkoong; Percy Liang
Affiliations: Stanford University; Stanford University; Stanford University; Stanford University
Year: 2018
Domain: General Fairness & Bias Mitigation

Abstract: Machine learning models (e.g., speech recognizers) are usually trained to minimize average loss, which results in representation disparity---minority groups (e.g., non-native speakers) contribute less to the training objective and thus tend to suffer higher loss. Worse, as model accuracy affects user retention, a minority group can shrink over time. In this paper, we first show that the status quo of empirical risk minimization (ERM) amplifies representation disparity over time, which can even make initially fair models unfair. To mitigate this, we develop an approach based on distributionally robust optimization (DRO), which minimizes the worst case risk over all distributions close to the empirical distribution. We prove that this approach controls the risk of the minority group at each time step, in the spirit of Rawlsian distributive justice, while remaining oblivious to the identity of the groups. We demonstrate that DRO prevents disparity amplification on examples where ERM fails, and show improvements in minority group user satisfaction in a real-world text autocomplete task.
Title: Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Authors: Maria De-Arteaga; Alexey Romanov; Hanna Wallach; Jennifer Chayes; Christian Borgs; Alexandra Chouldechova; Sahin Geyik; Krishnaram Kenthapadi; Adam Tauman Kalai
Affiliations: Carnegie Mellon University; University of Massachusetts Lowell; Microsoft Research; Microsoft Research; Microsoft Research; Carnegie Mellon University; LinkedIn; LinkedIn; Microsoft Research
Year: 2019
Domain: LLMs & NLP

Abstract: We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators---such as first names and pronouns---in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are "scrubbed," and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.
Title: 50 Years of Test (Un)fairness: Lessons for Machine Learning
Authors: Ben Hutchinson; Margaret Mitchell
Affiliations: Google; Google
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Quantitative definitions of what is unfair and what is fair have been introduced in multiple disciplines for well over 50 years, including in education, hiring, and machine learning. We trace how the notion of fairness has been defined within the testing communities of education and hiring over the past half century, exploring the cultural and social context in which different fairness definitions have emerged. In some cases, earlier definitions of fairness are similar or identical to definitions of fairness in current machine learning research, and foreshadow current formal work. In other cases, insights into what fairness means and how to measure it have largely gone overlooked. We compare past and current notions of fairness along several dimensions, including the fairness criteria, the focus of the criteria (e.g., a test, a model, or its use), the relationship of fairness to individuals, groups, and subgroups, and the mathematical method for measuring fairness (e.g., classification, regression). This work points the way towards future research and measurement of (un)fairness that builds from our modern understanding of fairness while incorporating insights from the past.
Title: Fairness and Abstraction in Sociotechnical Systems
Authors: Andrew D. Selbst; danah boyd; Sorelle A. Friedler; Suresh Venkatasubramanian; Janet Vertesi
Affiliations: Data & Society Research Institute; Microsoft; Haverford College; University of Utah; Princeton University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: A key goal of the fair-ML community is to develop machine-learning based systems that, once introduced into a social context, can achieve social and legal outcomes such as fairness, justice, and due process. Bedrock concepts in computer science---such as abstraction and modular design---are used to define notions of fairness and discrimination, to produce fairness-aware learning algorithms, and to intervene at different stages of a decision-making pipeline to produce "fair" outcomes. In this paper, however, we contend that these concepts render technical interventions ineffective, inaccurate, and sometimes dangerously misguided when they enter the societal context that surrounds decision-making systems. We outline this mismatch with five "traps" that fair-ML work can fall into even as it attempts to be more context-aware in comparison to traditional data science. We draw on studies of sociotechnical systems in Science and Technology Studies to explain why such traps occur and how to avoid them. Finally, we suggest ways in which technical designers can mitigate the traps through a refocusing of design in terms of process rather than solutions, and by drawing abstraction boundaries to include social actors rather than purely technical ones.
Title: Problem Formulation and Fairness
Authors: Samir Passi; Solon Barocas
Affiliations: Cornell University; Cornell University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Formulating data science problems is an uncertain and difficult process. It requires various forms of discretionary work to translate high-level objectives or strategic goals into tractable problems, necessitating, among other things, the identification of appropriate target variables and proxies. While these choices are rarely self-evident, normative assessments of data science projects often take them for granted, even though different translations can raise profoundly different ethical concerns. Whether we consider a data science project fair often has as much to do with the formulation of the problem as any property of the resulting model. Building on six months of ethnographic fieldwork with a corporate data science team---and channeling ideas from sociology and history of science, critical data studies, and early writing on knowledge discovery in databases---we describe the complex set of actors and activities involved in problem formulation. Our research demonstrates that the specification and operationalization of the problem are always negotiated and elastic, and rarely worked out with explicit normative considerations in mind. In so doing, we show that careful accounts of everyday data science work can help us better understand how and why data science problems are posed in certain ways---and why specific formulations prevail in practice, even in the face of what might seem like normatively preferable alternatives. We conclude by discussing the implications of our findings, arguing that effective normative interventions will require attending to the practical work of problem formulation.
Title: Racial categories in machine learning
Authors: Sebastian Benthall; Bruce D. Haynes
Affiliations: New York University; University of California Davis
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Controversies around race and machine learning have sparked debate among computer scientists over how to design machine learning systems that guarantee fairness. These debates rarely engage with how racial identity is embedded in our social experience, making for sociological and psychological complexity. This complexity challenges the paradigm of considering fairness to be a formal property of supervised learning with respect to protected personal attributes. Racial identity is not simply a personal subjective quality. For people labeled "Black" it is an ascribed political category that has consequences for social differentiation embedded in systemic patterns of social inequality achieved through both social and spatial segregation. In the United States, racial classification can best be understood as a system of inherently unequal status categories that places whites as the most privileged category while signifying the Negro/black category as stigmatized. Social stigma is reinforced through the unequal distribution of societal rewards and goods along racial lines that is reinforced by state, corporate, and civic institutions and practices. This creates a dilemma for society and designers: be blind to racial group disparities and thereby reify racialized social inequality by no longer measuring systemic inequality, or be conscious of racial categories in a way that itself reifies race. We propose a third option. By preceding group fairness interventions with unsupervised learning to dynamically detect patterns of segregation, machine learning systems can mitigate the root cause of social disparities, social segregation and stratification, without further anchoring status categories of disadvantage.