bluuebunny/crossref_metadata_2025 huggingface dataset:
Legitimating Emotion Tracking Technologies in Driver Monitoring Systems
Authors: Doerfler Aaron , Stark Luke
Published: November 2024

Contemporary automobiles are now incorporating digital technologies, including emotion recognition technologies intended to monitor and sometimes intervene on the driver’s mood, attentiveness, or emotional state. We investigate how the firms producing these technologies justify and legitimate their design, production, and use, and how these discourses of legitimation paint a picture of the desired social role of emotion recognition in the automotive sector. Through a critical discourse analysis of patents, advertising, and promotional materials from industry-leading companies Cerence and Affectiva/Smart Eye, we argue both companies use potentially spurious arguments about the accuracy of emotion recognition to rationalize their products. Both companies also use a variety of other legitimation techniques around driver safety, individual personalization, and increased productivity to re-frame the social aspects of digitally mediated autonomous vehicles on their terms.
“I Don’t See Myself Represented Here at All”: User Experiences of Stable Diffusion Outputs Containing Representational Harms across Gender Identities and Nationalities
Authors: Ghosh Sourojit , Lutz Nina , Caliskan Aylin
Published: November 2024

Though research into text-to-image generators (T2Is) such as Stable Diffusion has demonstrated their amplification of societal biases and potentials to cause harm, such research has primarily relied on computational methods instead of seeking information from real users who experience harm, which is a significant knowledge gap. In this paper, we conduct the largest human subjects study of Stable Diffusion, with a combination of crowdsourced data from 133 crowdworkers and 14 semi-structured interviews across diverse countries and genders. Through a mixed-methods approach of intra-set cosine similarity hierarchies (i.e., comparing multiple Stable Diffusion outputs for the same prompt with each other to examine which result is `closest' to the prompt) and qualitative thematic analysis, we first demonstrate a large disconnect between user expectations for Stable Diffusion outputs with those generated, evidenced by a set of Stable Diffusion renditions of `a Person' providing images far away from such expectations. We then extend this finding of general dissatisfaction into highlighting representational harms caused by Stable Diffusion upon our subjects, especially those with traditionally marginalized identities, subjecting them to incorrect and often dehumanizing stereotypes about their identities. We provide recommendations for a harm-aware approach to (re)design future versions of Stable Diffusion and other T2Is.
Interpretations, Representations, and Stereotypes of Caste within Text-to-Image Generators
Authors: Ghosh Sourojit
Published: November 2024

The surge in the popularity of text-to-image generators (T2Is) has been matched by extensive research into ensuring fairness and equitable outcomes, with a focus on how they impact society. However, such work has typically focused on globally-experienced identities or centered Western contexts. In this paper, we address interpretations, representations, and stereotypes surrounding a tragically underexplored context in T2I research: caste. We examine how the T2I Stable Diffusion displays people of various castes, and what professions they are depicted as performing. Generating 100 images per prompt, we perform CLIP-cosine similarity comparisons with default depictions of an `Indian person’ by Stable Diffusion, and explore patterns of similarity. Our findings reveal how Stable Diffusion outputs perpetuate systems of `castelessness’, equating Indianness with high-castes and depicting caste-oppressed identities with markers of poverty. In particular, we note the stereotyping and representational harm towards the historically-marginalized Dalits, prominently depicted as living in rural areas and always at protests. Our findings underscore a need for a caste-aware approach towards T2I design, and we conclude with design recommendations.
The PPOu Framework: A Structured Approach for Assessing the Likelihood of Malicious Use of Advanced AI Systems
Authors: Goldstein Josh A. , Sastry Girish
Published: November 2024

The diffusion of increasingly capable AI systems has produced concern that bad actors could intentionally misuse current or future AI systems for harm. Governments have begun to create new entities—such as AI Safety Institutes—tasked with assessing these risks. However, approaches for risk assessment are currently fragmented and would benefit from broader disciplinary expertise. As it stands, it is often unclear whether concerns about malicious use misestimate the likelihood and severity of the risks. This article advances a conceptual framework to review and structure investigation into the likelihood of an AI system (X) being applied to a malicious use (Y). We introduce a three-stage framework of (1) Plausibility (can X be used to do Y at all?), (2) Performance (how well does X do Y?), and (3) Observed use (do actors use X to do Y in practice?). At each stage, we outline key research questions, methodologies, benefits and limitations, and the types of uncertainty addressed. We also offer ideas for directions to improve risk assessment moving forward.
Compassionate AI for Moral Decision-Making, Health, and Well-Being
Authors: Graves Mark , Compson Jane
Published: November 2024

The rapid expansion of artificial intelligence (AI) technology promises plausible increases to human flourishing, health, and well-being but raises concerns about possible harms and increased suffering. By making AI compassionate, the alleviation of suffering becomes explicit, rather than proxied, and potential harms caused by AI automation can be turned into benefits. Compassionate healthcare is beneficial for patient health outcomes and satisfaction and improves caregiver resilience and burnout. AI automation has many benefits but may interfere with patient care and autonomy. Incorporating compassion into healthcare reduces potential harms, increases health benefits and well-being, and can protect patient autonomy while providing more responsive and equitable care.

Whether and how one conceives of AI as plausibly compassionate depends on ethical concerns and cultural context, including assumptions about human nature and AI personhood. Insights from Buddhism have contributed to scholarship on compassion and can extend incomplete Western perspectives on AI possibilities and limitations. Psychological research on the elements of compassion can guide development of compassionate AI and its incorporation into healthcare. Compassionate AI can be deployed especially into application areas where compassion plays an essential role with high demands on the compassion capacity of caregivers, such as dementia eldercare and palliative care.
Identifying Implicit Social Biases in Vision-Language Models
Authors: Hamidieh Kimia , Zhang Haoran , Gerych Walter , Hartvigsen Thomas , Ghassemi Marzyeh
Published: November 2024

Vision-language models, like CLIP (Contrastive Language Image Pretraining), are becoming increasingly popular for a wide range of multimodal retrieval tasks. However, prior work has shown that large language and deep vision models can learn historical biases contained in their training sets, leading to perpetuation of stereotypes and potential downstream harm. In this work, we conduct a systematic analysis of the social biases that are present in CLIP, with a focus on the interaction between image and text modalities. We first propose a taxonomy of social biases called So-B-It, which contains 374 words categorized across ten types of bias. Each type can lead to societal harm if associated with a particular demographic group. Using this taxonomy, we examine images retrieved by CLIP from a facial image dataset using each word as part of a prompt. We find that CLIP frequently displays undesirable associations between harmful words and specific demographic groups, such as retrieving mostly pictures of Middle Eastern men when asked to retrieve images of a "terrorist". Finally, we conduct an analysis of the source of such biases, by showing that the same harmful stereotypes are also present in a large image-text dataset used to train CLIP models for examples of biases that we find. Our findings highlight the importance of evaluating and addressing bias in vision-language models, and suggest the need for transparency and fairness-aware curation of large pre-training datasets.
ExploreGen: Large Language Models for Envisioning the Uses and Risks of AI Technologies
Authors: Herdel Viviane , Šćepanović Sanja , Bogucka Edyta , Quercia Daniele
Published: November 2024

Responsible AI design is increasingly seen as an imperative by both AI developers and AI compliance experts. One of the key tasks is envisioning AI technology uses and risks. Recent studies on the model and data cards reveal that AI practitioners struggle with this task due to its inherently challenging nature. Here, we demonstrate that leveraging a Large Language Model (LLM) can support AI practitioners in this task by enabling reflexivity, brainstorming, and deliberation, especially in the early design stages of the AI development process. We developed an LLM framework, ExploreGen, which generates realistic and varied uses of AI technology, including those overlooked by research, and classifies their risk level based on the EU AI Act regulation. We evaluated our framework using the case of Facial Recognition and Analysis technology in nine user studies with 25 AI practitioners. Our findings show that ExploreGen is helpful to both developers and compliance experts. They rated the uses as realistic and their risk classification as accurate (94.5%). Moreover, while unfamiliar with many of the uses, they rated them as having high adoption potential and transformational impact.
Afrofuturist Values for the Metaverse (Extended Abstract)
Authors: Hice-Fromille Theresa , Papazoglakis Sarah
Published: November 2024

Many emerging technologies, such as the immersive VR and AR devices forming the metaverse, are not just reminiscent of but inspired by devices found in popular science fiction texts. Yet, the stories that these technologies are drawn from do not often center marginalized communities and people of color. In this article, we propose that builders and users of these technologies turn to diverse creative texts as inspiration for the ethical codes that will shape the ways that these technologies are built and used. A study of 39 speculative fiction texts, including 20 that we identified as Afrofuturist, revealed three overarching themes that serve as recommendations for the creation and maintenance of a diverse and inclusive metaverse: Collective Power, Inclusive Engagement, and Cultural Specificity. We outline each recommendation through a textual analysis of three Afrofuturist texts – Esi Edugyan’s Washington Black (2018), Roger Ross Williams’ Traveling While Black (2019), and Ryan Coogler’s Black Panther (2018) – and specify the undercurrents of collectivity and co-production that bind them together. We suggest collaborative and critical reading methods for industry professionals and community members which may help to shape democratic processes governing the future of AI.
Breaking the Global North Stereotype: A Global South-centric Benchmark Dataset for Auditing and Mitigating Biases in Facial Recognition Systems
Authors: Jaiswal Siddharth , Ganai Animesh , Dash Abhisek , Ghosh Saptarshi , Mukherjee Animesh
Published: November 2024

Facial Recognition Systems (FRSs) are being developed and deployed all around the world at unprecedented rates. Most platforms are designed in a limited set of countries, but deployed in other regions too, without adequate checkpoints for region-specific requirements. This is especially problematic for Global South countries which lack strong legislation to safeguard persons facing disparate performance of these systems. A combination of unavailability of datasets, lack of understanding of how FRSs function and low-resource bias mitigation measures accentuate the problems at hand. In this work, we propose a self-curated face dataset composed of 6,579 unique male and female sports-persons (cricket players) from eight countries around the world. More than 50% of the dataset is composed of individuals from the Global South countries and is demographically diverse. To aid adversarial audits and robust model training, we curate four adversarial variants of each image in the dataset, leading to more than 40,000 distinct images. We also use this dataset to benchmark five popular facial recognition systems (FRSs), including both commercial and open-source FRSs, for the task of gender prediction (and country prediction for one of the open-source models as an example of red-teaming). Experiments on industrial FRSs reveal accuracies ranging from 98.2% (in case of Azure) to 38.1% (in case of Face++), with a large disparity between males and females in the Global South (max difference of 38.5% in case of Face++). Biases are also observed in all FRSs between females of the Global North and South (max difference of ~50%). A Grad-CAM analysis shows that the nose, forehead and mouth are the regions of interest for one of the open-source FRSs. Based on this crucial observation, we design simple, low-resource bias mitigation solutions using few-shot and novel contrastive learning techniques that demonstrate a significant improvement in accuracy with disparity between males and females reducing from 50% to 1.5% in one of the settings. For the red-teaming experiment using the open-source Deepface model we observe that simple fine-tuning is not very useful while contrastive learning brings steady benefits.
Vernacularizing Taxonomies of Harm is Essential for Operationalizing Holistic AI Safety
Authors: Kennedy Wm. Matthew , Vargas Campos Daniel
Published: November 2024

Operationalizing AI ethics and safety principles and frameworks is essential to realizing the potential benefits and mitigating potential harms caused by AI systems. To that end, actors across industry, academia, and regulatory bodies have created formal taxonomies of harm to support operationalization efforts. These include novel “holistic” methods that go beyond exclusive reliance on technical benchmarking. However, our paper argues that such taxonomies are still too general to be readily implemented in sector-specific AI safety operationalization efforts, and especially in underresourced or “high-risk” sectors. This is because many sectors are constituted by discourses, norms, and values that “refract” or even directly conflict with those operating in society more broadly. Drawing from emerging anthropological theories of human rights, we propose that the process of “vernacularization”—a participatory, decolonial practice distinct from doctrinary “translation” (the dominant mode of AI safety operationalization)—can help bridge this gap. To demonstrate this point, we consider the education sector, and identify precisely how vernacularizing a leading taxonomy of harm leads to a clearer view of how harms AI systems may cause are substantially intensified when deployed in educational spaces. We conclude by discussing the generalizability of vernacularization as a useful AI safety methodology.
What’s Your Stake in Sustainability of AI?: An Informed Insider’s Guide
Authors: Kim Grace C. , Rothschild Annabel , DiSalvo Carl , DiSalvo Betsy
Published: November 2024

It's no secret that AI systems come with a significant environmental cost. This raises the question: What are the roles and responsibilities of computing professionals regarding the sustainability of AI? Informed by a year-long informal literature review on the subject, we employ stakeholder identification, analysis, and mapping to highlight the complex and interconnected roles that five major stakeholder groups (industry, practitioners, regulatory, advocacy, and the general public) play in the sustainability of AI. Swapping the traditional final step of stakeholder methods (stakeholder engagement) for entanglement, we demonstrate the inherent entwinement of choices made with regard to the development and maintenance of AI systems and the people who impact (or are impacted by) these choices. This entanglement should be understood as a system of human and non-human agents, with the implications of each choice ricocheting into the use of natural resources and climate implications. We argue that computing professionals (AI-focused or not) may belong to multiple stakeholder groups, and that we all have multiple roles to play in the sustainability of AI. Further, we argue that the nature of regulation in this domain will look unlike others in environmental preservation (e.g., legislation around water contaminants). As a result, we call for ongoing, flexible bodies and policies to move towards the regulation of AI from a sustainability angle, as well as suggest ways in which individual computing professionals can contribute to fighting the environmental and climate effects of AI.
On the Trade-offs between Adversarial Robustness and Actionable Explanations
Authors: Krishna Satyapriya , Agarwal Chirag , Lakkaraju Himabindu
Published: November 2024

As machine learning models are increasingly being employed in various high-stakes settings, it becomes important to ensure that predictions of these models are not only adversarially robust, but also readily explainable to relevant stakeholders. However, it is unclear if these two notions can be simultaneously achieved or if there exist trade-offs between them. In this work, we make one of the first attempts at studying the impact of adversarially robust models on actionable explanations which provide end users with a means for recourse. We theoretically and empirically analyze the cost (ease of implementation) and validity (probability of obtaining a positive model prediction) of recourses output by state-of-the-art algorithms when the underlying models are adversarially robust vs. non-robust. More specifically, we derive theoretical bounds on the differences between the cost and the validity of the recourses generated by state-of-the-art algorithms for adversarially robust vs. non-robust linear and non-linear models. Our empirical results with multiple real-world datasets validate our theoretical results and show the impact of varying degrees of model robustness on the cost and validity of the resulting recourses. Our analyses demonstrate that adversarially robust models significantly increase the cost and reduce the validity of the resulting recourses, thus shedding light on the inherent trade-offs between adversarial robustness and actionable explanations.
Racial and Neighborhood Disparities in Legal Financial Obligations in Jefferson County, Alabama
Authors: Lara Yejas Óscar , Joshi Aakanksha , Martinez Andrew , Nelson Leah , Speakman Skyler , Thompson Krysten , Nishimura Yuki , Bond Jordan , Varshney Kush R.
Published: November 2024

Legal financial obligations (LFOs) such as court fees and fines are commonly levied on individuals who are convicted of crimes. It is expected that LFO amounts should be similar across social, racial, and geographic subpopulations convicted of the same crime. This work analyzes the distribution of LFOs in Jefferson County, Alabama and highlights disparities across different individual and neighborhood demographic characteristics. Data-driven discovery methods are used to detect subpopulations that experience higher LFOs than the overall population of offenders. Critically, these discovery methods do not rely on pre-specified groups and can assist scientists and researchers investigate socially-sensitive hypotheses in a disciplined way. Some findings, such as individuals who are Black, live in Black-majority neighborhoods, or live in low-income neighborhoods tending to experience higher LFOs, are commensurate with prior expectation. However others, such as high LFO amounts in worthless instrument (bad check) cases experienced disproportionately by individuals living in affluent majority-white neighborhoods, are more surprising. More broadly than the specific findings, the methodology is shown to identify structural weaknesses that undermine the goal of equal justice under law that can be addressed through policy interventions.
“Democratizing AI” and the Concern of Algorithmic Injustice (Extended Abstract)
Authors: Lin Ting-an
Published: November 2024

The call to make artificial intelligence (AI) more democratic, or to “democratize AI,” is sometimes framed as a promising response for mitigating algorithmic injustice or making AI more aligned with social justice. However, the notion of “democratizing AI” is elusive, as the phrase has been associated with multiple meanings and practices, and the extent to which it may help mitigate algorithmic injustice is still underexplored. In this paper, based on a socio-technical understanding of algorithmic injustice, I examine three notable notions of democratizing AI and their associated measures—democratizing AI use, democratizing AI development, and democratizing AI governance—regarding their respective prospects and limits in response to algorithmic injustice. My examinations reveal that while some versions of democratizing AI bear the prospect of mitigating the concern of algorithmic injustice, others are somewhat limited and might even function to perpetuate unjust power hierarchies. This analysis thus urges a more fine-grained discussion on how to democratize AI and suggests that closer scrutiny of the power dynamics embedded in the socio-technical structure can help guide such explorations.
Foundations for Unfairness in Anomaly Detection - Case Studies in Facial Imaging Data
Authors: Livanos Michael , Davidson Ian
Published: November 2024

Deep anomaly detection (AD) is perhaps the most controversial of data analytic tasks as it identifies entities that are specifically targeted for further investigation or exclusion. Also controversial is the application of AI to facial data, in particular facial recognition. This work explores the intersection of these two areas to understand two core questions: Who these algorithms are being unfair to and equally important why. Recent work has shown that deep AD can be unfair to different groups despite being unsupervised with a recent study showing that for portraits of people: men of color are far more likely to be chosen to be outliers. We study the two main categories of AD algorithms: autoencoder-based and single-class-based which effectively try to compress all the instances and those that can not be easily compressed are deemed to be outliers. We experimentally verify sources of unfairness such as the under-representation of a group (e.g people of color are relatively rare), spurious group features (e.g. men are often photographed with hats) and group labeling noise (e.g. race is subjective). We conjecture that lack of compressibility is the main foundation and the others cause it but experimental results show otherwise and we present a natural hierarchy amongst them.
Uncovering the Gap: Challeging the Agential Nature of AI Responsibility Problems (Extended Abstract)
Authors: Llorca Albareda Joan
Published: November 2024

In this paper, I will argue that the responsibility gap arising from new AI systems is reducible to the problem of many hands and collective agency. Systematic analysis of the agential dimension of AI will lead me to outline a disjunctive between the two problems. Either we reduce individual responsibility gaps to the many hands, or we abandon the individual dimension and accept the possibility of responsible collective agencies. Moreover, I will adduce that this conclusion reveals an underlying weakness in AI ethics: the lack of attention to the question of the disciplinary boundaries of AI ethics. This absence has made it difficult to identify the specifics of the responsibility gap arising from new AI systems as compared to the responsibility gaps of other applied ethics. Lastly, I will be concerned with outlining these specific aspects.
Social Scoring Systems for Behavioral Regulation: An Experiment on the Role of Transparency in Determining Perceptions and Behaviors
Authors: Loefflad Carmen , Chen Mo , Grossklags Jens
Published: November 2024

Recent developments in artificial intelligence research have advanced the spread of automated decision-making (ADM) systems used for regulating human behaviors. In this context, prior work has focused on the determinants of human trust in and the legitimacy of ADM systems, e.g., when used for decision support. However, studies assessing people's perceptions of ADM systems used for behavioral regulation, as well as the effect on behaviors and the overall impact on human communities are largely absent. In this paper, we experimentally investigate people's behavioral adaptations to, and their perceptions of an institutionalized decision-making system, which resembled a social scoring system. Using social scores as incentives, the system aimed at ensuring mutual fair treatment between members of experimental communities. We explore how the provision of transparency affected people’s perceptions, behaviors, as well as the well-being of the communities. While a non-transparent scoring system led to disparate impacts both within as well as across communities, transparency helped people develop trust in each other, create wealth, and enabled them to benefit from the system in a more uniform manner. A transparent system was perceived as more effective, procedurally just, and legitimate, and led people to rely more strongly on the system. However, transparency also made people strongly discipline those with a low score. This suggests that social scoring systems that precisely disclose past behaviors may also impose significant discriminatory consequences on individuals deemed non-compliant.
Beyond Participatory AI
Authors: Maas Jonne , Moreno Inglés Aarón
Published: November 2024

The ‘participatory turn’ in AI design has received much attention in the literature. In this paper, we provide various arguments and proposals to move the discussion of participatory AI beyond its current state and towards stakeholder empowerment. The participatory AI literature points to Arnstein’s understanding of ‘citizen power’ as the right approach to participation. Although we agree with this general idea, we argue that there is a lack of depth in analyzing the legal, economic, and political arrangements required for a genuine redistribution of power to prioritize AI stakeholders. We highlight two domains on which the current discourse on participatory AI needs to expand. These are (1) the legal-institutional background that could provide ‘participation teeth’ for stakeholder empowerment and (2) the political economy of AI production that fosters such power asymmetries between AI developers and other stakeholders. We conclude by offering ways forward to explore alternative legal arrangements and ownership models for participatory AI.
The Code That Binds Us: Navigating the Appropriateness of Human-AI Assistant Relationships
Authors: Manzini Arianna , Keeling Geoff , Alberts Lize , Vallor Shannon , Morris Meredith Ringel , Gabriel Iason
Published: November 2024

The development of increasingly agentic and human-like AI assistants, capable of performing a wide range of tasks on user's behalf over time, has sparked heightened interest in the nature and bounds of human interactions with AI. Such systems may indeed ground a transition from task-oriented interactions with AI, at discrete time intervals, to ongoing relationships -- where users develop a deeper sense of connection with and attachment to the technology. This paper investigates what it means for relationships between users and advanced AI assistants to be appropriate and proposes a new framework to evaluate both users' relationships with AI and developers' design choices. We first provide an account of advanced AI assistants, motivating the question of appropriate relationships by exploring several distinctive features of this technology. These include anthropomorphic cues and the longevity of interactions with users, increased AI agency, generality and context ambiguity, and the forms and depth of dependence the relationship could engender. Drawing upon various ethical traditions, we then consider a series of values, including benefit, flourishing, autonomy and care, that characterise appropriate human interpersonal relationships. These values guide our analysis of how the distinctive features of AI assistants may give rise to inappropriate relationships with users. Specifically, we discuss a set of concrete risks arising from user--AI assistant relationships that: (1) cause direct emotional or physical harm to users, (2) limit opportunities for user personal development, (3) exploit user emotional dependence, and (4) generate material dependencies without adequate commitment to user needs. We conclude with a set of recommendations to address these risks.
Measuring Human-AI Value Alignment in Large Language Models
Authors: Norhashim Hakim , Hahn Jungpil
Published: November 2024

This paper seeks to quantify the human-AI value alignment in large language models. Alignment between humans and AI has become a critical area of research to mitigate potential harm posed by AI. In tandem with this need, developers have incorporated a values-based approach towards model development where ethical principles are integrated from its inception. However, ensuring that these values are reflected in outputs remains a challenge. In addition, studies have noted that models lack consistency when producing outputs, which in turn can affect their function. Such variability in responses would impact human-AI value alignment as well, particularly where consistent alignment is critical. Fundamentally, the task of uncovering a model’s alignment is one of explainability – where understanding how these complex models behave is essential in order to assess their alignment.

This paper examines the problem through a case study of GPT-3.5. By repeatedly prompting the model with scenarios based on a dataset of moral stories, we aggregate the model’s alignment with human values to produce a human-AI value alignment metric. Moreover, by using a comprehensive taxonomy of human values, we uncover the latent value profile represented by these outputs, thereby determining the extent of human-AI value alignment.
Face the Facts: Using Face Averaging to Visualize Gender-by-Race Bias in Facial Analysis Algorithms
Authors: Owens Kentrell , Freiburger Erin , Hutchings Ryan , Sim Mattea , Hugenberg Kurt , Roesner Franziska , Kohno Tadayoshi
Published: November 2024

We applied techniques from psychology --- typically used to visualize human bias --- to facial analysis systems, providing novel approaches for diagnosing and communicating algorithmic bias. First, we aggregated a diverse corpus of human facial images (N=1492) with self-identified gender and race. We tested four automated gender recognition (AGR) systems and found that some exhibited intersectional gender-by-race biases. Employing a technique developed by psychologists --- face averaging --- we created composite images to visualize these systems' outputs. For example, we visualized what an "average woman" looks like, according to a system's output. Second, we conducted two online experiments wherein participants judged the bias of hypothetical AGR systems. The first experiment involved participants (N=228) from a convenience sample. When depicting the same results in different formats, facial visualizations communicated bias to the same magnitude as statistics. In the second experiment with only Black participants (N=223), facial visualizations communicated bias significantly more than statistics, suggesting that face averages are meaningful for communicating algorithmic bias.
Proxy Fairness under the European Data Protection Regulation and the AI Act: A Perspective of Sensitivity and Necessity
Authors: Papageorgiou Ioanna
Published: November 2024

This paper navigates the convergence of the European Data Protection Regulation and the AI Act within the paradigm of computational methods that operationalise fairness in the absence of demographic data, notably through the use of proxy variables and inferential techniques (Proxy Fairness). Particularly, it explores the legal nature of the data involved in Proxy Fairness under the European Data Protection Regulation, focusing on the legal notion of Sensitivity. Moreover, it examines the lawfulness of processing sensitive personal data for Proxy Fairness purposes under the AI Act, particularly focusing on the legal requirement of Necessity. Through this analysis, the paper aims to shed light on core aspects of the legitimacy of Proxy Fairness in the context of EU law, providing a normative foundation to this line of Fair-AI approaches.
A Model- and Data-Agnostic Debiasing System for Achieving Equalized Odds
Authors: Pinkava Thomas , McFarland Jack , Mashhadi Afra
Published: November 2024

As reliance on Machine Learning (ML) systems in real-world decision-making processes grows, ensuring these systems are free of bias against sensitive demographic groups is of increasing importance. Existing techniques for automatically debiasing ML models generally require access to either the models’ internal architectures, the models’ training datasets, or both. In this paper we outline the reasons why such requirements are disadvantageous, and present an alternative novel debiasing system that is both data- and model-agnostic. We implement this system as a Reinforcement Learning Agent and through extensive experiments show that we can debias a variety of target ML model architectures over three benchmark datasets. Our results show performance comparable to data- and/or model-gnostic state-of-the-art debiasers.
Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesis
Authors: Raj Chahat , Mukherjee Anjishnu , Caliskan Aylin , Anastasopoulos Antonios , Zhu Ziwei
Published: November 2024

Large Language Models (LLMs) perpetuate social biases, reflecting prejudices in their training data and reinforcing societal stereotypes and inequalities. Our work explores the potential of the Contact Hypothesis, a concept from social psychology for debiasing LLMs. We simulate various forms of social contact through LLM prompting to measure their influence on the model’s biases, mirroring how intergroup interactions can reduce prejudices in social contexts. We create a dataset of 108,000 prompts following a principled approach replicating social contact to measure biases in three LLMs (LLaMA 2, Tulu, and NousHermes) across 13 social bias dimensions. We propose a unique debiasing technique, Social Contact Debiasing (SCD), that instruction-tunes these models with unbiased responses to prompts. Our research demonstrates that LLM responses exhibit social biases when subject to contact probing, but more importantly, these biases can be significantly reduced by up to 40% in 1 epoch of instruction tuning LLaMA 2 following our SCD strategy.
Learning When Not to Measure: Theorizing Ethical Alignment in LLMs
Authors: Rathje William
Published: November 2024

LLMs and other forms of generative AI have shown immense promise in producing highly accurate epistemic judgements in domains as varied as law, education, and medicine – with GPT notably passing the legal Bar exam and various medical licensing exams. The safe extension of LLMs into safety-critical professional domains requires assurance not only of epistemic but ethical alignment. This paper adopts a theoretical and philosophical approach, drawing from metaethical theories to argue for a distinction hinging around quantitative, axiological comparability that separates Kantian ethics from not only the utilitarianism it is well-known to oppose, but from just distribution theories as well, which are key to debiasing LLM models. It presents the novel hypothesis that LLM ethical acquisition from both corpus induction and RLHF may encounter value conflicts between Kantian and just distribution principles that intensify as they come into improved alignment with both theories, hinging around the variability by which self-attention may statistically attend to the same characterizations as more person-like or more resource-like under distinct prompting strategies.
Estimating Environmental Cost Throughout Model’s Adaptive Life Cycle
Authors: Sangarya Vishwesh , Bradford Richard , Kim Jung-Eun
Published: November 2024

With the rapid increase in the research, development, and application of neural networks in the current era, there is a proportional increase in the energy needed to train and use models. Crucially, this is accompanied by the increase in carbon emissions into the environment. A sustainable and socially beneficial approach to reducing the carbon footprint and rising energy demands associated with the modern age of AI/deep learning is the adaptive and continuous reuse of models with regard to changes in the environment of model deployment or variations/changes in the input data. In this paper, we propose PreIndex, a predictive index to estimate the environmental and compute resources associated with model retraining to distributional shifts in data. PreIndex can be used to estimate environmental costs such as carbon emissions and energy usage when retraining from current data distribution to new data distribution. It also correlates with and can be used to estimate other resource indicators associated with deep learning, such as epochs, gradient norm, and magnitude of model parameter change. PreIndex requires only one forward pass of the data, following which it provides a single concise value to estimate resources associated with retraining to the new distribution shifted data. We show that PreIndex can be reliably used across various datasets, model architectures, different types, and intensities of distribution shifts. Thus, PreIndex enables users to make informed decisions for retraining to different distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here:
https://github.com/JEKimLab/AIES2024PreIndex
What Is Required for Empathic AI? It Depends, and Why That Matters for AI Developers and Users
Authors: Schaich Borg Jana , Read Hannah
Published: November 2024

Interest is growing in artificial empathy, but so is confusion about what artificial empathy is or needs to be. This confusion makes it challenging to navigate the technical and ethical issues that accompany empathic AI development. Here, we outline a framework for thinking about empathic AI based on the premise that different constellations of capabilities associated with empathy are important for different empathic AI applications. We describe distinctions of capabilities that we argue belong under the empathy umbrella, and show how three medical empathic AI use cases require different sets of these capabilities. We conclude by discussing why appreciation of the diverse capabilities under the empathy umbrella is important for both AI creators and users.
Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Prosocial Benchmark Dataset
Authors: Schmer-Galunder Sonja , Wheelock Ruta , Jalan Zaria , Chvasta Alyssa , Friedman Scott , Saltz Emily
Published: November 2024

With the growing prevalence of large language models, it is increasingly common to annotate datasets for machine learning using pools of crowd raters. However, these raters often work in isolation as individual crowdworkers. In this work, we regard annotation not merely as inexpensive, scalable labor, but rather as a nuanced interpretative effort to discern the meaning of what is being said in a text. We describe a novel, collaborative, and iterative annotator-in-the-loop methodology for annotation, resulting in a 'Bridging Benchmark Dataset' of comments relevant to bridging divides, annotated from 11,973 textual posts in the Civil Comments dataset. The methodology differs from popular anonymous crowd-rating annotation processes due to its use of an in-depth, iterative engagement with seven US-based raters to (1) collaboratively refine the definitions of the to-be-annotated concepts and then (2) iteratively annotate complex social concepts, with check-in meetings and discussions. This approach addresses some shortcomings of current anonymous crowd-based annotation work, and we present empirical evidence of the performance of our annotation process in the form of inter-rater reliability. Our findings indicate that collaborative engagement with annotators can enhance annotation methods, as opposed to relying solely on isolated work conducted remotely. We provide an overview of the input texts, attributes, and annotation process, along with the empirical results and the resulting benchmark dataset, categorized according to the following attributes: Alienation, Compassion, Reasoning, Curiosity, Moral Outrage, and Respect.
Automating Transparency Mechanisms in the Judicial System Using LLMs: Opportunities and Challenges
Authors: Shastri Ishana , Jain Shomik , Engelhardt Barbara , Wilson Ashia
Published: November 2024

Bringing more transparency to the judicial system for the purposes of increasing accountability often demands extensive effort from auditors who must meticulously sift through numerous disorganized legal case files to detect patterns of bias and errors. For example, the high-profile investigation into the Curtis Flowers case took seven reporters a full year to assemble evidence about the prosecutor's history of selecting racially biased juries. LLMs have the potential to automate and scale these transparency pipelines, especially given their demonstrated capabilities to extract information from unstructured documents. We discuss the opportunities and challenges of using LLMs to provide transparency in two important court processes: jury selection in criminal trials and housing eviction cases.
Public vs Private Bodies: Who Should Run Advanced AI Evaluations and Audits? A Three-Step Logic Based on Case Studies of High-Risk Industries
Authors: Stein Merlin , Gandhi Milan , Kriecherbauer Theresa , Oueslati Amin , Trager Robert
Published: November 2024

Artificial Intelligence (AI) Safety Institutes and governments worldwide are deciding whether they evaluate and audit advanced AI themselves, support a private auditor ecosystem or do both.
Auditing regimes have been established in a wide range of industry contexts to monitor and evaluate firms’ compliance with regulation. Auditing is a necessary governance tool to understand and manage the risks of a technology. This paper draws from nine such regimes to inform (i) who should audit which parts of advanced AI; and (ii) how much resources, competence and access public bodies may need to audit advanced AI effectively.
First, the effective responsibility distribution between public and private auditors depends heavily on specific industry and audit conditions. On the basis of advanced AI’s risk profile, the sensitivity of information involved in the auditing process, and the high costs of verifying safety and benefit claims of AI Labs, we recommend that public bodies become directly involved in safety critical, especially gray- and white-box, AI model audits. Governance and security audits, which are well-established in other industry contexts, as well as black-box model audits, may be more efficiently provided by a private market of auditors under public oversight.
Secondly, to effectively fulfill their role in advanced AI audits, public bodies need extensive access to models and facilities. Public bodies’ capacity should scale with the industry's risk level, size and market concentration, potentially requiring 100s of employees for auditing in large jurisdictions like the EU or US, like in nuclear safety and life sciences.
Misrepresented Technological Solutions in Imagined Futures: The Origins and Dangers of AI Hype in the Research Community
Authors: Thais Savannah
Published: November 2024

Technology does not exist in a vacuum; technological development, media representation, public perception, and governmental regulation cyclically influence each other to produce the collective understanding of a technology's capabilities, utilities, and risks. When these capabilities are overestimated, there is an enhanced risk of subjecting the public to dangerous or harmful technology, artificially restricting research and development directions, and enabling misguided or detrimental policy. The dangers of technological hype are particularly relevant in the rapidly evolving space of AI. Centering the research community as a key player in the development and proliferation of hype, we examine the origins and risks of AI hype to the research community and society more broadly and propose a set of measures that researchers, regulators, and the public can take to mitigate these risks and reduce the prevalence of unfounded claims about the technology.
Decolonial AI Alignment: Openness, Visesa-Dharma, and Including Excluded Knowledges
Authors: Varshney Kush R.
Published: November 2024

Prior work has explicated the coloniality of artificial intelligence (AI) development and deployment through mechanisms such as extractivism, automation, sociological essentialism, surveillance, and containment. However, that work has not engaged much with alignment: teaching behaviors to a large language model (LLM) in line with desired values, and has not considered a mechanism that arises within that process: moral absolutism---a part of the coloniality of knowledge. Colonialism has a history of altering the beliefs and values of colonized peoples; in this paper, I argue that this history is recapitulated in current LLM alignment practices and technologies. Furthermore, I suggest that AI alignment be decolonialized using three forms of openness: openness of models, openness to society, and openness to excluded knowledges. This suggested approach to decolonial AI alignment uses ideas from the argumentative moral philosophical tradition of Hinduism, which has been described as an open-source religion. One concept used is viśeṣa-dharma, or particular context-specific notions of right and wrong. At the end of the paper, I provide a suggested reference architecture to work toward the proposed framework.
A Relational Justification of AI Democratization
Authors: Wielinga Bauke , Buijsman Stefan
Published: November 2024

While much has been written about what democratized AI should look like, there has been surprisingly little attention for the normative grounds of AI democratization. Existing calls for AI democratization that do make explicit arguments broadly fall into two categories: outcome-based and legitimacy-based, corresponding to outcome-based and process-based views of procedural justice respectively. This paper argues that we should favor relational justifications of AI democratization to outcome-based ones, because the former additionally provide outcome-independent reasons for AI democratization. Moreover, existing legitimacy-based arguments often leave the why of AI democratization implicit and instead focus on the how. We present two relational arguments for AI democratization: one based on empirical findings regarding the perceived importance of relational features of decision-making procedures, and one based on Iris Marion Young’s conception of justice, according to which the main forms of injustice are domination and oppression. We show how these arguments lead to requirements for procedural fairness and thus also offer guidance on the how of AI democratization. Finally, we consider several objections to AI democratization, including worries concerning epistemic exploitation.