cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Algorithmic Fairness in Predicting Opioid Use Disorder using Machine Learning
Authors: Angela E. Kilby
Affiliations: Northeastern University
Year: 2021
Domain: Health & Clinical AI

Abstract: There has been recent interest by payers, health care systems, and researchers in the development of machine learning and artificial intelligence models that predict an individual's probability of developing opioid use disorder. The scores generated by these algorithms can be used by physicians to tailor the prescribing of opioids for the treatment of pain, reducing or foregoing prescribing to individuals deemed to be at high risk, or increasing prescribing for patients deemed to be at low risk. This paper constructs a machine learning algorithm to predict opioid use disorder risk using commercially available claims data similar to those utilized in the development of proprietary opioid use disorder prediction algorithms. We study risk scores generated by the machine learning model in a setting with quasi-experimental variation in the likelihood that doctors prescribe opioids, generated by changes in the legal structure for monitoring physician prescribing. We find that machine-predicted risk scores do not appear to correlate at all with the individual-specific heterogeneous treatment effect of receiving opioids. The paper identifies a new source of algorithmic unfairness in machine learning applications for health care and precision medicine, arising from the researcher's choice of objective function. While precision medicine should guide physician treatment decisions based on the heterogeneous causal impact of a course of treatment for an individual, allocating treatments to individuals receiving the most benefit and recommending caution for those most likely to experience harmful side effects, ML models in health care are often trained on proxies like individual baseline risk, and are not necessarily informative in deciding who would most benefit, or be harmed, by a course of treatment.
Title: Fairness Violations and Mitigation under Covariate Shift
Authors: Harvineet Singh; Rina Singh; Vishwali Mhasawade; Rumi Chunara
Affiliations: New York University; New York University; New York University; New York University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: We study the problem of learning fair prediction models for unseen test sets distributed differently from the train set. Stability against changes in data distribution is an important mandate for responsible deployment of models. The domain adaptation literature addresses this concern, albeit with the notion of stability limited to that of prediction accuracy. We identify sufficient conditions under which stable models, both in terms of prediction accuracy and fairness, can be learned. Using the causal graph describing the data and the anticipated shifts, we specify an approach based on feature selection that exploits conditional independencies in the data to estimate accuracy and fairness metrics for the test set. We show that for specific fairness definitions, the resulting model satisfies a form of worst-case optimality. In context of a healthcare task, we illustrate the advantages of the approach in making more equitable decisions.
Title: Towards Fair Deep Anomaly Detection
Authors: Hongjing Zhang; Ian Davidson
Affiliations: University of California Davis; University of California Davis
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Anomaly detection aims to find instances that are considered unusual and is a fundamental problem of data science. Recently, deep anomaly detection methods were shown to achieve superior results particularly in complex data such as images. Our work focuses on deep one-class classification for anomaly detection which learns a mapping only from the normal samples. However, the non-linear transformation performed by deep learning can potentially find patterns associated with social bias. The challenge with adding fairness to deep anomaly detection is to ensure both making fair and correct anomaly predictions simultaneously. In this paper, we propose a new architecture for the fair anomaly detection approach (Deep Fair SVDD) and train it using an adversarial network to de-correlate the relationships between the sensitive attributes and the learned representations. This differs from how fairness is typically added namely as a regularizer or a constraint. Further, we propose two effective fairness measures and empirically demonstrate that existing deep anomaly detection methods are unfair. We show that our proposed approach can remove the unfairness largely with minimal loss on the anomaly detection performance. Lastly, we conduct an in-depth analysis to show the strength and limitations of our proposed model, including parameter analysis, feature visualization, and run-time analysis.
Title: Measurement and Fairness
Authors: Abigail Z. Jacobs; Hanna Wallach
Affiliations: University of Michigan; Microsoft Research
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: We propose measurement modeling from the quantitative social sciences as a framework for understanding fairness in computational systems. Computational systems often involve unobservable theoretical constructs, such as socioeconomic status, teacher effectiveness, and risk of recidivism. Such constructs cannot be measured directly and must instead be inferred from measurements of observable properties (and other unobservable theoretical constructs) thought to be related to them---i.e., operationalized via a measurement model. This process, which necessarily involves making assumptions, introduces the potential for mismatches between the theoretical understanding of the construct purported to be measured and its operationalization. We argue that many of the harms discussed in the literature on fairness in computational systems are direct results of such mismatches. We show how some of these harms could have been anticipated and, in some cases, mitigated if viewed through the lens of measurement modeling. To do this, we contribute fairness-oriented conceptualizations of construct reliability and construct validity that unite traditions from political science, education, and psychology and provide a set of tools for making explicit and testing assumptions about constructs and their operationalizations. We then turn to fairness itself, an essentially contested construct that has different theoretical understandings in different contexts. We argue that this contestedness underlies recent debates about fairness definitions: although these debates appear to be about different operationalizations, they are, in fact, debates about different theoretical understandings of fairness. We show how measurement modeling can provide a framework for getting to the core of these debates.
Title: Re-imagining Algorithmic Fairness in India and Beyond
Authors: Nithya Sambasivan; Erin Arnesen; Ben Hutchinson; Tulsee Doshi; Vinodkumar Prabhakaran
Affiliations: Google; Google; Google; Google; Google
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Conventional algorithmic fairness is West-centric, as seen in its subgroups, values, and methods. In this paper, we de-center algorithmic fairness and analyse AI power in India. Based on 36 qualitative interviews and a discourse analysis of algorithmic deployments in India, we find that several assumptions of algorithmic fairness are challenged. We find that in India, data is not always reliable due to socio-economic factors, ML makers appear to follow double standards, and AI evokes unquestioning aspiration. We contend that localising model fairness alone can be window dressing in India, where the distance between models and oppressed communities is large. Instead, we re-imagine algorithmic fairness in India and provide a roadmap to re-contextualise data and models, empower oppressed communities, and enable Fair-ML ecosystems.
Title: "This Whole Thing Smacks of Gender": Algorithmic Exclusion in Bioimpedance-based Body Composition Analysis
Authors: Kendra Albert; Maggie Delano
Affiliations: Harvard University; Swarthmore College
Year: 2021
Domain: Health & Clinical AI

Abstract: Smart weight scales offer bioimpedance-based body composition analysis as a supplement to pure body weight measurement. Companies such as Withings and Fitbit tout composition analysis as providing self-knowledge and the ability to make more informed decisions. However, these aspirational statements elide the reality that these numbers are a product of proprietary regression equations that require a binary sex/gender as their input. Our paper combines transgender studies-influenced personal narrative with an analysis of the scientific basis of bioimpedance technology used as part of the Withings smart scale. Attempting to include nonbinary people reveals that bioelectrical impedance analysis has always rested on physiologically shaky ground. White nonbinary people are merely the tip of the iceberg of those who may find that their smart scale is not so intelligent when it comes to their bodies. Using body composition analysis as an example, we explore how the problem of trans and nonbinary inclusion in personal health tech goes beyond the issues of adding a third "gender" box or slapping a rainbow flag on the packaging. We also provide recommendations as to how to approach creating more inclusive technologies even while still relying on exclusionary data.
Title: Censorship of Online Encyclopedias: Implications for NLP Models
Authors: Eddie Yang; Margaret E. Roberts
Affiliations: University of California San Diego; University of California San Diego
Year: 2021
Domain: LLMs & NLP

Abstract: While artificial intelligence provides the backbone for many tools people use around the world, recent work has brought to attention that the algorithms powering AI are not free of politics, stereotypes, and bias. While most work in this area has focused on the ways in which AI can exacerbate existing inequalities and discrimination, very little work has studied how governments actively shape training data. We describe how censorship has affected the development of Wikipedia corpuses, text data which are regularly used for pre-trained inputs into NLP algorithms. We show that word embeddings trained on Baidu Baike, an online Chinese encyclopedia, have very different associations between adjectives and a range of concepts about democracy, freedom, collective action, equality, and people and historical events in China than its regularly blocked but uncensored counterpart - Chinese language Wikipedia. We examine the implications of these discrepancies by studying their use in downstream AI applications. Our paper shows how government repression, censorship, and self-censorship may impact training data and the applications that draw from them.
Title: When the Umpire is also a Player: Bias in Private Label Product Recommendations on E-commerce Marketplaces
Authors: Abhisek Dash; Abhijnan Chakraborty; Saptarshi Ghosh; Animesh Mukherjee; Krishna P. Gummadi
Affiliations: Indian Institute of Technology Kharagpur; Indian Institute of Technology Delhi; Indian Institute of Technology Kharagpur; Indian Institute of Technology Kharagpur; Max Planck Institute for Software Systems
Year: 2021
Domain: Recommender Systems

Abstract: Algorithmic recommendations mediate interactions between millions of customers and products (in turn, their producers and sellers) on large e-commerce marketplaces like Amazon. In recent years, the producers and sellers have raised concerns about the fairness of black-box recommendation algorithms deployed on these marketplaces. Many complaints are centered around marketplaces biasing the algorithms to preferentially favor their own `private label' products over competitors. These concerns are exacerbated as marketplaces increasingly de-emphasize or replace `organic' recommendations with ad-driven `sponsored' recommendations, which include their own private labels. While these concerns have been covered in popular press and have spawned regulatory investigations, to our knowledge, there has not been any public audit of these marketplace algorithms. In this study, we bridge this gap by performing an end-to-end systematic audit of related item recommendations on Amazon. We propose a network-centric framework to quantify and compare the biases across organic and sponsored related item recommendations. Along a number of our proposed bias measures, we find that the sponsored recommendations are significantly more biased toward Amazon private label products compared to organic recommendations. While our findings are primarily interesting to producers and sellers on Amazon, our proposed bias measures are generally useful for measuring link formation bias in any social or content networks.
Title: Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Odds
Authors: Alan Mishler; Edward H. Kennedy; Alexandra Chouldechova
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: In domains such as criminal justice, medicine, and social welfare, decision makers increasingly have access to algorithmic Risk Assessment Instruments (RAIs). RAIs estimate the risk of an adverse outcome such as recidivism or child neglect, potentially informing high-stakes decisions such as whether to release a defendant on bail or initiate a child welfare investigation. It is important to ensure that RAIs are fair, so that the benefits and harms of such decisions are equitably distributed.
Title: Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately
Authors: Fereshte Khani; Percy Liang
Affiliations: Stanford University; Stanford University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Spurious features interfere with the goal of obtaining robust models that perform well across many groups within the population. A natural remedy is to remove such features from the model. However, in this work, we show that removing spurious features can surprisingly decrease accuracy due to the inductive biases of overparameterized models. In noiseless overparameterized linear regression, we completely characterize how the removal of spurious features affects accuracy across different groups (more generally, test distributions). In addition, we show that removal of spurious features can decrease the accuracy even on balanced datasets (where each target co-occurs equally with each spurious feature); and it can inadvertently make the model more susceptible to other spurious features. Finally, we show that robust self-training produces models that no longer depend on spurious features without affecting their overall accuracy. The empirical results on the Toxic-Comment-Detection and CelebA datasets show that our results hold in non-linear models.
Title: The Effect of Differential Victim Crime Reporting on Predictive Policing Systems
Authors: Nil-Jana Akpinar; Maria De-Arteaga; Alexandra Chouldechova
Affiliations: Carnegie Mellon University; University of Texas at Austin; Carnegie Mellon University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Police departments around the world have been experimenting with forms of place-based data-driven proactive policing for over two decades. Modern incarnations of such systems are commonly known as hot spot predictive policing. These systems predict where future crime is likely to concentrate such that police can allocate patrols to these areas and deter crime before it occurs. Previous research on fairness in predictive policing has concentrated on the feedback loops which occur when models are trained on discovered crime data, but has limited implications for models trained on victim crime reporting data. We demonstrate how differential victim crime reporting rates across geographical areas can lead to outcome disparities in common crime hot spot prediction models. Our analysis is based on a simulation1 patterned after district-level victimization and crime reporting survey data for Bogotá, Colombia. Our results suggest that differential crime reporting rates can lead to a displacement of predicted hotspots from high crime but low reporting areas to high or medium crime and high reporting areas. This may lead to misallocations both in the form of over-policing and under-policing.
Title: Detecting Discriminatory Risk through Data Annotation Based on Bayesian Inferences
Authors: Elena Beretta; Antonio Vetrò; Bruno Lepri; Juan Carlos De Martin
Affiliations: Politecnico di Torino; Politecnico di Torino; Fondazione Bruno Kessler; Politecnico di Torino
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Thanks to the increasing growth of computational power and data availability, the research in machine learning has advanced with tremendous rapidity. Nowadays, the majority of automatic decision making systems are based on data. However, it is well known that machine learning systems can present problematic results if they are built on partial or incomplete data. In fact, in recent years several studies have found a convergence of issues related to the ethics and transparency of these systems in the process of data collection and how they are recorded. Although the process of rigorous data collection and analysis is fundamental in the model design, this step is still largely overlooked by the machine learning community. For this reason, we propose a method of data annotation based on Bayesian statistical inference that aims to warn about the risk of discriminatory results of a given data set. In particular, our method aims to deepen knowledge and promote awareness about the sampling practices employed to create the training set, highlighting that the probability of success or failure conditioned to a minority membership is given by the structure of the data available. We empirically test our system on three datasets commonly accessed by the machine learning community and we investigate the risk of racial discrimination.
Title: Avoiding Disparity Amplification under Different Worldviews
Authors: Samuel Yeom; Michael Carl Tschantz
Affiliations: Carnegie Mellon University; International Computer Science Institute
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: We mathematically compare four competing definitions of group-level nondiscrimination: demographic parity, equalized odds, predictive parity, and calibration. Using the theoretical framework of Friedler et al., we study the properties of each definition under various worldviews, which are assumptions about how, if at all, the observed data is biased. We argue that different worldviews call for different definitions of fairness, and we specify the worldviews that, when combined with the desire to avoid a criterion for discrimination that we call disparity amplification, motivate demographic parity and equalized odds. We also argue that predictive parity and calibration are insufficient for avoiding disparity amplification because predictive parity allows an arbitrarily large inter-group disparity and calibration is not robust to post-processing. Finally, we define a worldview that is more realistic than the previously considered ones, and we introduce a new notion of fairness that corresponds to this worldview.
Title: The Effect of the Rooney Rule on Implicit Bias in the Long Term
Authors: L. Elisa Celis; C. P. Hays; Anay Mehrotra; Nisheeth K. Vishnoi
Affiliations: Yale University; Yale University; Yale University; Yale University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: The Rooney Rule, originally proposed to counter implicit bias in hiring, has been implemented in the private and public sector in various settings. This rule requires that a decision-maker include at least one candidate from an underrepresented group in their shortlist of candidates. Recently, [42] proposed a mathematical model of implicit bias and studied the effectiveness of the Rooney Rule when applied to a single selection decision. However, selection decisions often occur repeatedly over time; e.g., a software firm is continuously hiring employees or a university makes admissions decisions every year. Further, it has been observed that, given consistent counterstereotypical feedback, implicit biases against underrepresented candidates can change.
Title: Can You Fake It Until You Make It? Impacts of Differentially Private GAN Outputs in Clinical Settings
Authors: Victoria Cheng; Vinith M. Suriyakumar; Natalie Dullerud; Shalmali Joshi; Marzyeh Ghassemi
Affiliations: Vector Institute; University of Toronto; University of Toronto; Vector Institute; University of Toronto
Year: 2021
Domain: Health & Clinical AI

Abstract: The recent adoption of machine learning models in high-risk settings such as medicine has increased demand for developments in privacy and fairness. Rebalancing skewed datasets using synthetic data created by generative adversarial networks (GANs) has shown potential to mitigate disparate impact on minoritized subgroups. However, such generative models are subject to privacy attacks that can expose sensitive data from the training dataset. Differential privacy (DP) is the current leading solution for privacy-preserving machine learning. Differentially private GANs (DP GANs) are often considered a potential solution for improving model fairness while maintaining privacy of sensitive training data. We investigate the impact of using synthetic images from DP GANs on downstream classification model utility and fairness. We demonstrate that existing DP GANs cannot simultaneously maintain model utility, privacy, and fairness. The images generated from GAN models trained with DP exhibit extreme decreases in image quality and utility which leads to poor downstream classification model performance. Our evaluation highlights the friction between privacy, fairness, and utility and how this directly translates into real loss of performance and representation in common machine learning settings. Our results show that additional work improving the utility and fairness of DP generative models is required before they can be utilized as a potential solution to privacy and fairness issues stemming from lack of diversity in the training dataset.
Title: Leveraging Administrative Data for Bias Audits: A Case Study in Child Welfare Decision Making
Authors: Amanda Coston; Neel Guha; Derek Ouyang; Lisa Lu; Alexandra Chouldechova; Daniel E. Ho
Affiliations: Carnegie Mellon University; Stanford University; Stanford University; Stanford University; Carnegie Mellon University; Stanford University
Year: 2021
Domain: General Fairness & Bias Mitigation

Abstract: Anonymized smartphone-based mobility data has been widely adopted in devising and evaluating COVID-19 response strategies such as the targeting of public health resources. Yet little attention has been paid to measurement validity and demographic bias, due in part to the lack of documentation about which users are represented as well as the challenge of obtaining ground truth data on unique visits and demographics. We illustrate how linking large-scale administrative data can enable auditing mobility data for bias in the absence of demographic information and ground truth labels. More precisely, we show that linking voter roll data---containing individual-level voter turnout for specific voting locations along with race and age---can facilitate the construction of rigorous bias and reliability tests. Using data from North Carolina's 2018 general election, these tests illuminate a sampling bias that is particularly noteworthy in the pandemic context: older and non-white voters are less likely to be captured by mobility data. We show that allocating public health resources based on such mobility data could disproportionately harm high-risk elderly and minority groups.