cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Authors: L. Elisa Celis; Lingxiao Huang; Vijay Keswani; Nisheeth K. Vishnoi
Affiliations: Yale University; ; ; Yale University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Developing classification algorithms that are fair with respect to sensitive attributes of the data has become an important problem due to the growing deployment of classification algorithms in various social contexts. Several recent works have focused on fairness with respect to a specific metric, modeled the corresponding fair classification problem as a constrained optimization problem, and developed tailored algorithms to solve them. Despite this, there still remain important metrics for which we do not have fair classifiers and many of the aforementioned algorithms do not come with theoretical guarantees; perhaps because the resulting optimization problem is non-convex. The main contribution of this paper is a new meta-algorithm for classification that takes as input a large class of fairness constraints, with respect to multiple non-disjoint sensitive attributes, and which comes with provable guarantees. This is achieved by first developing a meta-algorithm for a large family of classification problems with convex constraints, and then showing that classification problems with general types of fairness constraints can be reduced to those in this family. We present empirical results that show that our algorithm can achieve near-perfect fairness with respect to various fairness metrics, and that the loss in accuracy due to the imposed fairness constraints is often small. Overall, this work unifies several prior works on fair classification, presents a practical algorithm with theoretical guarantees, and can handle fairness metrics that were previously not possible.
Title: A comparative study of fairness-enhancing interventions in machine learning
Authors: Sorelle A. Friedler; Carlos Scheidegger; Suresh Venkatasubramanian; Sonam Choudhary; Evan P. Hamilton; Derek Roth
Affiliations: Haverford College; University of Arizona; University of Utah; University of Utah; Haverford College; Haverford College
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Computers are increasingly used to make decisions that have significant impact on people's lives. Often, these predictions can affect different population subgroups disproportionately. As a result, the issue of fairness has received much recent interest, and a number of fairness-enhanced classifiers have appeared in the literature. This paper seeks to study the following questions: how do these different techniques fundamentally compare to one another, and what accounts for the differences? Specifically, we seek to bring attention to many under-appreciated aspects of such fairness-enhancing interventions that require investigation for these algorithms to receive broad adoption.
Title: Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
Authors: Jiahao Chen; Nathan Kallus; Xiaojie Mao; Geoffry Svacha; Madeleine Udell
Affiliations: ; Cornell University; Cornell University; ; Cornell University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Assessing the fairness of a decision making system with respect to a protected class, such as gender or race, is challenging when class membership labels are unavailable. Probabilistic models for predicting the protected class based on observable proxies, such as surname and geolocation for race, are sometimes used to impute these missing labels for compliance assessments. Empirically, these methods are observed to exaggerate disparities, but the reason why is unknown. In this paper, we decompose the biases in estimating outcome disparity via threshold-based imputation into multiple interpretable bias sources, allowing us to explain when over- or underestimation occurs. We also propose an alternative weighted estimator that uses soft classification, and show that its bias arises simply from the conditional covariance of the outcome with the true class membership. Finally, we illustrate our results with numerical simulations and a public dataset of mortgage applications, using geolocation as a proxy for race. We confirm that the bias of threshold-based imputation is generally upward, but its magnitude varies strongly with the threshold chosen. Our new weighted estimator tends to have a negative bias that is much simpler to analyze and reason about.
Title: An Empirical Study of Rich Subgroup Fairness for Machine Learning
Authors: Michael Kearns; Seth Neel; Aaron Roth; Zhiwei Steven Wu
Affiliations: University of Pennsylvania; University of Pennsylvania; University of Pennsylvania; University of Minnesota
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Kearns, Neel, Roth, and Wu [ICML 2018] recently proposed a notion of rich subgroup fairness intended to bridge the gap between statistical and individual notions of fairness. Rich subgroup fairness picks a statistical fairness constraint (say, equalizing false positive rates across protected groups), but then asks that this constraint hold over an exponentially or infinitely large collection of subgroups defined by a class of functions with bounded VC dimension. They give an algorithm guaranteed to learn subject to this constraint, under the condition that it has access to oracles for perfectly learning absent a fairness constraint. In this paper, we undertake an extensive empirical evaluation of the algorithm of Kearns et al. On four real datasets for which fairness is a concern, we investigate the basic convergence of the algorithm when instantiated with fast heuristics in place of learning oracles, measure the tradeoffs between fairness and accuracy, and compare this approach with the recent algorithm of Agarwal, Beygelzeimer, Dudik, Langford, and Wallach [ICML 2018], which implements weaker and more traditional marginal fairness constraints defined by individual protected attributes. We find that in general, the Kearns et al. algorithm converges quickly, large gains in fairness can be obtained with mild costs to accuracy, and that optimizing accuracy subject only to marginal fairness leads to classifiers with substantial subgroup unfairness. We also provide a number of analyses and visualizations of the dynamics and behavior of the Kearns et al. algorithm. Overall we find this algorithm to be effective on real data, and rich subgroup fairness to be a viable notion in practice.
Title: Equality of Voice: Towards Fair Representation in Crowdsourced Top-K Recommendations
Authors: Abhijnan Chakraborty; Gourab K Patro; Niloy Ganguly; Krishna P. Gummadi; Patrick Loiseau
Affiliations: MPI for Software Systems; IIT Kharagpur; IIT Kharagpur; MPI for Software Systems; Univ. Grenoble Alpes
Year: 2019
Domain: Recommender Systems

Abstract: To help their users to discover important items at a particular time, major\nwebsites like Twitter, Yelp, TripAdvisor or NYTimes provide Top-K\nrecommendations (e.g., 10 Trending Topics, Top 5 Hotels in Paris or 10 Most\nViewed News Stories), which rely on crowdsourced popularity signals to select\nthe items. However, different sections of a crowd may have different\npreferences, and there is a large silent majority who do not explicitly express\ntheir opinion. Also, the crowd often consists of actors like bots, spammers, or\npeople running orchestrated campaigns. Recommendation algorithms today largely\ndo not consider such nuances, hence are vulnerable to strategic manipulation by\nsmall but hyper-active user groups.\n To fairly aggregate the preferences of all users while recommending top-K\nitems, we borrow ideas from prior research on social choice theory, and\nidentify a voting mechanism called Single Transferable Vote (STV) as having\nmany of the fairness properties we desire in top-K item (s)elections. We\ndevelop an innovative mechanism to attribute preferences of silent majority\nwhich also make STV completely operational. We show the generalizability of our\napproach by implementing it on two different real-world datasets. Through\nextensive experimentation and comparison with state-of-the-art techniques, we\nshow that our proposed approach provides maximum user satisfaction, and cuts\ndown drastically on items disliked by most but hyper-actively promoted by a few\nusers.\n
Title: Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking
Authors: Mahmoudreza Babaei; Abhijnan Chakraborty; Juhi Kulshrestha; Elissa M. Redmiles; Meeyoung Cha; Krishna P. Gummadi
Affiliations: Max Planck Institute for Software Systems; Max Planck Institute for Software Systems; GESIS - Leibniz Institute for the Social Sciences; ; Korea Advanced Institute of Science and Technology; Max Planck Institute for Software Systems
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Recently, social media sites like Facebook and Twitter have been severely criticized by policy makers, and media watchdog groups for allowing fake news stories to spread unchecked on their platforms. In response, these sites are encouraging their users to report any news story they encounter on the site, which they perceive as fake. Stories that are reported as fake by a large number of users are prioritized for fact checking by (human) experts at fact checking organizations like Snopes and PolitiFact. Thus, social media sites today are relying on their users' perceptions of the truthfulness of news stories to select stories to fact check.
Title: Measuring the Biases that Matter: The Ethical and Causal Foundations for Measures of Fairness in Algorithms
Authors: Bruce Glymour; Jonathan Herington
Affiliations: Kansas State University; Kansas State University
Year: 2019
Domain: General Fairness & Bias Mitigation

Abstract: Measures of algorithmic bias can be roughly classified into four categories, distinguished by the conditional probabilistic dependencies to which they are sensitive. First, measures of "procedural bias" diagnose bias when the score returned by an algorithm is probabilistically dependent on a sensitive class variable (e.g. race or sex). Second, measures of "outcome bias" capture probabilistic dependence between class variables and the outcome for each subject (e.g. parole granted or loan denied). Third, measures of "behavior-relative error bias" capture probabilistic dependence between class variables and the algorithmic score, conditional on target behaviors (e.g. recidivism or loan default). Fourth, measures of "score-relative error bias" capture probabilistic dependence between class variables and behavior, conditional on score. Several recent discussions have demonstrated a tradeoff between these different measures of algorithmic bias, and at least one recent paper has suggested conditions under which tradeoffs may be minimized.
Title: Dissecting Racial Bias in an Algorithm that Guides Health Decisions for 70 Million People
Authors: Ziad Obermeyer; Sendhil Mullainathan
Affiliations: Berkeley College; University of Chicago
Year: 2019
Domain: Health & Clinical AI

Abstract: A single algorithm drives an important health care decision for over 70 million people in the US. When health systems anticipate that a patient will have especially complex and intensive future health care needs, she is enrolled in a 'care management' program, which provides considerable additional resources: greater attention from trained providers and help with coordination of her care.
Title: Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
Authors: Inioluwa Deborah Raji; Andrew Smart; Rebecca N. White; Margaret Mitchell; Timnit Gebru; Ben Hutchinson; Jamila Smith-Loud; Daniel Theron; Parker Barnes
Affiliations: Partners In Health; Google; Google; Google; Google; Google; Google; Google; Google
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source. In this paper, we introduce a framework for algorithmic auditing that supports artificial intelligence system development end-to-end, to be applied throughout the internal organization development lifecycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization's values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity.
Title: Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination
Authors: Nathan Kallus; Xiaojie Mao; Angela Zhou
Affiliations: Cornell University; Cornell University; Cornell University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: The increasing impact of algorithmic decisions on people’s lives compels us to scrutinize their fairness and, in particular, the disparate impacts that ostensibly color-blind algorithms can have on different groups. Examples include credit decisioning, hiring, advertising, criminal justice, personalized medicine, and targeted policy making, where in some cases legislative or regulatory frameworks for fairness exist and define specific protected classes. In this paper we study a fundamental challenge to assessing disparate impacts in practice: protected class membership is often not observed in the data. This is particularly a problem in lending and healthcare. We consider the use of an auxiliary data set, such as the U.S. census, to construct models that predict the protected class from proxy variables, such as surname and geolocation. We show that even with such data, a variety of common disparity measures are generally unidentifiable, providing a new perspective on the documented biases of popular proxy-based methods. We provide exact characterizations of the tightest possible set of all possible true disparities that are consistent with the data (and possibly additional assumptions). We further provide optimization-based algorithms for computing and visualizing these sets and statistical tools to assess sampling uncertainty. Together, these enable reliable and robust assessments of disparities—an important tool when disparity assessment can have far-reaching policy implications. We demonstrate this in two case studies with real data: mortgage lending and personalized medicine dosing. This paper was accepted by Hamid Nazerzadeh, Management Science Special Section on Data-Driven Prescriptive Analytics.
Title: FlipTest: Fairness Testing via Optimal Transport
Authors: Emily Black; Samuel Yeom; Matt Fredrikson
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: We present FlipTest, a black-box technique for uncovering discrimination in classifiers. FlipTest is motivated by the intuitive question: had an individual been of a different protected status, would the model have treated them differently? Rather than relying on causal information to answer this question, FlipTest leverages optimal transport to match individuals in different protected groups, creating similar pairs of in-distribution samples. We show how to use these instances to detect discrimination by constructing a flipset: the set of individuals whose classifier output changes post-translation, which corresponds to the set of people who may be harmed because of their group membership. To shed light on why the model treats a given subgroup differently, FlipTest produces a transparency report: a ranking of features that are most associated with the model's behavior on the flipset. Evaluating the approach on three case studies, we show that this provides a computationally inexpensive way to identify subgroups that may be harmed by model discrimination, including in cases where the model satisfies group fairness criteria.
Title: Towards a Critical Race Methodology in Algorithmic Fairness
Authors: Alex Hanna; Emily Denton; Andrew Smart; Jamila Smith-Loud
Affiliations: Google; Google; Google; Google
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: We examine the way race and racial categories are adopted in algorithmic fairness frameworks. Current methodologies fail to adequately account for the socially constructed nature of race, instead adopting a conceptualization of race as a fixed attribute. Treating race as an attribute, rather than a structural, institutional, and relational phenomenon, can serve to minimize the structural aspects of algorithmic unfairness. In this work, we focus on the history of racial categories and turn to critical race theory and sociological work on race and ethnicity to ground conceptualizations of race for fairness research, drawing on lessons from public health, biomedical research, and social survey research. We argue that algorithmic fairness researchers need to take into account the multidimensionality of race, take seriously the processes of conceptualizing and operationalizing race, focus on social processes which produce racial inequality, and consider perspectives of those most affected by sociotechnical systems.
Title: Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
Authors: Manish Raghavan; Solon Barocas; Jon Kleinberg; Karen Levy
Affiliations: Cornell University; Cornell University; Cornell University; Cornell University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: There has been rapidly growing interest in the use of algorithms in hiring, especially as a means to address or mitigate bias. Yet, to date, little is known about how these methods are used in practice. How are algorithmic assessments built, validated, and examined for bias? In this work, we document and analyze the claims and practices of companies offering algorithms for employment assessment. In particular, we identify vendors of algorithmic pre-employment assessments (i.e., algorithms to screen candidates), document what they have disclosed about their development and validation procedures, and evaluate their practices, focusing particularly on efforts to detect and mitigate bias. Our analysis considers both technical and legal perspectives. Technically, we consider the various choices vendors make regarding data collection and prediction targets, and explore the risks and trade-offs that these choices pose. We also discuss how algorithmic de-biasing techniques interface with, and create challenges for, antidiscrimination law.
Title: Reducing Sentiment Polarity for Demographic Attributes in Word Embeddings using Adversarial Learning
Authors: Chris Sweeney; Maryam Najafian
Affiliations: Massachusetts Institute of Technology; Massachusetts Institute of Technology
Year: 2020
Domain: LLMs & NLP

Abstract: The use of word embedding models in sentiment analysis has gained a lot of traction in the Natural Language Processing (NLP) community. However, many inherently neutral word vectors describing demographic identity have unintended implicit correlations with negative or positive sentiment, resulting in unfair downstream machine learning algorithms. We leverage adversarial learning to decorrelate demographic identity term word vectors with positive or negative sentiment, and re-embed them into the word embeddings. We show that our method effectively minimizes unfair positive/negative sentiment polarity while retaining the semantic accuracy of the word embeddings. Furthermore, we show that our method effectively reduces unfairness in downstream sentiment regression and can be extended to reduce unfairness in toxicity classification tasks.
Title: Bias in Word Embeddings
Authors: Orestis Papakyriakopoulos; Simon Hegelich; Juan Carlos Medina Serrano; Fabienne Marco
Affiliations: Technical University of Munich; Technical University of Munich; Technical University of Munich; Technical University of Munich
Year: 2020
Domain: LLMs & NLP

Abstract: Word embeddings are a widely used set of natural language processing techniques that map words to vectors of real numbers. These vectors are used to improve the quality of generative and predictive models. Recent studies demonstrate that word embeddings contain and amplify biases present in data, such as stereotypes and prejudice. In this study, we provide a complete overview of bias in word embeddings. We develop a new technique for bias detection for gendered languages and use it to compare bias in embeddings trained on Wikipedia and on political social media data. We investigate bias diffusion and prove that existing biases are transferred to further machine learning models. We test two techniques for bias mitigation and show that the generally proposed methodology for debiasing models at the embeddings level is insufficient. Finally, we employ biased word embeddings and illustrate that they can be used for the detection of similar biases in new data. Given that word embeddings are widely used by commercial companies, we discuss the challenges and required actions towards fair algorithmic implementations and applications.
Title: Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy
Authors: Kaiyu Yang; Klint Qinami; Li Fei-Fei; Jia Deng; Olga Russakovsky
Affiliations: Princeton University; Princeton University; Stanford University; Princeton University; Princeton University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: Computer vision technology is being used by many but remains representative of only a few. People have reported misbehavior of computer vision models, including offensive prediction results and lower performance for underrepresented groups. Current computer vision models are typically developed using datasets consisting of manually annotated images or videos; the data and label distributions in these datasets are critical to the models' behavior. In this paper, we examine ImageNet, a large-scale ontology of images that has spurred the development of many modern computer vision methods. We consider three key factors within the "person" subtree of ImageNet that may lead to problematic behavior in downstream computer vision technology: (1) the stagnant concept vocabulary of WordNet, (2) the attempt at exhaustive illustration of all categories with images, and (3) the inequality of representation in the images within concepts. We seek to illuminate the root causes of these concerns and take the first steps to mitigate them constructively.