Identifying Bias in AI using Simulation

1 INTRODUCTION . Machine learned classifiers are becoming increasingly prevalent and important . Many systems contain components that leverage trained models for detecting or classifying patterns in data . Whether decisions are made entirely , or partially based on the output of these models , and regardless of the number of other components in the system , it is vital that their characteristics are well understood . However , the reality is that with many complex systems , such as deep neural networks , many of the “ unknowns ” are unknown and need to identified ( Lakkaraju et al. , 2016 ; 2017 ) . Imagine a model being deployed in law enforcement for facial recognition , such a system could encounter almost infinite scenarios ; which of these scenarios will the classifier have a blind-spot for ? We propose an approach for helping diagnose biases within such a system more efficiently . Many learned models exhibit bias as training datasets are limited in size and diversity ( Torralba & Efros , 2011 ; Tommasi et al. , 2017 ) , or they reflect inherent human-biases ( Caliskan et al. , 2017 ) . It is difficult for researchers to collect vast datasets that feature equal representations of every key property . Collecting large corpora of training examples requires time , is often costly and is logistically challenging . Let us take facial analysis as an exemplar problem for computer vision systems . There are numerous companies that provide services of face detection and tracking , face recognition , facial attribute detection , and facial expression/action unit recognition ( e.g. , Microsoft ( msf , 2018 ) , Google ( goo , 2018 ) , Affectiva ( McDuff et al. , 2016 ; aff , 2018 ) ) . However , studies have revealed systematic biases in results of these systems ( Buolamwini , 2017 ; Buolamwini & Gebru , 2018 ) , with the error rate up to seven times larger on women than men . Such biases in performance are very problematic when deploying these algorithms in the real-world . Other studies have found that face recognition systems misidentify [ color , gender ( women ) , and age ( younger ) ] at higher error rates ( Klare et al. , 2012 ) . Reduced performance of a classifier on minority groups can lead to both greater numbers of false positives ( in a law enforcement domain this would lead to more frequent targeting ) or greater numbers of false negatives ( in a medical domain this would lead to missed diagnoses ) . Taking face detection as a specific example of a task that all the services mentioned above rely upon , demographic and environmental factors ( e.g. , gender , skin type , ethnicity , illumination ) all influence the appearance of the face . Say we collected a large dataset of positive and negative examples of faces within images . Regardless of how large the dataset is , these examples may not be evenly distributed across each demographic group . This might mean that the resulting classifier performs much less accurately on African-American people , because the training data featured few examples . A longitudinal study of police departments revealed that African-American individuals were more likely to be subject to face recognition searches than others ( Garvie , 2016 ) . To further complicate matters , even if one were to collect a dataset that balances the number of people with different skin types , it is highly unlikely that these examples would have similar characteristics across all other dimensions , such as lighting , position , pose , etc . Therefore , even the best efforts to collect balanced datasets are still likely to be flawed . The challenge then is to find a way of successfully characterizing the performance of the resulting classifier across all these dimensions . The concept of fairness through awareness was presented by Dwork et al . ( 2012 ) , the principle being that in order to combat bias we need to be aware of the biases and why they occur . This idea has partly inspired proposals of standards for characterizing training datasets that inform consumers of their properties ( Gebru et al. , 2018 ; Holland et al. , 2018 ) . Such standards would be very valuable . However , while transparency is very important , it will not solve the fundamental problem of how to address the biases caused by poor representation . Nor will it help identify biases that might still occur even with models trained using carefully curated datasets . Attempts have been made to improve facial attribute detection by including gender and racial diversity . In one example , by Ryu et al . ( 2017 ) , results were improved by scraping images from the web and learning facial representations from a held-out dataset with a uniform distribution across race and gender intersections . However , a drawback of this approach is that even images available from vast sources , such as Internet image search , may not be evenly balanced across all attributes and properties and the data collection and cleaning is still very time consuming . To address the problem of diagnosing bias in real world datasets we propose the use of high-fidelity simulations ( Shah et al. , 2018 ) to interrogate models . Simulations allow for large volumes of diverse training examples to be generated and different parameter combinations to be systematically tested , something that is challenging with “ found ” data scrapped from the web or even curated datasets . Simulated data can be created in different ways . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) are becoming increasingly popular for synthesizing data ( Shrivastava et al. , 2017 ) . For example , GANs could be used to synthesize images of faces at different ages ( Yang et al. , 2017 ) . However , GANs are inherently statistical models and are likely to contain some of the biases that the data used to train them contain . A GAN model trained with only a few examples of faces with darker skin tones will likely fail to produce a diverse set of high quality synthesized images with this attribute . Parameterized graphics models are an alternative for training and testing vision models ( Veeravasarapu et al. , 2015a ; b ; 2016 ; Vazquez et al. , 2014 ) . Specifically , it has been proposed that graphics models be used for performance evaluation ( Haralick , 1992 ) . As an example , this approach has been used for models for pedestrian detection ( Vazquez et al. , 2014 ) . To the best of our knowledge graphics models have not been employed for detecting demographic biases within vision models . We believe that demographic biases in machine learned systems is significant enough a problem to warrant further attention . The contributions of this paper are to : ( 1 ) present a simulated model for generating synthetic facial data , ( 2 ) show how simulated data can be used to identify the limitations of existing face detection algorithms , and ( 3 ) to present a sample efficient approach that reduces the number of simulations required . The simulated model used in this paper is made available . 2 RELATED WORK . 2.1 BIAS IN MACHINE LEARNING . With more adoption of machine learned algorithms in real-world applications there is growing concern in society that these systems could discriminate between people unfairly12 . In cases where these systems provide a reliable signal that the output has a low confidence , these can be described as known unknowns . However , many learned models are being deployed while their performance in specific scenarios is still unknown , or their prediction confidence in scenarios is not well characterized . These can be described as unknown unknowns . Lakkaraju et al . ( Lakkaraju et al. , 2016 ; 2017 ) published the first example of a method to help address the discovery of unknowns in predictive models . First , the search-space is partitioned into groups which can be given interpretable descriptions . Second , an explore-exploit strategy is used to navigate through these groups systematically based on the feedback from an oracle ( e.g. , a human 1https : //www.newscientist.com/article/2166207-discriminating-algorithms-5-times-ai-showed-prejudice/ 2https : //www.technologyreview.com/s/608986/forget-killer-robotsbias-is-the-real-ai-danger/ labeler ) . Bansal and Weld proposed a new class of utility models that rewarded how well the discovered unknown unknowns help explain a sample distribution of expected queries ( Bansal & Weld , 2018 ) . We also employ an explore-exploit strategy in our work , but rather than rely on an oracle we leverage synthetic data . Biases often result from unknowns within a system . Biases have been identified in real-world automated systems applied in domains from medicine ( Zheng et al. , 2017 ) to criminal justice ( Angwin et al. , 2016 ) . While the exact nature of the problem is debated ( Flores et al. , 2016 ) it is clear that further research is needed to address bias in machine learned systems and actionable remedies to help practitioners in their day to day work are necessary . Numerous papers have highlighted biases within data typically used for machine learning ( Caliskan et al. , 2017 ; Torralba & Efros , 2011 ; Tommasi et al. , 2017 ) and machine learned classifiers ( Buolamwini & Gebru , 2018 ) . Biases in datasets can be caused in several ways . Torralba and Efros ( 2011 ) identified selection bias , capture bias , and negative set bias as problems . “ Selection bias ” is related to the tendency for certain types , or classes , of images to be included in datasets in the first place . “ Capture bias ” is related to how the data are acquired and may be influenced by the collectors ’ preferences ( e.g . in images of certain people , say celebrities , or points of view , lighting conditions , etc. ) . Taking faces as an example , these biases typically manifest as a greater proportion of images including white males , frontal head poses and smiles or neutral expressions . “ Negative set bias ” is related to the examples in the dataset of the “ rest of the world ” . As an example , if the negative examples included in a dataset of faces are skewed , the model will learn a warped representation of what is not a face . However , even if one were to make every effort to address these three types of bias , the problem may still exist that artifacts created by humans contain the biases that human have without our explicit knowledge ( Caliskan et al. , 2017 ) . In our discussion , we will call this “ human bias ” . With increasing frequency , data hungry machine learning classifiers are being trained on large-scale corpora collected from Internet sources ( Deng et al. , 2009 ; Fabian Benitez-Quiroz et al. , 2016 ) ) and/or via crowdsourcing ( McDuff et al. , 2018 ) . Such data collection makes curation challenging as there are a vast number of samples to label and sort . The Internet is a data source that is likely to be subject to “ selection bias ” , “ capture bias ” , “ negative set bias ” and “ human bias ” . Furthermore , Tommasi et al . ( 2017 ) argue that more powerful feature descriptors ( such as those generated by a deep convolutional neural network versus a simple hand crafted approach ) may actually exacerbate the problem of bias and accentuate biases in the resulting classifier . Even in the best case scenario , feature representations alone can not intrinsically remove the negative bias problem . Thus there is still a need for approaches that help address the issues of bias within machine learning , and this problem is becoming more acute . To this end , we propose a practical approach to use high-fidelity simulations to diagnoses biases efficiently in machine vision classifiers .

Summary: The paper uses a proof-of-concept Bayesian parameter search-based simulation in virtual environment to probe biases of an already trained model towards specific categories that may have been sparsely represented in the training set. Understanding bias in trained models, especially in models involving end-to-end deep neural networks learners, is of high importance in machine learning. More specifically, probing the source of unintentional bias introduced as a result of skewed distribution in the training set and dissecting the biased performance is important for many applications such as surveillance, criminal profiling, medical diagnosis and predicting creditworthiness of a person.