AestheticNet: Reducing bias in facial data sets under ethical considerations

1 MOTIVATION . In 2016 Beauty.AI , a Hong-Kong based technology company , hosted the first international beauty contest judged by artificial intelligence ( beauty.ai , 2016 ) but the results were heavily biased , for example , against dark skin ( Levin , 2016 ) subjects . “ Machine learning models are prone to biased decisions , due to biases in data-sets ” ( Sharma et al. , 2020 ) . Biased training data potentially leads to discriminatory models , as the datasets are created by humans or derived from human activities in the past , for example hiring algorithms ( Bogen , 2019 ) . The reason for racist and discriminatory tendencies must be identified . As the learning algorithms become more complex , understanding why the decisions are made , or even how , prove to be nearly impossible ( Bostrom & Yudkowsky , 2018 ) . Therefore , the development of non-biased and fair training data and AI algorithms ( defined by the European Commission High-Level Expert Group on Artificial Intelligence ( European Commission High-Level Expert Group on Artificial Intelligence [ AI HLEG ] , 2019 ) ) is a new and increasingly complex challenge for scientists around the world ( Bellamy et al. , 2018 ) . The specific field of aesthetic judgement is especially vulnerable to being biased , as aesthetic judgement itself is already a subjective rating ( Richmond , 2017 ) . The purpose of facial beauty prediction ( FBP ) research is to classify images mimicking subjective human judgements . Investigations related to machine perception in a ground-truth free setting show that the data source depends on the measurement of human perception ( Prijatelj et al. , 2020 ) . Therefore , artificial networks need a process to determine labels of the average person ’ s judgement . Our data analysis has already proven that people consider their own ethnicity to be more attractive than others ( Gerlach et al. , 2020 ) , this is the major bias in our experiments and within our dataset . With this tendency , it becomes difficult to generate input data to train a machine-learning algorithm , which assesses a person ’ s attractiveness without bias . This work not only helps to achieve moral enhancement through AI ( see appendix B.1 ) , but also helps eliminating social problems with this new technology ( see appendix B.2 ) . 2 STATE OF THE ART . While research on the estimation of images or portraits is not a new trend , it has gained increasing attention since the emergence of artificial intelligence ( Zhang & Kreiman , 2021 ) . Although , for many applications like autonomous driving , or image classification , AI undoubtedly is the best solution , applications that are affected by unconscious bias , like beauty prediction ( Dornaika et al. , 2020 ) , tend to reflect bias that is likely to be prevalent within given datasets . Especially , when people subjective preferences play a role , such as in attractiveness judgement ( Shank & DeSanti , 2018 ) or human resource evaluation ( Lloyd , 2018 ) , bias is almost certain to happen . Carrera ( 2020 ) conducted a piece of research on the implication of racism in image databases , that analysed the association of aggressiveness , kindness , beauty and ugliness with different images and found that the decisions of many people are affected by subconscious racism . Since researchers are aware of such effect , they found different ways to reduce subconscious bias in machine learning . Since the problem originates from the given databases , either the databases , or the training need to be changed . The possibilities to change the databases include adding data , also referred to as fair preprocessing ( Bellamy et al. , 2018 ) , either by selection or augmentation to insert underrepresented samples . While , deleting images is usually a bad idea , since it increases the chances of the network overfitting , it could theoretically be used to eliminate overrepresented images . On the other side , training can be altered by selecting only images , that do not increase the variance of each class currently used as training input . For example , variational autoencoders can be used to extract the features of the image , to later determine their variance , and only select images as input that do not increase the variance within given classes . Bellamy et al . ( 2018 ) also describe a third method , they called fair post-processing . Since their pipeline aimed to create debiased databases , the post-processing step is usually not applicable for most machine learning applications not creating databases . 3 BIASED AI . 3.1 BIAS FROM HUMAN INDICATIONS . First , we propose hypothesis 1 : The results of the evaluation of the attractiveness of female pictures in the Asia-Europe data set by annotators in China and Germany are implicitly biased . We use our latest data set , which includes a total of 12,034 images of people from different social and ethnical backgrounds , with a total of 5.4 million annotations . Chinese and German participants have rated the pictures in the data set . We then have a comparison result to prove whether hypothesis 1 is true and mark this evaluation result as a ground truth . Figure 1 and fig . 2 confirm the statement of the first hypothesis . Further more , in this process , the results of our research also shows that aesthetic bias is not only related to ethnic background , but also related to age which also has been proven by other researchers ( Gerlach et al. , 2020 ) , ( Akbari et al. , 2020 ) . 3.2 AI TAKES ON HUMAN BIAS . We propose hypothesis 2 : artificial intelligence will copy the human bias . We use convolutional neural networks ( CNN ) to predict facial aesthetic scores and introduce AestheticNet . Related Work . With the introduction of CNNs and large-scale image repositories , facial image and video tasks get more powerful ( Krizhevsky et al. , 2017 ; Zeiler & Fergus , 2013 ; Deng et al. , 2009 ) . Xie et al . ( Xie et al. , 2015a ) present the SCUT-FBP500 dataset , containing 500 Asian female subjects with attractiveness ratings . Since “ FBP is a multi-paradigm computation problem ” the successor SCUT-FBP5500 ( Liang et al. , 2018 ) is introduced in 2018 , including an increased database of 5500 frontal faces with multiple attributes : male/female , Asian/Caucasian , age , beauty score . Liang et al . ( 2018 ) have evaluated their database “ using different combinations of feature and predictor , and various deep learning methods ” on AlexNet ( Krizhevsky et al. , 2017 ) , ResNet-18 ( He et al. , 2015 ) and ResNeXt-50 and achieved the Pearson Correlation PC : 0.8777 ; mean average error MAE : 0.2518 ; root-mean-square error RMSE : 0.3325 as a benchmark . In summary it can be said that all deep CNN models are superior to the shallow predictor with hand-crafted geometric feature or appearance feature ( Liang et al. , 2018 ) . Benchmark Dataset . The SCUT-FBP 5500 data set is a small data set for deep learning tasks . Therefore , it is an even greater challenge to train soft features like aesthetic or beauty . In order to measure the accuracy of the network and to be comparable to recent experiments in facial beauty prediction , we calculate the Pearson correlation coefficient ( PC ) , mean absolute error ( MAE ) and root mean square error ( RMSE ) . AestheticNet predictor architecture . The VGG Face architecture ( Simonyan & Zisserman , 2015 ) is the basis of our AestheticNet . Inspired by an idea of the paper from Shelhamer et al . ( 2017 ) we then add modifications to the network by exploiting feature maps from the third and fourth convolution block . Since the size of the features maps differ from the size of the resulting feature vector , we implement an additional max pooling layer to achieve the wanted output . For the predictions of the network , we concatenate the softmax results into a single feature vector as shown in fig . 3 . Our proposed network achieves a Pearson correlation coefficient of 0.9601 , which indicates an almost linear correspondence between annotations and predictions . Our training results have a very high accuracy and outperform state-of-the-art results . The normalised mean square error is 3.896 % and the normalised root mean square error is 5.580 % . These are measurements of the average error of the predicted labels , which are used to evaluate the accuracy of the network . The results are normalised because there are different datasets with different score ranges . Reannotation of SCUT-FBP5500 dataset . Since 2013 , for our study of facial aesthetics , we conducted online surveys on multiple image datasets ( mentioned in table 2 ) where thousands of students and their relatives participated . With this process we have been able to gather enough data to train a convolutional neural network with the goal to improve facial beauty prediction . During training convolutional neural networks ( CNN ) on this data , we recognised a large bias in this data . This led us to evaluate the annotations from Chinese and German universities and take a closer look at the bias . Our null hypothesis was that there is no bias in dependency of the ethical group , the proof for the presence of bias was done by reductio ad absurdum . In null hypothesis significance testing , the p-value is the probability of obtaining test results at least as extreme as the results actually observed , under the assumption that the null hypothesis is correct ( Aschwanden , 2015 ) . The precise calculation of the p-value in this experiment is difficult because the factorials raise too high , to be reasonably computed on the thousands of labelled values . We calculate the p-value on 300 representative annotations which lead to a p-value of approximately 0.063 % , therefore it is safe to say the null hypothesis can be rejected and we do have an ethical bias . 4 TRAINING OF UNBIASED AI . In general , there are three main paths to reach the goal of unbiased predictions : fair pre-processing , fair in-processing and fair post-processing ( Bellamy et al. , 2018 ) . Within this paper , we present two approaches based on those paths to train an unbiased network with biased data , for FBP . The first approach relies on data pre-processing before training to introduce fairness , we call it “ balanced training ” . The second approach relies on a categorical cross entropy loss function , for the network to learn the bias and decrease it . Those processes are explained in the following sections . 4.1 DATASET AND GAN IMAGES . Machine learning has evolved in the past decades and stands out due to the fact that the knowledge in the system is not provided by experts . Facial beauty prediction ( FBP ) that is consistent with human perception , is a significant visual recognition problem and a much-studied subject in recent decades . Eisenthal et al . ( 2006 ) and ( Kagian et al. , 2008 ) were among the first to publish their research about automatic facial attractiveness predictors and supervised learning techniques , based on the extraction of feature landmarks on faces . We analysed the data that we gathered with our Analysis Toolbox and could measure a significant bias within the prediction of aesthetics through different ethnicities . Therefore , training a network with the goal to create unbiased results is still a challenge in deep learning tasks . In the following we will first describe our data set blend and the accompanying Analysis Toolbox and we explain how we used a GAN to create artificial portraits with European and Asian ethnicities . Starting in 2017 , we used the Asian-European-dataset SCUT-FBP ( Xie et al. , 2015b ; Liang et al. , 2018 ) to evaluate biased annotations from Chinese and German universities . The results proved the assumption that German students favour images of European women and vice versa Chinese students rate Asian portraits higher . Since the SCUT-FBP 5500 dataset is a small dataset for deep learning tasks , we use data augmentation methods to enlarge the sample size of the training set by generating GAN images with either Asian or European or mixed images as input and new synthesised images as output . This augmentation method proves superior to geometric transformations like cropping and rotating . All images are preprocessed , by normalisation methods to harmonise face pose , facial landmark positions and image size . For the purpose of a thorough analysis , we blend multiple datasets in the domain of facial aesthetics together . Our complete set of databases which is described in table 2 , consists of multiracial and multiethnic individuals . In total , this data set includes 12,034 portrait images from persons of different ethnicities with individual social backgrounds . These images are labelled and annotated in surveys over a period of 8 years with a total number of 5.4 million annotations . Additionally , recently we add the FairFace ( Kärkkäinen & Joo , 2019 ) database , which includes male and female portraits of seven different ethnic groups . The synthesised Eurasians images are artificially generated with StarGAN v2 ( Choi et al. , 2020 ) to determine the influence of the biased view of annotators on aesthetics of persons from different ethnicities . We used different customised input for the source and reference images to control the amount of ethnic admixture . Figure 4 shows one exemplary set of images for the Eurasians dataset . After annotating the dataset , the unconscious bias in the annotations can be uncovered . Figure 5 shows the biased average score of our networks on the SCUT-FBP dataset and the Eurasian dataset . Figure 6 illustrates the analysis on the distribution of aesthetic score and age for Asians , Europeans and three mixed-racial subgroups . The different group annotation points are displayed in different colours . We calculate the following metrics for each group cluster i : Horizontal dashed lines are average attractiveness values ai . Vertical dashed lines are average age values yi . As can be seen , the interval of ai has a small span , yet however the interval of yi has a significantly larger span . Each ai and yi values intersection point forms an per group attractiveness-age-factor AAFi = ai/yi . In a fair machine , these AAFi points would be closer together , as the yi span is small . This idea is further elaborated in section 4 .

Summary: In this paper, the authors study the problem of bias in facial beauty prediction problem. To this end, they first show that there do exist bias in an existing dataset. Then, they show that deep networks trained with such a biased dataset do capture and reflect the bias. Finally, they propose two solutions for addressing such bias.