Impact of facial tattoos and paintings on face recognition systems

IET BiometricsEarly View CASE STUDYOpen Access Impact of facial tattoos and paintings on face recognition systems Mathias Ibsen, Corresponding Author mathias.ibsen@h-da.de orcid.org/0000-0002-7531-806X Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, Germany Correspondence Mathias Ibsen, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Haardtring 100, 64295 Darmstadt, Germany. Email: mathias.ibsen@h-da.deSearch for more papers by this authorChristian Rathgeb, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorThomas Fink, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorPawel Drozdowski, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorChristoph Busch, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this author Mathias Ibsen, Corresponding Author mathias.ibsen@h-da.de orcid.org/0000-0002-7531-806X Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, Germany Correspondence Mathias Ibsen, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Haardtring 100, 64295 Darmstadt, Germany. Email: mathias.ibsen@h-da.deSearch for more papers by this authorChristian Rathgeb, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorThomas Fink, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorPawel Drozdowski, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this authorChristoph Busch, Department of Computer Science, Biometrics and Internet Security Research Group, Hochschule Darmstadt, Darmstadt, GermanySearch for more papers by this author First published: 16 April 2021 https://doi.org/10.1049/bme2.12032AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract In recent years, face recognition technologies have shown impressive recognition performance, mainly due to recent developments in deep convolutional neural networks. Notwithstanding those improvements, several challenges which affect the performance of face recognition systems remain. The authors investigate the impact that facial tattoos and paintings have on current face recognition systems. To this end, they first collected an appropriate database containing image-pairs of individuals with and without facial tattoos or paintings. The assembled database was used to evaluate how facial tattoos and paintings affect the detection, quality estimation, as well as the feature extraction and comparison modules of a face recognition system. The impact on these modules was evaluated using state-of-the-art open-source and commercial systems. The obtained results show that facial tattoos and paintings affect all the tested modules, especially for images where a large area of the face is covered with tattoos or paintings. The authors' work is an initial case study and indicates a need to design algorithms which are robust to the visual changes caused by facial tattoos and paintings. 1 INTRODUCTION Face recognition is used in a range of security applications for automatically recognising individuals and has been an active research area for decades [1-3]. Towards implementing robust and reliable face recognition systems several factors have been identified which can negatively affect the performance of face recognition systems, for example changes in age, facial pose, and facial expression [3, 4]. Additionally, it has been shown that facial changes caused by different facial manipulations like plastic surgery, makeup, or facial retouching, might have a negative effect on the performance of face recognition systems [5]. It is therefore of interest whether other types of face manipulations, in particular, facial tattoos and facial paintings, have an impact on the biometric performance of face recognition systems. Buttle and East [6] conducted a study on the impact of facial tattoos on human perception; they showed a strong deterioration of humans' ability to recognise faces with traditional spiral-like tattoos. However, to the best of the authors' knowledge, there exists no work which investigates the impact of facial tattoos or paintings on automated face recognition systems. During the enrolment phase of a classical face recognition system, a reference image is captured by the capture device whereafter the face is detected, alignment is performed, and sample quality is estimated. Lastly, a biometric feature set useful for face recognition is extracted and stored as a reference template in an enrolment database. At the time of authentication, a probe face image is captured and processed in the same way as during enrolment; subsequently, it is compared against a reference template corresponding to a claimed identity (verification) or up to all stored reference templates (identification). Traditionally, handcrafted texture descriptors such as Local Binary Patterns (LBP) [7, 8], Histogram of Oriented Gradients (HOG) [9], and Gabor filters [10], have been used for extracting features useful for face recognition. The reader is referred to [1-3] for an overview of such systems. Current state-of-the-art face recognition systems, however, take advantage of the availability of large 2D face image databases and deep learning techniques to learn complex representations of faces. This paradigm shift has led to an impressive improvement of the biometric performance results of face recognition systems [11-14]. Figure 1 gives an overview of different types of facial manipulations. Recent research has shown that the efficacy of face recognition systems is affected by certain types of manipulations (e.g. morphing [15], retouching [5], and makeup [16]). Motivated by the need for deploying robust and reliable systems, the authors investigate the impact of tattoos and paintings on face recognition systems. A facial tattoo is a form of permanent facial manipulation induced by inserting ink into the dermis layer of skin on a person's face. The main intent of tattoos concerns aesthetic or descriptive purposes, but they have historically also been used for various functional purposes such as identification [17]. Other notable uses of tattoos include permanent makeup, which is a cosmetic technique where tattoos are employed on the skin to resemble makeup; however, permanent makeup is not considered in this work. A facial painting is a non-permanent manipulation where paint is applied to a face. As with tattoos, facial paintings are often intended for descriptive or aesthetic purposes; they are commonly seen at festivals and sports events. FIGURE 1Open in figure viewerPowerPoint Overview of the main types of facial manipulations. Example images are shown for the facial manipulations highlighted in bold In this work, focus is put on the scenario in which a face recognition system is presented with an image pair to be compared where either the reference or probe image has been altered by facial tattoos or paintings. It has been noted that tattoos and paintings are expected to have an impact on face recognition systems [5] but this topic has seen limited attention in the biometric scientific community. Suppose face recognition is applied to images where one or more individuals have facial tattoos or paintings. In such cases, the facial manipulations caused by tattoos or paintings likely affect face recognition. As such, it is of interest to investigate the impact these manipulations have on different modules of face recognition systems, which in turn affects the different scenarios where these systems are used. 1.1 Face detection For facial tattoos and paintings which cause severe facial changes, it is likely that they affect the ability of a face recognition system to detect the region of interest, that is the face. If so, it is possible that tattoos and paintings can be used as a presentation attack for identity concealment for facial images captured in unsupervised acquisition conditions. Furthermore, if these manipulations affect face detection, this impact would be propagated to subsequent modules of the face recognition pipeline. 1.2 Face quality estimation Another relevant scenario to consider is the effect of facial tattoos and paintings on face quality estimation. Assume, for instance, that facial tattoos and paintings have a low quality score compared to similar face images without any facial manipulations. Consequently, in systems where automated quality estimation is used prior to enrolment into the database, it might not be possible for all individuals with facial tattoos or paintings to be enrolled into the system. Considering how widespread face recognition is in both governmental (e.g. border control) and personal security applications (e.g. smartphone applications), ensuring inclusiveness and accessibility for everyone irrespective of individual appearance is an important concern. 1.3 Feature extraction and comparison Lastly, it is of interest to test whether the visual changes caused by facial tattoos and paintings have an impact on biometric recognition performance. This is relevant for all practical applications of facial biometrics; for example, in a criminal investigation when the face of a suspect with facial tattoos is compared to an old stored photo of the same suspect without tattoos. As such, it is interesting to assess whether face recognition systems are capable of reliably extracting features and distinguishing individuals with facial tattoos and paintings. To address the use-cases mentioned above, a database consisting of 500 image-pairs of individuals with and without facial tattoos or paintings was assembled from online sources. Thereafter, state-of-the-art commercial and open-source systems were used in an empiric evaluation in accordance with the current practices and international standards. The obtained results indicate that especially large facial tattoos and paintings have a significant impact on the performance of face recognition systems. To facilitate reproducible research, the used database is made available [18]. The work conducted herein is, to our knowledge, the first of its kind. Section 2 provides relevant background information and related works. Section 3 presents the assembled database. Sections 4 and 5 describe the conducted experiments and achieved results, respectively. Sections 6 and 7 conclude the work with a discussion of the obtained results and contributions. 2 RELATED WORK The following subsections briefly summarise the most relevant related works with respect to facial manipulations in the physical (Section 2.1) and digital domain (Section 2.2). 2.1 Physical facial manipulations Most research concerning the impact of permanent facial manipulations in the physical domain has been focussed on plastic surgery. Pioneering work in this field was done by Singh et al., who created databases containing facial images of individuals before and after plastic surgery [19, 20]. In [20], the authors evaluated the impact of plastic surgery on six face recognition algorithms. They concluded that more work should be done towards designing algorithms capable of dealing with the appearance changes occurring as a result of a facial plastic surgery operation. More recently, Rathgeb et al. [21] introduced a new database of plastic surgery images which are mostly compliant with the ICAO quality requirements [22]. Using the new database as well as the database proposed in [20], Rathgeb et al. showed that the impact of plastic surgery on state-of-the-art commercial and open-source face recognition systems is less significant than initially reported by other researchers. For a more detailed overview regarding the impact of plastic surgery on face recognition systems and the work conducted in this field, the reader is referred to [5]. More works addressed the impact of temporary face manipulations on face recognition. In [23], Ueda and Koyama investigated the impact that makeup has on humans' ability to recognise faces and found that it significantly decreased, especially in the presence of heavy makeup. In [24], Dantcheva et al. found that the presence of makeup negatively affects the accuracy of face recognition systems and concluded that more work was needed to address the challenges imposed by facial makeup. Similar findings were reported by Wang et al. in [25], where the authors investigated the impact of recognising human faces under disguise and makeup. Makeup is usually used for aesthetic purposes, but due to its ability to significantly alter the appearance of a face, attacks based on makeup can be effective against face recognition systems. In [26], the authors showed that face recognition systems are vulnerable to makeup-induced presentation attacks with the aim of impersonation. In [27, 28] Rathgeb et al. showed that such makeup presentation attacks of good quality could compromise the security of face recognition systems. Comparable work was done with the Computer Vision Dazzle Camouflage concept initiated by Adam Harvey [29], where hairstyles and makeup were used to prevent face detection, that is presentation attacks with the aim of concealment. The idea of the concept is to use prior knowledge of a specific algorithm to design looks which prevent detection by that algorithm. No dedicated research has investigated the impact of facial tattoos or paintings on face recognition systems. However, in [30], the authors investigated the impact of disguises on face recognition systems using a database which contains, among other types of disguises, facial paintings. The results show that even the top-performing algorithms that were tested perform poorly for strongly disguised faces. Notably, it was shown that a majority of genuine samples which were misclassified had occlusions over the face region and close to the periocular region. These findings are in agreement with other studies which have shown that partial facial occlusions negatively affect the performance of face recognition systems. In this context, it is believed that especially partially occluded faces captured in unconstrained environments will be a major challenge in the future [31]. 2.2 Digital facial manipulations Initial research concerning the impact that digital face manipulations have on face recognition was carried out by Ferrara et al. [32, 33] who, among other techniques, investigated the impact of beautification (retouching) and morphing. For facial retouching, the authors reported a significant drop in performance on all tested systems. These findings were confirmed by Bharati et al. [34, 35]. In [36], Rathgeb et al. showed that face recognition systems might be robust to facial images which have been digitally altered by the application of moderate facial retouching. The process of digitally editing attributes of a face has been eased by the creation of GAN-based algorithms which can automatically alter attributes of a face, for example the age or hair of an individual. The STGAN approach presented in [37] is an example of such an algorithm. For morphing, Ferrara et al. showed that the tested face recognition systems were vulnerable to attacks based on morphed images [32]. Since the initial experiments, much research has aimed at detecting morphed images; the reader is referred to [15] for a comprehensive survey. Besides the aforementioned, there exist several other types of digital facial manipulations, for example entire face synthesis, identity swap, as well as attribute manipulation and expression swapping; interested readers are referred to [38, 39] for more details. The so-called "deepfakes", where neural networks are used to create fake images, can also be used to create manipulated biometric images [38, 40]. A study [41] showed that deepfake videos pose a challenge for both face recognition systems and detection algorithms. Many researchers have undertaken the task of detecting manipulated biometric face images in recent years, and benchmarks have already been carried out, for example in [42]. 3 FACIAL PAINTINGS AND TATTOO DATABASE While numerous 2D face databases have been collected for the purpose of evaluating face recognition systems [43], there exists no publicly available database which can be used to investigate the impact of facial tattoos and paintings on face recognition systems. Therefore, an appropriate database was assembled using face images found online with Google image search and on YouTube. The collected database contains 500 pairs of face images of individuals with and without facial tattoos or paintings. Each image contains only a single visible face and, by visual inspection, it has been ensured that there is not a significant age gap between two face images in the same pairing. Examples of image pairs in the assembled database are given in Figure 2. FIGURE 2Open in figure viewerPowerPoint Examples of image pairs in the collected database. (a) Image pairs where one image in a pair has facial tattoos. (b) Image pairs where one image in a pair has facial paintings Four different scenarios were considered for the data collection of image pairs with facial tattoos, as illustrated in Figure 3. FIGURE 3Open in figure viewerPowerPoint Example image pairs illustrating the scenarios considered during the data collection. (a) Semi-permanent facial tattoo with a real before image. (b) Real tattoo with real before image. (c) Real tattoo with real after picture (surgical removal). (d) Real tattoo with an image where tattoos have been digitally removed The collected database has been balanced such that there are a total of 250 face images with facial tattoos and 250 with facial paintings; 232 of the image pairs are of males, and the remaining 268 images are of females. Out of the 250 images with facial tattoos, 73.6% of the images belong to men and 26.4% to females. On the other hand, 80.8% of the images with facial paintings belong to women. More details about the assembled database are given in Table 1 and examples for small, medium, or large coverage are shown in Figure 4. TABLE 1. Overview of the assembled database Category Subcategory Distribution Type Tattoo 50.0% Painting 50.0% Size Small: <1/3 facial coverage 31.2% Medium: 1/3–2/3 facial coverage 39.4% Large: >2/3 facial coverage 29.4% Gender Male 46.4% Female 53.6% Race Caucasian 41.0% Hispanic 23.0% Black 14.6% Indian 16.8% Others 4.6% Age Very young: <10 years old 13.8% Young: 10–30 years old 53.0% Middle-aged: 30–60 years old 31.0% Old: >60 years old 2.2% FIGURE 4Open in figure viewerPowerPoint Example images showing (a) small, (b) medium, and (c) large coverage of facial tattoos and paintings 4 EXPERIMENTAL SETUP The purpose of the experiments was to determine how facial tattoos and paintings influence algorithms' ability to detect faces in an image, estimate face quality, as well as extract and compare face features. Table 2 lists which algorithms were used to test each of the aforementioned modules. For the evaluation, the database described in Section 3 was used. Additionally, for the comparison and feature extraction module, subsets of constrained images from the FRGCv2 [44] and FERET [45] database were used. TABLE 2. An overview of the tested face recognition modules and used algorithms. COTS refers to the used commercial-off-the-shelf system Module Algorithms Face detection dlib, MTCNN, COTS Face quality estimation SER-FIQ, FaceQnetv1 Feature extraction and comparison ArcFace, COTS Abbreviation: COTS, commercial-off-the-shelf. The following subsections describe the experimental setup designed for evaluating the impact of facial tattoos and paintings on the detection (Section 4.1), quality estimation (Section 4.2), as well as the feature extraction and comparison (Section 4.3) modules of a face recognition system. In the following, let N be the images in the assembled database without tattoos or paintings, let T P be the facial images with tattoos or paintings, and let T P s m a l l , T P m e d i u m , and T P l a r g e be the subset of facial images with small, medium, and large facial tattoos or paintings, respectively. The obtained detection confidence scores, quality estimation scores, and comparison scores were normalised to the range [0,1]. For systems where the system produced a narrow range of scores, the results were scaled using min–max normalisation. 4.1 Face detection To test the impact of facial tattoos and paintings on face detection, two open-source algorithms (dlib [46], MTCNN [47]) and one commercial-off-the-shelf (COTS) system were used. The detection performance of the algorithms was measured on the N , T P , T P s m a l l , T P m e d i u m and T P l a r g e subsets of the assembled database. To this end, the confidence value (response score) produced by each algorithm for a detected region was recorded. The confidence value measures the likelihood that a detected region contains a face and as such a region with a high confidence value is likely to contain a face; whereas a region with a low confidence score is not. In case multiple regions were detected by an algorithm for a given image, the region with the highest confidence value was used. The confidence values produced for each of the subsets were compared using descriptive statistics calculated for each considered subset. Moreover, the detection error ( D ξ ), which is the percentage of facial images in a considered subset for which the algorithm detected zero faces, was recorded for each subset. 4.2 Face quality estimation Face quality estimation is used in a variety of application areas to estimate the utility of a facial image for face recognition; this is especially useful in unconstrained scenarios where low-quality facial images unsuitable for face recognition are likely to be captured. The FaceQnetv1 [48] and SER-FIQ systems [49] were used for measuring the quality score of the facial images. For FaceQnet, the images in the assembled database were cropped using the boundary box of the face detected by MTCNN. For SER-FIQ, the MTCNN face detector was used to normalise the facial images. For both systems, the quality score was estimated for each face in the database where MTCNN detected at least one face. In case multiple faces were detected, the face with the highest confidence score was used. Results were computed and compared for the N , T P , T P s m a l l , T P m e d i u m and T P l a r g e subsets using descriptive statistics. As the MTCNN detector was used for pre-processing for both SER-FIQ and FaceQnet, the errors were not reported as they were similar to the D ξ (see Section 4.1) scores of MTCNN. 4.3 Feature extraction and comparison The impact that facial paintings and tattoos have on a face recognition system's ability to extract and compare features was evaluated using state-of-the-art open-source (ArcFace [13]) and commercial (COTS) systems. Both considered systems are based on a deep-convolutional network and ArcFace has been trained on the MS1MV2 dataset [13]. The following scenarios were considered: G T P : Genuine comparisons where one image in a pairing is from N and the other from T P (see Section 3). G T P s m a l l : Genuine comparisons where one image in a pairing is from N and the other from T P s m a l l (see Section 3). G T P m e d i u m : Genuine comparisons where one image in a pairing is from N and the other from T P m e d i u m (see Section 3). G T P l a r g e : Genuine comparisons where one image in a pairing is from N and the other from T P l a r g e (see Section 3). G: Genuine comparisons where both images have no facial tattoos or paintings. Subsets of constrained facial images from the FRGCv2 and FERET image databases were used. I: Impostor comparisons where both images have no facial tattoos or paintings. Subsets of constrained facial images from the FRGCv2 and FERET image databases were used. For each of the above scenarios, the similarity scores were computed for both ArcFace and COTS. Table 3 shows the number of comparisons for each considered scenario. The obtained comparison scores were compared using descriptive statistics. Additionally, the equal error rate (EER), failure-to-enrol rate (FTE), and false non-match rate (FNMR) at a fixed false match rate (FMR) of 0.1% were calculated. Finally, the false reject rate (FRR) was estimated at a fixed FMR of 0.1%. The reader is referred to the ISO/IEC 19795-1 standard for the definitions of the used performance metrics [50]. TABLE 3. Number of biometric comparisons for each of the considered scenarios and systems Scenario ArcFace COTS G T P 460 475 G T P s m a l l 155 154 G T P m e d i u m 191 191 G T P l a r g e 114 130 G 4075 4076 I 561,804 561,804 Abbreviation: COTS, commercial-off-the-shelf. 5 RESULTS The following subsections report the results obtained for the experiments (Section 4) on face detection (Section 5.1), face quality (Section 5.2), as well as feature extraction and comparison (Section 5.3). 5.1 Face detection The scatterplots in Figure 5 compare the detection confidence scores obtained on the image pairs in the assembled database and illustrate that images with tattoos and paintings achieve comparably lower detection scores than the facial images without any manipulations for all three algorithms. These findings are further observed in Figure 6, which shows boxplots for the detection confidence scores for different subsets of the assembled database where it can also be observed that it becomes more difficult for algorithms to detect faces when the coverage of tattoos and paintings in a face region is increased. Table 4 shows the mean (μ) and standard deviation (σ) for the obtained detection confidence scores as well as the detection errors ( D ξ ) for different subsets of the database. The results show that facial tattoos and paintings decrease the detection confidence scores. Further, they cause significant increases in the percentage of times where the three algorithms fails to detect any faces, that is all three algorithms have a D ξ score above 10% when a large area of the face has been manipulated by tattoos or paintings. FIGURE 5Open in figure viewerPowerPoint Comparison of face detection confidence scores for images with and without facial tattoos or paintings. (a) dlib (b) MTCNN (c) COTS FIGURE 6Open in figure viewerPowerPoint Boxplots of face detection confidence scores. (a) dlib (b) MTCNN (c) COTS TABLE 4. Descriptive statistics of obtained confidence scores and error rates for face detection dlib MTCNN COTS Subset μ σ D ξ % μ σ D ξ % μ σ D ξ % N 0.45 0.14 0.60 1.00 0.02 1.00 0.61 0.17 0.60 T P 0.30 0.15 7.00 0.96 0.13 7.00 0.39 0.16 5.80 T P s m a l l 0.36 0.15 1.28 0.99 0.02 0.00 0.48 0.16 0.64 T P m e d i u m 0.29 0.14 6.60 0.97 0.10 2.54 0.39 0.15 4.06 T P l a r g e 0.23 0.14 13.61 0.91 0.20 20.41 0.31 0.14 13.61 Abbreviation: COTS, commercial-off-the-shelf. Not all detections where an algorithm produced the highest confidence value correspond to a correctly detected face. Figure 7 illustrates the distances between the eyes of the detected faces for each algorithm. While most of the eye distances are around 100 pixels or above, several outliers are also shown, which contain cases where faces have been incorrectly detected. Figure 8 shows examples of incorrect detections. Additionally, examples of facial images where zero faces were detected by one of the algorithms are shown in Figure 9. An analysis of the failure cases reveals that on many images where the algorithms failed to detect any faces, manipulations have occurred near the periocular region or a large area of the face was covered in tattoos or paintings, including areas around the mouth, nose, and eyes. However, it is not always the case that the algorithms fail to detect a face despite manipulations near the aforementioned areas. FIGURE 7Open in figure viewerPowerPoint Eye distances of detected faces. (a) dlib (b) MTCNN (c) COTS FIGURE 8Open in figure viewerPowerPoint Examples where the detection with the highest confidence score