# Face Beneath the Ink: Synthetic Data and Tattoo Removal with Application to Face Recognition
###### Abstract  
Systems that analyse faces have seen significant improvements in recent years and are today used in numerous application scenarios. However, these systems have been found to be negatively affected by facial alterations such as tattoos. To better understand and mitigate the effect of facial tattoos in facial analysis systems, large datasets of images of individuals with and without tattoos are needed. To this end, we propose a generator for automatically adding realistic tattoos to facial images. Moreover, we demonstrate the feasibility of the generation by using a deep learning-based model for removing tattoos from face images. The experimental results show that it is possible to remove facial tattoos from real images without degrading the quality of the image. Additionally, we show that it is possible to improve face recognition accuracy by using the proposed deep learning-based tattoo removal before extracting and comparing facial features.  
facial tattoos; synthetic data generation; tattoo removal; face recognition +
Footnote †: journal: Medical  
that tattoos are used for recognition purposes, which is something that raises ethical issues as discussed in [8].  
However, one major problem with face completion for tattoo removal is the lack of sufficient and high-quality training data, as no extensive database of facial tattoos is currently available.  
The main focus of this work is, therefore, two-fold. First, we propose a method for synthetically adding tattoos to facial images, which we use to create a large database of facial images with tattoos. The proposed method uses face detection and landmark localisation to divide the face into regions, whereafter suitable placements of tattoos are found. Subsequently, we approximate depth and construct depth and cut-out maps used to blend tattoos onto a face realistically. It has recently been shown that synthetic data can be beneficial for face analysis tasks and be a good alternative to real data [9; 10]. Secondly, we show the usefulness of our synthetic data by training a deep learning-based model for tattoo removal (as illustrated in Figure 1) and evaluate the impact of removing facial tattoos on a state-of-the-art face recognition system using a database comprising real facial images with tattoos.  
The approach for synthetically adding tattoos to a facial image in a fully automated way is, to the authors' best knowledge, the first of its kind. The proposed generator can be used to create large databases which can be used in related fields, e.g., tattoo detection or studying the effects of tattoos on human perception. Additionally, we are the first to measure the effect of removing facial tattoos on face recognition systems.  
In summary, this work makes the following contributions:  
* A novel algorithm for synthetically adding facial tattoos to face images.
* An algorithm for removing tattoos from facial images trained on only facial images with synthetically added tattoos. We refer to this algorithm as TRNet.
* An experimental analysis of the quality of the tattoo removal.
* Showcasing the application of tattoo removal in a face recognition system by conducting an experimental analysis on the effect of removing facial tattoos on a face recognition system.  
Figure 1: Examples of using deep learning-based tattoo removal.  
The outline of the remaining article is as follows: Section 2 describes prominent related works, Section 3 describes an automated approach for synthetically blending tattoos to facial images, which is used in Section 4 to generate a database of facial images with tattoos. Sections 5 and 6 show the feasibility of the synthetic generation by training a deep learning-based model for tattoo removal and evaluating if it can improve biometric recognition performance, respectively. Finally, Section 7 provides a summary of this work.