# A Comprehensive Survey of Masked Faces: Recognition, Detection, and Unmasking
## 1 Introduction
### Challenges in MFR  
The advent of face masks amid the COVID-19 pandemic has posed formidable challenges to facial recognition systems, leading to a significant decline in performance. The concealment of crucial facial features, including the nose, mouth, and chin, has triggered a cascade of obstacles, and addressing these challenges becomes imperative for advancing MFR technology. * Scarcity of Datasets: The scarcity of datasets tailored for masked face recognition constitutes a pivotal challenge. Training any deep learning model requires a robust dataset, yet the shortage of publicly available datasets featuring masked faces complicates the development of effective MFR methods. Researchers tackling this challenge often resort to creating synthetic datasets by introducing masks to existing public face datasets like CASIA-WebFace [4], CelebA [5], and LFW [6]. To simulate masked-unmasked pairs, popular methods involve using deep learning-based tools such as MaskTheFace [7] or leveraging generative adversarial networks like CycleGAN [8]. Manual editing using image software, exemplified by the approach in [3], further supplements dataset generation efforts. Figure 1: Illustration showcasing the tasks of masked face recognition (MFR), face mask recognition (FMR), and face unmasking (FU) with varied outputs for the same input. Training on Synthetic Data:  
While synthetic datasets offer a practical solution when real-world masked face datasets are scarce, relying solely on AI-generated data introduces certain challenges. Models trained exclusively on synthetic data may overfit to the specific features or artifacts inherent in the data generation process, rather than learning to recognize real-world occlusions. This can result in a diminished performance when these models are deployed in real-world scenarios involving masked faces. To mitigate this risk, it is essential to balance the use of synthetic datasets with real-world data or apply fine-tuning techniques on real-world samples, ensuring the model's generalizability. Additionally, models trained on AI-generated masks may develop an oversensitivity to synthetic artifacts, further necessitating the use of adversarial training approaches to enhance their robustness across a wide range of mask types, whether synthetic or real. * Dataset Bias: In addition to the scarcity of publicly masked datasets, a prominent challenge lies in the bias inherent in existing benchmark datasets for MFR. Many widely used datasets exhibit a notable skew towards specific demographics, primarily favoring male and Caucasian or Asian individuals. This bias introduces a risk of developing MFR systems that may demonstrate reduced accuracy when applied to individuals from other demographic groups. To mitigate dataset bias in MFR, efforts should be directed towards creating more inclusive and representative benchmark datasets. This involves intentionally diversifying dataset populations to encompass a broader spectrum of demographics, including gender, ethnicity, and age. * Occlusion Complexity: The complexity introduced by facial occlusion, particularly the masking of the mouth, poses a significant hurdle to existing face recognition methods. The diverse sizes, colors, and types of masks exacerbate the challenge, impacting the training of models for various masked face tasks, including recognition, detection, and unmasking. Strategies to address this complexity vary by task. Recognition methods may employ attention models [9; 10] that focus on the upper half of the face or exclusively train on this region. Another approach involves using face mask removal methods as a pre-step before recognition. In unmasking tasks, researchers may introduce a pre-stage to detect the mask area, as demonstrated by generating a binary mask map in the first stage in [2]. Training datasets are further diversified by incorporating various mask types, colors, and sizes to enhance model robustness.