SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxin Cao*
Yian Li*
Yumeng Zhu
Ping An Technology
(Shenzhen) Co., Ltd.
Derui Wang
Minhui Xue
Anti-spoofing detection has become a necessity for face recognition systems due to the security threat posed by spoofing attacks. Despite great success in traditional attacks, most deep-learning-based methods perform poorly in 3D masks, which can highly simulate real faces in appearance and structure, suffering generalizability insufficiency while focusing only on the spatial domain with single frame input. This has been mitigated by the recent introduction of a biomedical technology called rPPG (remote photoplethysmography). However, rPPG-based methods are sensitive to noisy interference and require at least one second ( $>25$  frames) of observation time, which induces high computational overhead. To address these challenges, we propose a novel 3D mask detection framework, called FASTEN (Flow-Attention-based Spatio-Temporal aggrEgation Network). We tailor the network for focusing more on fine-grained details in large movements, which can eliminate redundant spatio-temporal feature interference and quickly capture splicing traces of 3D masks in fewer frames. Our proposed network contains three key modules: 1) a facial optical flow network to obtain non-RGB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation to aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms eight competitors for both intra-dataset and cross-dataset evaluations in terms of multiple detection metrics. Moreover, FASTEN has been deployed in real-world mobile devices for practical 3D mask detection.
Face recognition has been deeply involved in various prevalent applications, such as access control systems and e-commerce [1]. Nevertheless, similar to the potential security threat in fields of images [2, 3] and streaming videos [4, 5], face recognition systems are encountered with security issues mainly caused by face presentation attacks in the forms of printed photos, replayed videos, 3D masks [6, 7]. Since face images are easily accessible on social networks, one can mass-produce spoofing faces at low acquisition and manufacturing costs. To mitigate the detriment caused by face presentation attacks, a plethora of face presentation attack detection (PAD) methods have sprung up in the last decade [8-18]. Despite the acceptable detection performance against printings/screens, 3D masks can undermine these PAD methods by a large margin due to advanced 3D printing
*This work was done while Yuxin and Yian worked as interns at Ping An Technology (Shenzhen) Co., Ltd.
technologies [19, 20]. 3D masks made up of soft silicone are vivid enough, in terms of color, texture, and geometry structure, that even humans find it confusing to differentiate. This has gradually made 3D mask defense from a branch of face anti-spoofing to a relatively independent sub-topic [19, 21, 22].
Existing 3D mask PAD methods can be categorized into three branches: Handcrafted-Features-based methods (HFMs) [9-12, 23], Deep-Learning-based methods (DLMs) [15-18] and remotePhotoplethysmography-based methods (rPPGMs) [21, 24-28]. In earlier years, 3D masks were mainly distinguished by manually annotated features. In the last decade, many works have greatly improved the performance of spoofing detection through the powerful advantages of deep learning. However, HFMs and DLMs do not fully consider temporal differences, resulting in average effectiveness in cross-dataset evaluation. There has been a new breakthrough with the emergence of rPPG technology, which captures the color variations caused by heartbeat when the face skin is illuminated by light. rPPGMs achieve splendid performance on cross-dataset evaluation, but rPPG signals are touchy about noise interference [28]. The release of the HiFiMask dataset [29] also challenges rPPGMs since the light frequency they used is similar to that of the heartbeat, leading to loss of utility of rPPG signals. We summarize the limitations of these methods as follows. 1) Data availability. Most methods obtain data from devices that are difficult to deploy in various practical scenarios, e.g., depth information captured by RGBD cameras [30], rPPG signals obtained by light [27]. 2) Temporal information involvement. Few works focus on temporal features, which can carry variations of facial details. 3) Computational cost. The shortest observation time required for rPPGMs is one second (> 25 frames), which is still long enough to trigger a high computational cost.
To address the limitations above, we propose FASTEN (Flow Attention-based Spatio-Temporal aggrEgation Network), which only needs five frames of the input RGB video (no depth, rPPG, or other information is needed) to distinguish between 3D masks and real faces. FASTEN fully utilizes temporal and spatial information by focusing more on fine-grained details in large movements, which can avoid redundant feature interference and lower the computational cost. We consider frame-wise attention to aggregate the temporal and spatial feature information of multiple frames, rather than simple concatenation, since the latter practice neglects the frame significance discrepancy, especially when the input frames include large movements, e.g., eye blinking and mouth opening. Concretely, we first train a facial optical flow network, named FlowNetFace, to obtain RGB-free inter-frame flow information. The frame weights are then calculated by facial optical flow and assigned to each frame as their attention scores. Finally, the high-level spatial features and temporal transition features are aggregated for detection. In summary, the contributions of this paper are listed as follows.
- We propose a flow-attention-based spatio-temporal aggregation network, called FASTEN, for 3D mask PAD. FASTEN outperforms existing works in intra-/cross-dataset evaluations.
- In order to thoroughly take advantage of the temporal information of spatial features, we train a facial optical flow model, FlowNetFace, to learn the small variations in facial features between frames. The facial features are applied simultaneously to the calculation of frame weights and temporal transitional features.
- We address the long-time issue that the state-of-the-art rPPGM still needs about one second ( $>25$  frames) for observation, and propose a frame-based method that only needs any five frames of a 3D mask video, by adding frame weights to multi-frame spatial features. We also release a lightweight application for FASTEN to mobile devices, showing that FASTEN can achieve practical 3D mask PAD on resource-constrained platforms.
Handcrafted-Features-based Methods. In early studies, due to the conspicuous artifacts in presentation attacks such as printings and screens, facial characteristics and handcrafted descriptors, such as LBP [31, 32], HOG [8] and SIFT [33], were utilized to distinguish spoofing images. These characteristics, however, are not adaptable to 3D masks, since masks do not incorporate face quality defects such as Moiré patterns [34] or reflection [35]. Analyzing color texture [9] in chroma images is also a common kind of feature to detect color reproduction artifacts in masks. From the perspective of dynamic features, facial cues such as eye-blinking [10, 23] and face movement [11, 12] are commonly used to discriminate printings and screens, but are not congenial to 3D masks [36].
Deep-Learning-based Methods. In recent years, a plethora of DLMs [15-18, 37-39] have burst out.  $\mathrm{CDCN} + +$  [15] uses Central Difference Convolution (CDC) to learn central gradient information after
Figure 1: Overview of our proposed FASTEN.
which a multi-scale attention module is added to fuse multi-scale features. Due to its practicality and popularity, some follow-up work [40, 41] is based on  $\mathrm{CDCN}++$ . HRFP [37] separates faces into several parts to extract fine-grained information. It came out first in a 3D high-fidelity mask attack detection challenge held by the International Conference on Computer Vision (ICCV) Workshop in 2021. Temporal Transformer Network (TTN) [30] has recently been proposed by learning subtle temporal differences from depth maps captured by RGBD cameras. The depth maps of all kinds of spoofing images are set as 0 during training. However, 3D masks actually contain depth information and cannot be ignored, leading to mediocre detection ability towards 3D masks. Besides, depth information is hard to obtain in practical scenarios. Thanks to the weak generalizability of deep learning networks, DLMs find it tough to detect unseen 3D masks.
rPPG-based Methods. Emerging in recent years, rPPG achieves non-contact monitoring of human heart activity by using RGB cameras to detect subtle color variations in facial skin caused by heartbeat under room light [42, 43]. Since light transmission is blocked in 3D masks, the heartbeat can only be detected from real faces [44]. As a result, rPPGMs achieve much success when distinguishing 3D masks [21, 24-28]. Among advanced work, MCCFrPPG [28] improves CFrPPG [44] by extracting temporal variation from rPPG signals through a multi-channel time-frequency analysis scheme to mitigate interference of noise on rPPG signals. To shorten the required observation time from 10 seconds to within one second, LeTSrPPG [21] evaluates the temporal similarity of the rPPG features and proposes a self-supervised-learning-based spatio-temporal convolution network to distinguish 3D masks in terms of the consistency of the rPPG signals. However, a one-second video still includes many frames ( $>25$ ) and requires a significant amount of computation. The HiFiMask dataset [29] also challenges rPPGMs since the light frequency they used is similar to that of heartbeat, leading to loss of distinguishment utility of rPPG signals.
Given a video  $v = [x_{1}, x_{2}, \dots, x_{T}]$  containing  $T$  ( $T \geq \lambda$ ) frames of a real face or a face wearing a 3D mask, our goal is to build a discriminative network that can accurately distinguish 3D masks from real faces, where  $\lambda$  denotes the required input frame number (5 in our case). Different from the existing 3D mask detection method, FASTEN relaxes the restrictions on the input, i.e., we only need any  $\lambda$  frames in sequential order from the video  $x = [x_{a_{1}}, x_{a_{2}}, \dots, x_{a_{\lambda}}]$ ,  $1 \leq a_{1} < a_{2} < \dots < a_{\lambda} \leq T$ , where the selected adjacent frames can be either equally spaced or unequally spaced. The framework of FASTEN is depicted in Figure 1. In our design, three key components are integrated into the backbone network: 1) We first employ an optical flow model FlowNetFace to generate facial optical flow between adjacent frames. 2) The flow attention module is used to calculate the frame weights over time and assign different attention to each frame. 3) Finally, the temporal transition features are extracted from FlowNetFace and aggregated with the high-level spatial features for detection.
3.1 Feature extractor
Spatial features capture facial details, especially information on facial texture, which is conducive to 3D mask detection. We use  $0.75 \times$  MobileNetV3-Small [45], which contains four blocks, as our
backbone to balance accuracy and efficiency. Considering that it is rather computationally expensive if low-level spatial features  $F = [F_{a_1}, F_{a_2}, \dots, F_{a_\lambda}]$  are in high resolution, the low-level spatial features of  $\lambda$  frames are extracted from the first two stages of MobileNet with shared network weights. Also, the receptive field is 19 after the first two blocks. Corresponding to our input face region with a resolution of  $96 \times 112$ , it is just right at this stage to extract the features of a specific part (e.g., eyes or corners of mouth), hence the fusion of multi-frame spatial features can be better aligned.
Temporal features encode the motion patterns of facial muscles, which is beneficial to distinguish fake faces bearing a strong resemblance to real ones in appearance. We choose facial optical flow maps to represent temporal features based on the following insights: 1) Unlike rPPG, optical flow, which embeds the motion speed and direction of each pixel, can preserve the facial structure while partially neglecting the interference caused by color diversity. This helps to withstand multiple challenges of robustness such as acquisition equipment and facial skin color. 2) By comparing the optical flow of every two adjacent frames in a clip, we can identify moments with larger movement and more exposed traces, guiding the network to allocate attention reasonably and thereby reducing the demand for frame numbers.
3.2 Facial optical flow network
Since there are no existing optical flow models specifically designed for faces, we propose an active-learning-based alternative strategy to train a facial optical flow prediction network ourselves. As one of the classic methods in the field of optical flow, FlowNet [46] generally contains an encoder/decoder structure and works well on the proposed generic optical flow dataset named Flying Chairs. The variants of FlowNet include FlowNetS, FlowNetC, FlowNet2.0, etc, whose performance improves gradually as well as computational complexity. Unlike the Flying Chairs dataset, which is composed of high-resolution images with multiple targets, face images are rather simple. Therefore, we build a facial optical flow model evolved from FlowNetS by reducing the layer number to 2 in both the encoder and the decoder. In this case, our optical flow model, namely FlowNetFace, can obtain the temporal features required at the lowest computational cost. In addition, we use FlowNet2.0 [47], a stacked deep-learning-based optical flow estimator, to generate the ground truth of the facial optical flow. Following prior work [46, 47], the average endpoint error (AEPE) is used to measure the distance between the predicted and ground truth optical flow vectors.
3.3 Flow attention
The most significant difference between 3D masks and real faces is basically the splicing traces on the face. Especially, when a person is asked to blink the eyes or open the mouth, the unnatural details are particularly noticeable, e.g., compared to static faces, the detailed features around the eyes when eyes are closed are more helpful in distinguishing masks from real faces. As aforementioned, mask-splicing clues change with facial expressions. Therefore, scoring spatial features through temporal information can make the network autonomously attend to significant frames, thereby amplifying the features of facial details under special actions as well as weakening interfering information. Concretely, we input optical flow of the selected  $\lambda$  frames, sequenced by a  $3\times 3$  convolution layer to calculate the frame weights  $W = [w_{a_1},w_{a_2},\dots,w_{a_\lambda}]$  for the spatial features of each frame. Then, the frame weights are fused to the spatial features  $F$ , and the output fused feature can be expressed as
$$
F _ {f u s e} = \sum_ {i = 1} ^ {\lambda} w _ {a _ {i}} F _ {a _ {i}}, \quad \text {s . t .} \sum_ {i = 1} ^ {\lambda} w _ {a _ {i}} = 1. \tag {1}
$$
3.4 Spatio-temporal aggregation
As for spatial information, the fused feature  $F_{fuse}$  is then fed into the latter two blocks of the backbone to obtain the spatial feature  $F_{spa}$ . It is apparent that there exist smooth facial fluctuations and muscle movements in real faces when a video is captured, while no such phenomena are presented in 3D masks. Thus, we additionally consider extracting the temporal transitional feature  $F_{temp}$  that can represent the temporal differences between consecutive frames. Since the resolution of the optical flow features is eight times that of  $F_{spa}$ , we input them into the module composed of the latter three blocks of MobileNet, resulting in temporal features of the same size as deep-level spatial features. Since the spatial feature can reflect the splicing traces in the input face image, and the temporal transitional feature brings the transitional movement cues, we aggregate both the spatial and temporal features by concatenation and then input the aggregated into the classifier which contains a fully
connected layer followed by a Softmax layer. Due to the data imbalance between real faces and 3D masks, we use the weighted BCE loss to train the network. Given a batch of  $N$  face images, we have
$$
\mathcal {L} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ - \mu y _ {i} \log p _ {i} - \left(1 - y _ {i}\right) \log \left(1 - p _ {i}\right) \right], \tag {2}
$$
where  $y_{i}$  denotes the true class of the  $i$ -th image, with 1 standing for real faces, and 0 representing 3D masks.  $p_{i}$  denotes the output score of the classifier.  $\mu$  denotes the balancing weight.
3.5 Training strategy
Considering that the proposed FASTEN network includes several modules, the training strategy is given as follows. 1) We first separately train the facial optical flow network FlowNetFace using the face dataset and the ground truth flow obtained by FlowNet2.0  ${}^{2}$  . Since optical flow prediction is independent of detection, training FlowNetFace alone can not only converge the network as soon as possible but also avoid conflicts caused by two different tasks. 2) Then we use the MobileNetV3-Small model pre-trained on ImageNet  ${}^{3}$  as our backbone. We finally use a small learning rate to finetune the entire network on 3D mask datasets, with a smaller learning rate for FlowNetFace. Please refer to our code for more details  ${}^{4}$  .
Datasets. We conduct our experiments on three frequently used datasets, 3DMAD (3D Mask Attack Dataset) [19], HKBU-MARs V1+ (Hong Kong Baptist University 3D Mask Attack with Real-World Variations) [48] and HiFiMask (High-Fidelity Mask dataset) [29]. 1) 3DMAD [19] contains 255 videos of 17 subjects wearing custom-built Thatsmyface masks where the eyes and nose holes are cut out to present more natural wearing effects. The videos are recorded by a Kinetic camera at  $640 \times 480$  resolution, 30 fps. Both color and depth data are collected. 2) HKBU-MARs V1+ [48] includes 180 videos of 12 subjects with two mask types, Thatsmyface and REAL-f. Videos in HKBU-MARs V1+ are captured by Logitech C920 under room light. 3) HiFiMask [29] is a large, high-resolution mask dataset containing 54,600 videos from 75 subjects wearing three types of masks: plaster masks, resin masks, and transparent masks. HiFiMask considers six different scenes, including light variation and head motion. The frequency of the periodic flashlight is set within  $[0.7, 4]\mathrm{Hz}$ , which is consistent with that of ordinary people's heartbeat, to a great extent challenging rPPGMs since pseudo 'liveness' signals can be obtained for masks. More details of the datasets are shown in Table 1. HiFiMask is the most challenging dataset for PAD methods, owing to its large quantity and high fidelity.
Competitors. We select eight existing detection methods for spoofing images or specifically for 3D masks from HFMs, DLMs, and rPPGMs as our competitors, i.e., MS-LBP [32], CTA [9],  $\mathrm{CDCN} + +$  [15], HRFP [37], ViTranZFAS [38], MD-FAS [39], CFrPPG [44], and LeTSrPPG [21]. In the early ages, MS-LBP [32] and CTA [9], as two classic feature descriptors, exhibit edges in finding quality defects in terms of color and texture. Although most of the DLMs are not tailored for 3D masks, we select  $\mathrm{CDCN} + +$  [15], one of the most popular and frequently used deep-learning networks, to represent this branch of detection methods, since the fine-grained features of spoofing faces learned by central difference convolution and multi-scale attention module are expected to work for 3D masks. HRFP [37] was crowned with laurel in the 3D high-fidelity mask attack detection challenge held by ICCV in 2021. By segmenting faces into several parts for feature extraction, HRFP rejects the input face once an anomaly exists in any part of the input. Two rPPGMs, CFrPPG [44] and LeTSrPPG [21], are also included for comparison due to their strong performance in identifying 3D masks based on biotechnology. We further consider two DLMs using pre-trained models due to their outstanding performance in face anti-spoofing tasks. ViTranZFAS [38] used transfer learning from a pre-trained vision transformer (ViT) to distinguish spoofing images. MD-FAS [39] was later proposed to improve domain adaptation by only using target domain data to update the pre-trained face anti-spoofing model. For all the competitors, we follow their original settings.
$^{2}$ https://github.com/NVIDIA/flownet2-pytorch
<sup>3</sup>https://github.com/d-li14/mobilenetv3.pytorch
<sup>4</sup>https://github.com/JosephCao0327/FASTEN
Table 1: Concrete details of different datasets we used. The resolution of HiFiMask varies.
Dataset	Year	#Subject	#Mask	Mask type	Scenes	Light	Camera	#Live videos	#Masked videos	#Total videos	Resolution	Facial resolution
3DMAD [19]	2013	17	17	2	1	1	1	170	85	255	640 × 480	80 × 80
HKBU-MARs V1+ [48]	2018	12	12	2	1	1	1	120	60	180	1280 × 720	200 × 200
HiFiMask [29]	2021	75	75	3	6	6	7	13,650	40,950	54,600	High fidelity	400 × 400
Statement for rPPGMs. Due to the lack of open-source codes for rPPGMs, we are unable to obtain rPPG signals to replicate their method. Thus, we cite the results reported in their paper on 3DMAD and HKBU-MARs V1+. We do not involve one recent work MCCFrPPG [28] since the observation time is too long (10~12 seconds) to be fairly compared with our method, which only needs five frames. Instead, we select its previous version CFrPPG [44] since the detection results with an observation time of one second are available in the paper of LeTSrPPG [21]. Note that HiFiMask only contains 10 frames sampled from each video at equal intervals, the sampling frequency of which is much lower than twice the highest light frequency (4Hz for periodic flashlight). According to the Nyquist-Shannon Sampling Theorem [49], recovering rPPG analog signals will inevitably lead to frequency aliasing and signal distortion. Ergo, we do not consider rPPGMs on HiFiMask.
Metrics. Following prior work [15, 21, 28, 37], we use five metrics to evaluate the detection performance, i.e., Half Total Error Rate (HTER), Equal Error Rate (ERR), Area Under Curve (AUC), Attack Presentation Classification Error Rate (APCER), and Bonafide Presentation Classification Error Rate (BPCER). We use "B@A=0.1/0.01" to represent the BPCER when APCER = 0.1/0.01.
Parameter settings. In the training process, we adopt commonly utilized augmentations including Gaussian noise, random brightness contrast, and cut-out. FlowNetFace is trained (finetuned) using AdamW optimizer with a learning rate of 1e-4 (2e-5) and weight decay as 4e-4 (1e-2 for 3DMAD and HKBU-MARs V1+, and 1e-3 for HiFiMask). The rest of the network is finetuned using the same optimizer with a learning rate of 2e-4, which is adjusted by cosine annealing with warm restarts. We set the warm-up epoch number as 2 and the total epoch number as 150. We set the required input frame number  $\lambda = 5$ , the balancing weight  $\mu = 0.5$  for 3DMAD and HKBU-MARs V1+ and  $\mu = 3$  for HiFiMask. The batch size is 120, and all models are trained using two Tesla V100 GPUs. We perform face alignment [50] on images before inputting them into the network, so that spatial features can be fused without misalignment.
Intra-dataset evaluation. For the intra-dataset evaluation, we adopt leave one out cross-validation (LOOCV) protocol on 3DMAD and HKBU-MARs V1+. Following [28], a test subject is randomly selected first for both two datasets. Then we randomly select eight/five subjects for training and the leftover eight/six subjects for validation on 3DMAD/HKBU-MARs V1+. LOOCV is not suitable for large datasets such as HiFiMask, since LOOCV is subject to high variance or overfitting [51]. Instead, we use  $k$ -fold cross validation (KCV) when training on HiFiMask. In this case, we follow Protocol 1 [29], where 45, 6 and 24 subjects are selected for training, validation and testing, respectively.