SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The intra-dataset evaluation results reported in Table 2 show that FASTEN performs better than the other eight competitors, with AUCs of  $99.8\%$ ,  $99.7\%$ , and  $99.8\%$ , and HTERs of  $1.17\%$ ,  $2.13\%$  and  $2.11\%$  on 3DMAD, HKBU-MARs V1+ and HiFiMask, respectively. We attribute the outperformance to FASTEN's strong ability to distinguish 3D masks by assigning different weights to each frame and aggregating both spatial and temporal information. MS-LBP ranks second on 3DMAD since handcrafted features perform well under fixed lighting conditions. However, the performance of MS-LBP is limited in complex lighting and different mask types. HRFP comes out second and third in both HKBU-MARs V1+ and HiFiMask since the fine-grained features of face parts at high resolution in HRFP help identify subtle differences in faces.  $\mathrm{CDCN}++$  also performs well owing to the multi-scale attention which can effectively aggregate multi-level features of central difference convolution. We do not see significant advantages for rPPGMs due to noise interference by various lighting conditions and camera settings. As previously discussed, rPPGMs are challenged by HiFiMask, which may induce a loss of utility.
Cross-dataset evaluation. We also carry out a cross-dataset evaluation to verify the generalizability of FASTEN when test scenarios have never been seen before. Models are trained on one dataset and then tested on the test set of another dataset. Table 3 reports the cross-dataset evaluation results. We first consider the scenarios between two small datasets, 3DMAD and HKBU-MARs V1+. The performance of HFMs drops dramatically due to the limitations of simple color and texture features.
Table 2: Intra-dataset evaluation on different datasets. 'U': unavailable. 'N/A': not applicable.
Method	3DMAD	HKBU-MARs V1+	HiFiMask (Protocol 1)
HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01
MS-LBP [32]	1.92±3.4	3.28	99.4	0.32	6.78	24.0±25.6	22.5	85.8	48.6	95.1	48.3	40.5	63.7	78.9	97.3
CTA [9]	4.40±9.7	4.24	99.3	1.60	11.8	23.4±20.5	23.0	82.3	53.8	89.2	40.7	31.6	74.9	64.1	92.9
CDCN++ [15]	4.20±7.1	8.34	96.7	6.62	59.6	4.83±7.6	8.70	96.0	7.77	66.2	3.67	2.64	99.6	0.83	6.79
HRFP [37]	2.34±5.2	2.41	99.7	0.67	5.87	3.34±5.7	4.33	99.2	1.34	6.23	2.20	2.26	99.7	0.28	4.35
ViTranZFAS [38]	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	2.63	2.48	99.7	0.21	6.58
MD-FAS [39]	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	5.45	4.12	99.4	0.34	15.2
CFrPPG (1s) [28]	32.7±7.4	32.5	70.8	U	U	42.1±5.6	42.0	60.8	U	U	N/A	N/A	N/A	N/A	N/A
LeTSrPPG [21]	11.8±8.6	11.9	94.4	U	U	15.8±6.5	15.7	91.5	U	U	N/A	N/A	N/A	N/A	N/A
FASTEN(Ours)	1.17±0.7	1.06	99.8	0.03	0.31	2.13±5.1	2.56	99.7	0.81	2.83	2.11	2.18	99.8	0.13	4.13
Table 3: Cross-dataset evaluation. 'U': unavailable. 'N/A': not applicable.
Method	3DMAD → HKBU-MARs	V1+	HKBU-MARs V1+ → 3DMAD	HiFiMask → 3DMAD	HiFiMask → HKBU-MARs V1+
HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01
MS-LBP	47.7±7.0	48.3	52.4	86.4	97.6	43.2±7.3	43.7	58.8	87.5	99.2	47.8	41.3	62.6	76.3	98.0	50.4	57.0	39.1	93.0	98.8
CTA	51.5±2.4	55.3	48.9	90.5	98.8	68.2±7.7	65.4	40.1	94.7	99.9	50.1	71.5	32.2	97.6	99.8	49.5	63.3	42.9	90.0	94.5
CDCN++	50.3±2.7	55.3	42.3	93.2	99.9	41.1±6.8	33.1	66.2	74.1	98.9	36.8	37.7	72.1	65.9	87.1	35.7	29.7	80.8	52.9	78.4
HRFP	25.8±3.1	32.3	69.2	81.3	98.4	6.76±0.9	7.25	98.6	5.63	22.0	21.3	12.4	95.8	14.6	32.5	7.68	8.33	99.1	1.52	11.4
ViTranZFAS	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	26.3	16.5	91.1	23.1	60.8	32.7	20.6	88.3	34.2	60.2
MD-FAS	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	33.5	34.4	69.9	67.0	75.3	9.38	8.61	97.4	5.67	43.4
CFrPPG (1s)	39.2±1.4	40.1	63.6	75.5	U	40.1±2.3	40.6	62.3	79.1	U	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
LeTSrPPG	15.7±0.5	16.6	90.1	25.2	U	12.9±0.8	13.1	93.4	15.8	U	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
FASTEN(Ours)	11.8±2.1	16.8	91.0	18.8	24.4	3.85±1.9	4.35	99.1	3.00	8.88	2.35	2.71	99.8	0.27	4.76	7.38	8.55	98.3	5.40	17.7
Despite the strong performance in intra-dataset, DLMs exhibit a significant decline in performance, e.g., the AUC for  $\mathrm{CDCN}++$  on 3DMAD drops from 96.7 to 42.3 when testing HKBU-MARs V1+. Such results are attributed to the fact that DLMs lack generalization when encountered with unseen inputs. Another major reason why FASTEN outperforms these methods in cross-dataset evaluation could be the aggregation between spatial and temporal features, the latter of which carries a large amount of interframe information and can better capture the flaws in splicing traces. Although rPPGMs have been verified to have strong generalization ability, their accuracy is easily affected by noise and light frequency, resulting in slightly lower performance than FASTEN.
Due to the large gap between the two small datasets and HiFiMask in data quantity, we discard the evaluation from small datasets to HiFiMask. Conversely, we conduct cross-dataset evaluation when the model is trained on HiFiMask, and test on 3DMAD and HKBU-MARs V1+. FASTEN achieves over  $98\%$  AUCs with HTER below  $8\%$  in both scenarios. As for rPPGMs, they cannot be implemented on HiFiMask due to the sampling problem. Since the light frequency is similar to that of heartbeat, the rPPG signals of 3D masks contain pseudo 'liveness' cues in both time and frequency domains, misguiding rPPGMs to incorrectly classify 3D masks as real faces. Therefore, we point out that their performance could be very limited, even if the proposer of HiFiMask [29] provided complete videos before sampling and the recovered rPPG signals. Overall, the outperformance of FASTEN shows the generalizability of our proposed framework on both small and large datasets.
Discussion of ViTranZFAS and MD-FAS. ViTranZFAS requires large amount of data for training, while MD-FAS focuses more on domain adaptation, whose experimental setting is slightly different from 3D mask defense. Therefore, we only conduct experiments on HiFiMask. We attribute the poor performance of ViTranZFAS to three main reasons: 1) Lack of temporal information: ViT focuses primarily on analyzing individual images without considering temporal cues, but 3D mask detection often requires understanding the dynamic changes over time. 2) Small regions of interest: Liveness cues often involve subtle facial regions, such as the edge of facial features or muscle contractions. However, ViT's patch-based approach may not focus on these small regions of interest effectively, leading to challenges in capturing fine-grained liveness indicators. 3) Training difficulty: ViT requires a pre-trained model and high computational costs. MD-FAS aims to solve the forgetting issue when learning new domain data and improve the adaptability of the model. It cannot work well with unseen target data, as shown in Table 3. In addition, the baseline SRENet selected by this scheme is a region-based fine classification task, which has high requirements for the diversity of attack categories, as well as the resolution of images. When we use HiFiMask as a training set for a fair comparison, its accuracy performance remains poor.
Overall evaluation. We calculate the average AUC, ERR, and HTER of all the PAD methods in both intra-dataset and cross-dataset evaluations to compare the overall performance. The results in
Figure 2: Comparison of both intra and cross-dataset evaluations.
Figure 2 demonstrate that simultaneously considering temporal and spatial features helps FASTEN achieve the best detection performance and generalizability.
Computational overhead. We further compare the computational cost of different PAD methods when detecting 3D masks. Since MS-LBP and CTA only need handcrafted features for detection and rPPGMs require long videos with rPPG signal extraction, we only compare  $\mathrm{CDCN}++$  and HRFP with our proposed method under the same inputs. The average inference time of  $\mathrm{CDCN}++$ , HRFP and FASTEN is  $28.8\mathrm{ms}$ ,  $286.5\mathrm{ms}$  and  $25.3\mathrm{ms}$  respectively. Compared to HRFP, FASTEN reduces computational cost and saves inference time with the benefit of smaller networks. Although  $\mathrm{CDCN}++$  achieves a comparable time cost with FASTEN, the lack of consideration of temporal features leads to insufficient detection performance. Compared with the best rPPGMs, LeTSrPPG which requires one second ( $>25$  frames) for observation, FASTEN only needs five frames as input, which can significantly reduce computational overhead.
Effectiveness of components. There are two main components in FASTEN, the spatial branch and the temporal branch. To verify the effectiveness of each component, we consider the following five scenarios. 1) Only spatial features considered (Spat.): the temporal branch, including optical flow prediction, frame weight assignment, and temporal transitional feature extraction, is removed. 2) Only temporal features considered (Temp.): the spatial feature extraction and frame weight assignment are removed. 3) Equal frame weights (Spat. + Temp.(Eq.)): the frame weight assignment is replaced by equally assigning frame weights ( $\forall i \in \{1, 2, \dots, \lambda\}$ ,  $w_{a_i} = \frac{1}{\lambda}$ ) to each frame. 4) Different frame weights (Spat. + Temp.(UnEq.), also FASTEN): FASTEN under its default settings. 5) The FlowNetFace is frozen when finetuning the network (Frozen FlowNetFace). Table 4 reports the intra-dataset evaluation on HiFiMask. Similar to most existing DLMs, only considering spatial information neglects the subtle transitional features and unnatural prosthetic traces, resulting in higher HTER ( $3.11\%$ ). When only temporal information is considered, the trained model takes no notice of the splicing traces and unnatural gloss on the masks when the subject is closing their eyes or opening their mouth. Therefore, detection performance drops dramatically (with  $7.47\%$  HTER). When all frames are weighted equally, assigning lower weights to frames with obvious actions (e.g., eye-blinking) also leads to performance abatement. This also indicates that FASTEN performs better than simply concatenating temporal and spatial features.
The performance gap between the first three scenarios and FASTEN demonstrates that fully and reasonably utilizing spatial and temporal features is beneficial for improving detection accuracy. Intuitively, the network might perform better if FlowNetFace is frozen when the network is finetuned, as the optical flow seems to remain fixed when the network is learning the distinguishment between 3D masks and real faces. The experimental results in the last scenario nevertheless show that slightly adjusting FlowNetFace with a minuscule learning rate (2e-5) can improve the detection performance. We draw a conclusion that the joint finetuning of the optical flow model and detection model is instrumental in improving the supervision of temporal features over spatial features, thereby providing more adaptive spatial information for detection.
Possible variants of FASTEN. Instead of fixing the input frame number  $\lambda$  as 5, the number could be varied. We conduct an ablation study on HiFiMask under different input frame numbers, and the results are also reported in Table 4. Since the total frame number of HiFiMask is 10, we set  $\lambda$  as 2, 5, 8 and 10. We follow the same frame selection strategy as FASTEN, where the  $\lambda$  frames are randomly selected and sorted in sequential order. When  $\lambda = 2$ , the lack of optical flow information results in
Table 4: Ablation study of FASTEN on HiFiMask.
Component	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01	λ	HTER (%)	EER (%)	AUC	B@A =0.1	B@A =0.01
Spat.	3.11	3.06	99.64	0.58	7.28	2	3.73	3.93	99.55	0.62	8.48
Temp.	7.47	7.51	98.15	5.56	24.8	5	2.11	2.18	99.85	0.13	4.13
Spat. + Temp.(Eq.)	2.91	2.89	99.67	0.52	6.76	8	2.78	2.76	99.73	0.52	7.34
Spat. + Temp.(UnEq.)	2.11	2.18	99.85	0.13	4.13	10	2.90	3.01	99.55	0.71	7.25
Frozen FlowNetFace	2.76	2.80	99.66	0.65	6.30
Figure 3: Heatmap of frame weights.
Figure 4: Diagram of eye and mouth state score calculation.
Figure 5: Screenshot of a 3D mask attack and a real face on mobile deployment.
the temporal information not being able to effectively guide spatial features. Hence, the AUC (99.55) is lower than when only spatial features are considered (99.64), and the HTER is  $19.9\%$  higher. When  $\lambda$  rises to 8 or 10, although more temporal information brings about marginal improvement compared with  $\lambda = 2$ , processing more pairs of optical flows requires a larger computational budget and longer inference time. When there are no strict requirements for accuracy or inference time, variants of FASTEN with different  $\lambda$  values are also practicable.
Contribution of flow attention. To further demonstrate the effectiveness of flow attention, we plot the relationship between facial relative state scores and frame weights as a heatmap in Figure 3. Given  $n$  videos, we first get the eye/mouth state score  $s_{ij}$  of the  $j$ -th frame in the  $i$ -th video by dividing the height of the eye/mouth by their corresponding width (as shown in Figure 4). The eye state score is averaged between both eyes. Next, we calculate the set  $S$  of relative eye/mouth state scores:
$$
\mathbf {S} = \phi \left(\bigcup_ {i = 1} ^ {n} \left\{\frac {e ^ {s _ {i j}}}{\sum_ {l = 1} ^ {\lambda} e ^ {s _ {i l}}} \right\} _ {j = 1} ^ {\lambda}\right), \tag {3}
$$
where  $\phi (\cdot)$  is a truncation-normalization function that retains the values between the  $5\% -95\%$  distribution of all  $n\times \lambda$  values and then normalizes them to [0, 1]. Larger eye/mouth relative state scores represent the relative trend of closing eyes and opening mouths. Similar to intuition, we observe that frames with larger action amplitudes tend to have higher frame weights. FASTEN pays the greatest attention to these frames with both mouth opening and eye closing (upper right corner). We also observe that the frame weight is high for frames in which subjects are about to open their mouths widely or close their eyes. Considering that blinking happens very quickly and the opening of the mouth is not always fully open, the maximum weight value is not at the maximum score in these cases (upper left corner and lower right corner). Overall, flow attention can effectively help FASTEN assign larger frame weights to frames with large movement.
4.4 Real-world deployment in mobile devices
To show the inference performance of our proposed FASTEN in resource-constrained platforms (e.g., mobile devices), we package our model into an Android application and deploy it to the mobile end in the real world. Figure 5 shows two
screenshots of mobile deployment. The test experiments are conducted on a Samsung and a Xiaomi smartphone. Table 5 shows that FASTEN only consumes 9.21MB for storage, with a required run-time memory of less than 14MB. Furthermore, FASTEN achieves an average inference time of less than  $224\mathrm{ms}$  per image. We do not use batch processing when inferring. Considering the pre-processing time for video parsing, face detection with landmark, and face alignment (no post-processing), the overall response time is about  $879~\mathrm{ms}$ . Due to the low computational cost and model
Table 5: Mobile deployment performance.
Device	Processor	GPU	Time	Storage	Memory
Samsung S8	Snapdragon 835	Adreno 540	224ms	9.21MB	13.89MB
Xiaomi 8	Snapdragon 845	Adreno 630	145ms	9.21MB	10.02MB
parameters, quantization acceleration is unnecessary when deploying on mobile devices, so we can practically use it with almost no loss of accuracy. The overall system achieves a defense accuracy of  $99.3\%$  against 3D masks, indicating that our proposed framework can realize practical 3D mask PAD.
5 Discussion on research focus
Detecting 3D masks has become a rising challenge for most existing PAD methods owing to the rapid development and maturity of 3D printing [22]. Realistic 3D masks are hard to differentiate even for human eyes. To date, effective defenses against 3D masks remain a missing piece. Therefore, our method focuses on this type of attack. As a deep learning method by extracting features, we believe that our method can also be adapted to other attack types, e.g., printing, replay, since their features are relatively simpler. Moreover, in actual face anti-spoofing security applications, it is usually an aggregation of multiple defense models. Other attacks can be mitigated separately using the corresponding tactics. Deploying our defense will enhance the overall security of the system.
This paper innovatively proposes a 3D mask detection framework, FASTEN, which addresses the generalizability deficiency of HFMs and DLMs and the high computational complexity of rPPGMs. Mainly including a facial optical flow network, FlowNetFace, a flow attention module, and a spatio-aggregation process, FASTEN fully utilizes spatial and temporal information to assign different frame weights to each frame. In addition, FASTEN only needs any five frames in sequential order from the RGB video as input, which can reduce computational overhead. FASTEN shows excellent superiority and generalizability compared with eight competitors in both intra-dataset and cross-dataset evaluations. Furthermore, we achieve real-world practical detection for 3D masks by releasing an application to mobile devices, which also illustrates the landing feasibility and high potential of our proposed method for practical usage.
Limitation and future work. The aforementioned contribution of flow attention has demonstrated that focusing on frames with more obvious movement details works well for detection. Actually, the distinguishability of different regions within each frame also varies. In future work, we will be dedicated to exploring inter- and intra-frame attention to further improve detection accuracy.
Our proposed defense alleviates 3D mask spoofing attacks against face recognition systems (FRS). Such defense would be beneficial in minimizing the security concerns associated with FRS in the financial and public service sectors. However, we would also like to extend our considerations to privacy and fairness issues of the defense and its serving FRS. 1) Privacy. Our defense applies to the specific task of spoofing detection in which facial information is presented to the associated FRS as credentials. To ensure that no private information is memorized, our model is trained on public datasets and does not use any private photos for training purposes. 2) Fairness. In our work, we use HiFiMask as one of the main datasets. HiFiMask provides an equal number of subjects (25 each) for each ethnicity to facilitate fair artificial intelligence and mitigate bias [29]. Our training method also encourages the fairness of the trained defense. However, considering that defense is an independent component in the FRS, it cannot improve the fairness of the overall FRS. Moreover, there might be fairness issues for newborns or people with certain skin diseases, since they do not have enough representation in the training dataset. Another possibility would be that the dataset itself contains unbalanced data or faces with even one single skin tone. If this happens, incremental training after collecting obfuscated/synthetic data of the corresponding group of people can be used to mitigate the fairness issue. We also encourage future research in this area to incorporate fairness training techniques to mitigate possible issues.
This work was supported in part by facilities of Ping An Technology (Shenzhen) Co., Ltd., China and CSIRO's Data61, Australia. Minhui Xue and Derui Wang are the corresponding authors.
[41] H. Wu, D. Zeng, Y. Hu, H. Shi, and T. Mei, "Dual spoof disentanglement generation for face anti-spoofing with depth uncertainty learning," IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[42] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.” Optics express, vol. 18, no. 10, pp. 10762–10774, 2010.
[43] G. De Haan and V. Jeanne, "Robust pulse rate from chrominance-based rppg," IEEE Transactions on Biomedical Engineering, vol. 60, no. 10, pp. 2878-2886, 2013.
[44] S. Liu, P. C. Yuen, S. Zhang, and G. Zhao, "3d mask face anti-spoofing with remote photoplethysmography," in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14. Springer, 2016, pp. 85-100.
[45] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al., “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324.
[46] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, "Flownet: Learning optical flow with convolutional networks," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2758-2766.
[47] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2462-2470.
[48] S.-Q. Liu, X. Lan, and P. C. Yuen, "Remote photoplethysmography correspondence feature for 3d mask face presentation attack detection," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 558-573.
[49] C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948.
[50] S. Umeyama, "Least-squares estimation of transformation parameters between two point patterns," IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 13, no. 04, pp. 376-380, 1991.
[51] T.-T. Wong, "Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation," Pattern Recognition, vol. 48, no. 9, pp. 2839-2846, 2015.