Flow-Attention-based Spatio-Temporal Aggregation Network for 3D Mask Detection
Authors: Yuxin Cao
Yian Li
Yumeng Zhu
Derui Wang
Minhui Xue
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D mask detection
spatio-temporal aggregation
optical flow
deep learning
TL;DR: We propose a new 3D mask detection method by considering flow attention and fully utilizing spatial and temporal information.

Abstract: Anti-spoofing detection has become a necessity for face recognition systems due to the security threat posed by spoofing attacks. Despite great success in traditional attacks, most deep-learning-based methods perform poorly in 3D masks, which can highly simulate real faces in appearance and structure, suffering generalizability insufficiency while focusing only on the spatial domain with single frame input. This has been mitigated by the recent introduction of a biomedical technology called rPPG (remote photoplethysmography). However, rPPG-based methods are sensitive to noisy interference and require at least one second (> 25 frames) of observation time, which induces high computational overhead. To address these challenges, we propose a novel 3D mask detection framework, called FASTEN (Flow-Attention-based Spatio-Temporal aggrEgation Network). We tailor the network for focusing more on fine-grained details in large movements, which can eliminate redundant spatio-temporal feature interference and quickly capture splicing traces of 3D masks in fewer frames. Our proposed network contains three key modules: 1) a facial optical flow network to obtain non-RGB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation to aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms eight competitors for both intra-dataset and cross-dataset evaluations in terms of multiple detection metrics. Moreover, FASTEN has been deployed in real-world mobile devices for practical 3D mask detection.

Flow-Attention-basee Spatio-Temporal A Aggregation Network for 3D Mask Detection
Yuxin Cao* Tsinghua University China
Yian Li ShanghaiTech University China
Yumeng Zhu Ping An Technology (Shenzhen) Co., Ltd. China
Derui Wang CSIRO's Data6l
Minhui Xue CSIRO's Data61
Australia
Australia
Abstract
Anti-spoofing detection has become a necessity for face recognition systems due the security threat posed by spoofing attacks. Despite great success in traditional attacks, most geep-learning-bsee methods perform poorly in 3D masks, which can highly simulate real faces in appearance and structure, suffering generalizability insufficiency while focusing only on the spatial domain with single frame input. This has been mitigated by the recent introduction of a biomedical technology called rPPG (remote photoplethysmographhe sensitive to noisy interference and require at least one second (> 25 frames) observation time. which induces high computational overhead. To address these challenges, we propose a novel 3D mask detection framework, called FASTEN (Flow-Attention-basse
However, rPPG-based methods are
Tempoo-e aggrEgation Network). We tailor the network for focusing more on fine-gra grained details large movements, which eliminate redundant spatio-tempora feature interference and quickly capture splicing traces 3D masks in fewer frames. Our proposed network contains three key modules: facial optical flow network obtain non-RGBB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms eight competitors for both intra-dataset and cross- dataset evaluations in terms of multiple detection metrics. Moreover, FASTEN has been deployed in real-world mobile devices for practical 3D mask detection.
Introduction
Face recognition has been deeply involved various prevalent applications, such as access control systems and e-commerce (1. Nevertheless, similar to the potential security threat in fields of images
3 and streaming videos 4 [], face recognition systems are encountered with security issues mainly caused by face presentation attacks in the forms of printed photos, replayed videos, 3D masks 66 77. Since face images are easily accessible on social networks, one can prosspe spoofing faces at low acquisition and manufacturing costs. To mitigate the detriment caused by face presentation attacks, a plethora of face presentation attack detection (PAD) methods have sprung up in the last decade [8-18). Despite the acceptable detection performance against printings/screens, 3D masks can undermine these PAD methods by a large margin due to advanced 3D printing
"This work done while Yuxin and Yian worked as inters Ping Technology (Shenzhen) Co., Ltd.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

technologies 19 20 3D masks made up of soft silicone are vivid enough, in terms of color, texture, and geometry structure, that even humans find it confusing to differentiate. This has gradually made 3D mask defense from a branch of face anti-spoofing to a relatively independent sub-topic [19 21 22
Existing 3D mask PAD methods
be categorized into three branches: Handcrafted- Features- based methods (HFMs) [9H2 (3). Deep-Learning-bare based methods (DLMs) [15H88] and remote- coooopllthyysmorrppyy
methods (rPPGMs) [21] (4/228). In earlier years, 3D masks were mainly distinguished by manually annotated features. In the last decade, many works have greatly improved the performance of spoofing detection through the powerful advantages of deep learning. However, HFMs and DLMs do not fully consider temporal differences, resulting in average effective- ness in cross-dataset evaluation. There has been a new breakthrough with the emergence of rPPG technology. which captures the color variations caused by heartbeat when the face skin is illuminated by light. rPPGMs achieve splendid performance on cross-dataset evaluation, but rPPG signals are touchy about noise interference [28]- The release of the HiFiMask dataset [29] also challenges
rPPGMs since the light frequency they used similar to that of the heartbeat, leading to loss of utility of rPPG signals. We summarize the limitations of these methods as follows. 1) Data availability. Most methods obtain data from devices that are difficult to deploy in various practical scenarios, e.g., depth information captured by RGBD cameras (30,, rPPG signals obtained by light [27]. 2) Temporal information involvement. Few works focus on temporal features, which can carry variations of facial details. 3) Computational cost. The shortest observation time required for rPPGMs is one second (> 25 frames), which is still long enough to trigger a high computational cost.
To address the limitations above, we propose FASTEN (Flow Attention-based Spatio-Tempone aggrEgation Network), which only needs five frames of the input RGB video (no depth, rPPG, or other information is needed) to distinguish between 3D masks and real faces. FASTEN fully utilizes temporal and spatial information by focusing more on fine-grained details in large movements, which can avoid redundant feature interference and lower the computational cost. We consider frame-wise attention to aggregate the temporal and spatial feature information of multiple frames, rather than simple concatenation, since the latter practice neglects the frame significance discrepancy, especially when the input frames include large movements, e.g., eye blinking and mouth opening. Concretely, we first train a facial optical flow network, named FlowNetFFace, to obtain RGB-free inter-frame flow information. The frame weights are then calculated by facial optical flow and assigned to each frame as their attention scores. Finally, the high-level spatial features and temporal transition features aggregated for detection. In summary, the contributions of this paper are listed as follows.
We propose a dow-attention-baxe for 3D mask PAD. FASTEN outperforms existing works in intra-/cross-dataen evaluations.
aggregation network, called FASTEN,
In order to thoroughly take advantage of the temporal information of spatial features, we train a facial optical flow model, FlowNetFace, to learn the small variations in facial features between frames. The facial features are applied simultancously to the calculation of frame weights and temporal transitional features.
We address the long- time issue that the state-of wofftheeart rPPGM still needs about second (> 25 frames) for observation, and propose a frame-based method that only needs any five frames of a 3D mask video, by adding frame weights to multi-frame spatial features. We also release a lightweight application for FASTEN to mobile devices, showing that FASTEN can achieve practical 3D mask PAD on resource-constrained platforms.
2 Related work
Ceaedrrrated-FFFtrre------ Features- Methods. In early studies, due to the conspicuous artifacts in pre- sentation attacks such printings and screens, facial characteristies and handcrafted descriptors. such as LBP BI 32), HOG [8] and SIFT [3), were utilized to distinguish spooting images. These characteristics, however, are not adaptable to 3D masks, since masks do not incorporate face quality defects such as Moiré patterns [34] or reflection [35]. Analyzing color texture 9 in chroma images is also a common kind of feature to detect color reproduction artifacts in masks. From the perspective of
Handcrafted
dynamic features, facial cues such as eye-blinking [00 (23] and face movement [2 are commonly
used discriminate printings and screens, but are not congenial to 3D masks
36
heep-leerning--asee Methods. In recent years, a plethora of DLMS 15
37 39
have burst out.
CDCN++ [15] uses Central Difference Convolution (CDC) to Iearn central gradient information after

---

Trappedl Trmosiion mna Seeteer
Shars Neaka
Spuraal Footeer
Podistal
Myg M/
Myp Min
Clasiike
Spatis-Temporal Aggregation
Spatial
Extractor
Attention
Figure 1: Overview of our proposed FASTEN.
which a multi-scale attention module is added fuse multi-scale features. Due its practicality and popularity, some follow-up work [40 41 is based on CDCN++. HRFP 37 separates faces into several parts to extract fineggrained information. came out first in 3D high-fidelity mask attack detection challenge held by the International Conference on Computer Vision (ICCV) Workshop in 2021. Temporal Transformer Network (TTN) (30] has recently been proposed by learning subtle temporal differences from depth maps captured by RGBD cameras. The depth maps of all kinds of spoofing images are set as during training. However, 3D masks actually contain depth information and cannot be ignored, leading to mediocre detection ability towards 3D masks. Besides, depth information is hard to obtain in practical scenarios. Thanks to the weak generalizability of deep learning networks, DLMs find it tough to detect unseen 3D masks.
PPGGbased Methods. Emerging recent years, rPPG achieves non-contact monitoring of human heart activity by using RGB cameras to detect subtle color variations in facial skin caused by heartbeat under room light 42
43 Since light transmission is blocked 3D masks, the heartbeat can only be As result, rPPGMs achieve much success when distinguishing 3D masks 221, 28) Among advanced work, MCCFTPPG (28 improves CFrPPG (44) by extracting temporal variation from rPPG signals through multi-channel time-frequency analysis scheme to mitigate interference of noise on rPPG signals. To shorten the required observation time from 10 seconds within one second, LeTSPPPG [21] evaluates the temporal similarity of the rPPG features based spatio-temporal convolution network to distinguish
detected from real faces 44.
and proposes a self-supervisee-leaate 3D masks in terms of the consistency of the rPPG signals. However, one-second vidco still includes many frames (> 25) and requires a significant amount of computation. The HiFiMask dataset [29] also challenges rPPGMs since the light frequency they used is similar to that of heartbeat, leading to loss of distinguishment utility of rPPG signals.
3 Methods
Given video = [x1, F2; mask, our goal is to build discriminative network that can accurately distinguish 3D masks from real faces, where À denotes the required input frame number (5 in case). Different from the existing 3D mask detection method, FASTEN relaxes the restrictions on the input, i.e., we only need any frames in sequential order from the video x [Ia Teg "e,, Ia S ai < <a0 < < ax S T. where the selected adjacent frames can be either equally spaced or unequally spaced. The framework of FASTEN depicted in Figure T In our design, three key components are integrated into the backbone network: 1) We first employ an optical flow model FlowNetFace to generate facial optical flow between adjacent frames. 2) The flow attention module is used calculate the frame weights over time and assign different attention to each frame. 3) Finally, the temporal transition features are extracted from FlowNetFFace and aggregated with the high-level spatial features for detection.
1 "2.;...y77 TT) containing T (T A) frames of a real face a face wearing a 3D
3.1 Feature extractor
Spatial features capture facial details, especially information on facial texture, which conducive to 3D mask detection. We use 0.75x MobileNerV3-Small [45), which contains four blocks, as

---

backbone to balance accuracy and efficiency. Considering that it is rather computationally expensive if low-level spatial features F [F.. Fazs.., F.. are in high resolution, the low-level spatial features of À frames are extracted from the first two stages of MobileNet with shared network weights. Also, the receptive field is 19 after the first two blocks. Corresponding to our input face region with a resolution of 96 x 112, it is just right at this stage to extract the features of specific part (e.g,, eyes or corners of mouth), hence the fusion of multi-framee spatial features can be better aligned.
Temporal features encode the motion patterns of facial muscles, which is beneficial to distinguish fake faces bearing a strong resemblance to real ones in appearance. We choose facial optical flow maps to represent temporal features based on the following insights: 1) Unlike rPPG, optical flow, which embeds the motion speed and direction of each pixel. can preserve the facial structure while partially neglecting the interference caused by color diversity. This helps to withstand multiple challenges of robustness such as acquisition equipment and facial skin color. 2) By comparing the optical flow of every two adjacent frames in clip, we can identify moments with larger movement and more exposed traces, guiding the network to allocate attention reasonably and thereby reducing the demand for frame numbers.
3.2 Facial optical flow network
Since there are no existing optical flow models specifically designed for faces, we propose an active- learning-based alternative strategy to train facial optical flow prediction network ourselves. As one of the classic methods in the field of optical flow, FlowNet [46] generally contains an encoder/decoder structure and works well on the proposed generic optical flow dataset named Flying Chairs. The variants of FlowNet include FlowNetS, FlowNetC, FlowNet2.0,, etc, whose performance improves gradually as well as computational complexity. Unlike the Flying Chairs dataset, which composed of high-resolution images with multiple targets, face images are rather simple. Therefore, we build a facial optical flow model evolved from FlowNetS by reducing the layer number to both the encoder and the decoder. In this case, our optical flow model, namely FlowNetFace, can obtain the temporal features required the lowest computational cost. In addition, we use FlowNet2.0 [47), a stacked deep-learning- based optical flow estimator, to generate the ground truth of the facial optical flow. Following prior work [46. 47], the average endpoint error (AEPE) is used to measure the distance between the predicted and ground truth optical flow vectors.
3.3
Flow attention
The most significant difference between 3D masks and real faces basically the splicing traces on the face. Especially, when a person is asked to blink the eyes or open the mouth, the unnatural details are particularly noticeable, e.g., compared to static faces, the detailed features around the eyes when eyes are closed are more helpful in distinguishing masks from real faces. As aforementioned, mask- splicing clues change with facial expressions. Therefore, scoring spatial features through temporal information can make the network autonomously attend to significant frames, thereby amplifying the features of facial details under special actions as well as weakening interfering information. Concretely, we input optical flow of the selected À frames, sequenced by a 3x33 convolution layer to calculate the frame weights W [Way Wag .... Way] for the spatial features of each frame. Then, the frame weights are fused to the spatial features F, and the output fused feature can be expressed as
F Fruse
Σ Way Par. S.L.
Wa: = 1.
3.4 Spatio-temporal aggregation
As for spatial information, the fused feature Fruse is then fed into the latter two blocks of the backbone to obtain the spatial feature Fapa-- It is apparent that there exist smooth facial fluctuations and muscle movements real faces when a video captured, while no such phenomena are presented in 3D masks. Thus, we additionally consider extracting the temporal transitional feature Fremp that can represent the temporal differences between consecutive frames. Since the resolution of the optical flow features is eight times that of F Espa input them into the module composed of the latter three blocks of MobileNet, resulting in temporal features of the same size as deep-level spatial features. Since the spatial feature can reflect the splicing traces in the input face image, and the temporal transitional feature brings the transitional movement cues, we aggregate both the spatial and temporal features by concatenation and then input the aggregated into the classifier which contains fully

---

connected layer followed by Softmax layer. Due to the data imbalance between real faces and 3D masks, we use the weighted BCE loss to train the network. Given batch of N face images, we have
N I py log p. - (1 - 21) log (1 - ps]l.
where y/ denotes the true class of the i-th image, with standing for real faces, and O representing 3D masks. pi denotes the output score of the classifier. j denotes the balancing weight.
3.5 Training strategy
Considering that the proposed FASTEN network includes several modules, the training strategy is given as follows. 1) We first separately train the facial optical flow network FlowNetFace using the face dataset and the ground truth flow obtained by FlowNet2.0f Since optical flow prediction is independent of detection, training FlowNetFace alone can not only converge the network as soon as possible but also avoid conflicts caused by two different tasks. 2) Then we use the MobileNetV3- Small model pre-trained trained on ImageNet finetune the entire network on 3D mask datasets, with a smaller learning rate for FlowNetFace. Please refer to our code for more details +
our backbone. We finally use small learning rate to
4 Experiments
4.1 Experimental setup
Datasets. We conduct our experiments on three frequently used datasets, 3DMAD (3D Mask Attack
Dataset) [19], HKBU-MARs V1+ (Hong Kong Baptist University 3D Mask Attack with Real-World
Variations) 488 and HiFiMask Ffigh-Fideliity Mask dataset) [29]. 3DMAD [9] contains 255 videos of 17 subjects wearing custom-built Thatsmyface masks where the eyes and nose holes are cut out to present more natural wearing effects. The videos are recorded by a Kinetic camera at 640 x 480 resolution, 30 fps. Both color and depth data are collected. 2) HKBU-MARs V1+ (88 includes 180 videos of 12 subjects with two mask types, Thatsmyface and REAL-f. Videos in HKBU-MARs VI+ are captured by Logitech C920 under room light. 3) HiFiMask
29 is a large, high-resolution mask dataset containing 54,600 videos from 75 subjects wearing three types of masks: plaster masks, resin masks, and transparent masks. HiFiMask considers six different scenes, including light variation and head motion. The frequency of the periodic flashlight is set within [0.7, 4] Hz, which is consistent with that of ordinary people's 's heartbeat, to great extent challenging rPPGMs since pseudo 'liveness' signals can be obtained for masks. More details of the datasets are shown in Table I HiFiMask is the most challenging dataset for PAD methods, owing to its large quantity and high fidelity.
Competitors. We select eight existing detection methods for spoofing images or specifically for 3D masks from HFMs, DLMs, and rPPGMs as our competitors, i.e., MS-LBP (32), CTA (9)
CDCN+ 13), HRFP (37), ViTranZFAS (38], MD-FAS (39), CF+PPG [44], and TeTS [21].
In the early ages, MS-LBP 32 and CTA 9. as two classic feature descriptors. exhibit edges in finding quality defects in terms of color and texture. Although most of the DLMS are not tailored for 3D masks, we select CDCN+ [15), one of the most popular and frequently used deep-learning networks, to represent this branch of detection methods, since the fine-graine features of spoofing faces learned by central difference convolution and multi-scale attention module are expected to work for 3D masks. HRFP B7 was crowned with laurel in the 3D high-fidelity mask attack detection challenge held by ICCV in 2021. By segmenting faces into several parts for feature extraction, HRFP rejects the input face once an anomaly exists in any part of the input. Two rPPGMs, CFrPPG 44 and LSTSPPPG (1I] also included for comparison due to their strong performance in identifying 3D masks based on biotechnology. We further consider two DLMs using pre-trained models due to their outstanding performance in face anti-spoofing tasks. ViTranZFAS (38 used transfer learning from a pre-trained vision transformer (ViT) to distinguish spoofing images. MD-FAS (39] was later proposed improve domain adaptation by only using target domain data to update the pre-trained face anti- anti-spoofing model. For all the competitors, we follow their original settings.
5

---

Table I: Concrete details of different datasets we used. The resolution of HiFiMask varies.
Damnet
Year *Subyect PMast Mast type Scenes Light Camen FLuv videos
#Masiiee videos
Fioul videos
Reselution
Facial resolition
3DMAD
2013
17
17
170
85
255
640
450
80
80
HXBBU MARS
VI+
83
2018
12
12
120
60 400950
180
1280
720
200 200
HiRiMaok
2031
75
75
13,590
$4.000
High Ribiliy
400 400
Statement for rPPGMs. Due to the lack of open-source codes for rPPGMs, we are unable to obtain rPPG signals to replicate their method. Thus, we cite the results reported in their paper on 3DMAD and HKBU-MARs VI+. We do not involve one recent work MCCFiPPG (28) since the observation time is too long (10~12 seconds) to be fairly compared with our method, which only needs five frames. Instead, we select its previous version CFrPPG 44 since the detection results with an observation time of one second available in the paper of LeTSPPPG (21) Note that HiFiMask only contains 10 frames sampled from each video at equal intervals, the sampling frequency of which is much lower than twice the highest light frequency (4Hz for periodic flashlight). According to the Myquist-Shannod Sampling Theorem [49]. recovering rPPG analog signals will inevitably lead to frequency aliasing and signal distortion. Ergo, we do not consider rPPGMs HiFiMask.
Metrics. Following prior work [5]
28 37,, we use five metrics to evaluate the detection performance, i.e., Half Total Error Rate (HTER), Equal Error Rate (ERR), Area Under Curve (AUC), Attack Presentation Classification Error Rate (APCER), and Bonafide Presentation Classification Error Rate (BPCER). We use "B
to represent the BPCER when APCER 1/0.00..
Parameter settings. the training process, we adopt commonly utilized augmentations including Gaussian noise, random brightness contrast, and cut-out. FlowNetFace is trained (finetuned) using AdamW optimizer with a learning rate of le-4 (2e-5) and weight decay as 4e-4 (1e-2 for 3DMAD and HKBU-MARs V1+, and 1e-3 for HiFiMask). The rest of the network is finetuned using the same optimizer with a learning rate of 2e-4, which is adjusted by cosine annealing with warm restarts. We set the warm-up epoch number as 2 and the total epoch number as 150. We set the required input frame number = 5, the balaneing weight μ = 0.5 for 3DMAD and HKBU-MARs V1+ and μ = for HiFiMask. The batch size is 120, and all models are trained using two Tesla V100 GPUs. We perform face alignment (50) on images before inputting them into the network, so that spatial features can be fused without misalignment.
4.2 Experimental results
Intra-dataset evaluation. For the intra-dataset evaluation, we adopt leave one out vols-vvaii (LOOCV) protocol on 3DMAD and HKBU-MARs VI+. Following (28). test subject is randomly selected first for both two datasets. Then we randomly select eight/five subjects for training and the leftover eight/six subjects for validation on 3DMAD/HKBB--ARS VI+. LOOCV is not suitable for large datasets such as HiFiMask, since LOOCV is subject to high variance or overfitting [51. Instead, we use k-fold cross validation (KCV) when training on HiFiMask. In this case, we follow Protocol 1 [29]. where 45, 6 and 24 subjects are selected for training, validation and testing, respectively.
The intra-dataset evaluation results reported in Table 2 Ssow that FASTEN performs better than the other eight competitors, with AUCs of 99.8%, 99.7%, and 99.8%, and HTERs of 1.17%, 2. and 2. 2.11% on 3DMAD, HKBU-MARs VI+ and HiFiMask, respectively. We attribute the outperformance to FASTEN's strong ability to distinguish 3D masks by assigning different weights to each frame and aggregating both spatial and temporal information. MS-LBP ranks second on 3DMAD since handcrafted features perform well under fixed lighting conditions. However, the performance of MS-LBP limited in complex lighting and different mask types. HRFP comes out second and third in both HKBU-MARs V1+ and HiFiMask since the fine-grained features of face parts at high resolution in HRFP help identify subtle differences in faces. CDCN++ also performs well owing to the multi-scale attention which effectively aggregate multi-level features of central difference convolution. We do not see significant advantages for rPPGMs due to noise interference by various lighting conditions and camera settings. As previously discussed, rPPGMs are challenged by HiFiMask, which may induce a loss of utility.
Cross-dataset evaluation. We also carry out cross- coos--dtaset evaluation to verify the generalizability of FASTEN when test scenarios have never been seen before. Models are trained on one dataset and then tested on the test set of another dataset. Table 3 reports the gross-dataset evaluation results. We first consider the scenarios between two small datasets, 3DMAD and HKBU-MARs V1+. The performance of HFMs drops dramatically due to the limitations of simple color and texture features.

---

Table 2: Intra-dataset evaluation on different datasets. 'U'; unavailable. 'N/A': not applicable.
3DMAD
HKBU-MARs VI+
HiFiMask (Protocol I)
Method
HTER
EER
B@
B@A
HTER
EER
B@A
B@
HTER
EER
B@A
B&A A
AUC
AUC
AUC
(%)
(%)
=0.1
=0.01
(%)
(%)
=0.1
=0.01
(%)
(%)
=0.1
=0.01
MS-LBP 82
1.9243.4
3.28
99.4
0.32
6.78
24.0=22.6
22.5
85.8
48.6
95.1
48.3
40.5
63.7
78.9
97,3
CTA
4.40:19.7
4.24
99.3
1.60
11.8
23.4+20.5
23.0
82.3
53.8
89.2
40.7
31.6
74,9
64.1
92.9
CDCN+
15
4.20+17.1
8.34
96.7
6.62
59.6
4.83+ 7.6
8.70
96.0
7.77
66.2
3.67
2.64
99.6
0.83
6.79
HRFP (37
2.34:5.2
2,41
99.7
0.67
5.87
3.34
5.7
4.33
99.2
1.34
6.23
2.20
2.26
99.7
0.28
4.35
ViTraZFAAS
X8
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
2.63
2.48
99.7
0.21
6.58
MD-FAS
39
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
5.45
4.12
99.4
0.34
15.2
CRiPGG (Is)
28
32.747.4
32.5
70.8
U
U
42.1
5.6
42.0
60.8
U
U
N/A
NIA
N/A
N/A
NIA
LeTS:PG
21
11.8+8.6
11.9
94.4
6.5
15.7
91.5
N/A
N/A
FASTE
1,17-0.7
1.06
99.8
0.03
0.31
2.13:5.11
2.56
99.7
0.81
2.83
2.11
2.18
99.8
0.13
4.13
Table 3: cross-dttaset evaluation. UU: unavailable. 'N/A': not applicable.
3DMAD
HKBU-MARS VI+
HKBU-MARs
V1+
3DMAD
HiFiMask
3DMAD
HillMask
HKBU-MAR VI+
Method
HITER
(9)
EER
(5)
AUC
BEA
=0.1
=0.01
B@A
IITER (8)
EER
(5)
AUC
B6A A
=0.1
=0.01
B&A
IITER
(9)
EER
(%)
AUC
BBA
=0.1
=0.01
BBA
HTTR
(36
EER
(%)
AUCC
BOA
=0.1
B@A
=0.01
MS-LBP
47.1+7.
48.3
52.4
97.6
43.247.3
43.7
58.8
99.2
47.8
41.3
62.6
76.3
08.0
57.0
93.0
CTA
5155=24
55.3
48.9
90.5
98.8
08.2=7.3
65.4
20.1
94.7
99.9
50.1
71.5
32.2
97.6
99.8
49.5
63.3
42.9
90.0
94.5
CDCN++
50.3:2277
55.3
42.3
93.2
99.9
4L.1168
33.1
66.2
74.1
98.9
36.8
37.7
72.1
65.9
87.1
35.7
29.7
50.
52.9
78.4
HRFP
25.813.1
32.3
69.1
81,3
98.4
6.76.0.9
7.25
98.5
5.63
22.0
21.3
12.4
95.8
14.6
32.5
7.68
8,31
99.1
1.52
11.4
ViTRIZFAS
N/
N/A
N/A
N/A
34.2
MD-FAS
N/A
N/A
N/A
NIA
NIA
M/A
NA
NIA
N/A
N/A
33.5
34.4
69.9
67.0
75.3
8.61
974
567
43.4
CRIPG (1s)
39.211.4
40.1
63.6
75.5
U
40.112.3
40.6
62.3
79.1
U
NAA
NI
N/A
N/A
N/A
NIA
N/A
NAA
NIA
NIA
LeTSiPPG
15.7:0.5
16.5
90,1
25.2
U
12.9108
13.1
93,4
15.8
0
N/A
NAA
NAA
N/A
N/A
NIA
N/A
NIA
NIA
NIA
BASTEN(Ous))
11.8.2.1
16.8
91.0
18.8
24.4
1,85
4.35
99.1
3,00
8,88
2.35
2.71
99.8
0.27
4.76
7.38
8.55
98.3
5.40
17.7
Despite the strong performance in intra-dataset, DLMs exhibit significant decline in performance, e.g., the AUC for CDCN++ on 3DMAD drops from 96.7 to 42.3 when testing HKBU-MARs VI+. Such results are attributed to the fact that DLMs lack generalization when encountered with unseen inputs. Another major reason why FASTEN outperforms these methods in cross- pross-dataser evaluation could be the aggregation between spatial and temporal features, the latter of which carries large amount of interframe information and can better capture the flaws in splicing traces. Although rPPGMs have been verified to have strong generalization ability, their accuracy is easily affected by noise and light frequency, resulting in slightly lower performance than FASTEN.
Due to the large gap between the two small datasets and HiFiMask in data quantity, we discard the evaluation from small datasets HiFiMask. Conversely, we conduct cross-dataset evaluation when the model is trained on HiFiMask, and test on 3DMAD and HKBU-MARs VI+. FASTEN achieves over 98% AUCs with HTER below 8% in both scenarios. As for rPPGMs, they cannot be implemented on HiFiMask due to the sampling problem. Since the light frequency is similar to that of heartbeat, the rPPG signals of 3D masks contain pseudo 'liveness' cues in both time and frequency domains, misguiding rPPGMs to incorrectly classify 3D masks as real faces. Therefore, we point out that their performance could be very limited, even if the proposer of HiFiMask (29) provided complete videos before sampling and the recovered rPPG signals. Overall, the outperformance of FASTEN shows the generalizability of our proposed framework on both small and large datasets.
Discussion of ViTranZFAS and MD-FAS. ViTranZFAS requires large amount of data for training, while MD-FAS focuses more on domain adaptation, whose experimental setting is slightly different from 3D mask defense. Therefore, we only conduct experiments on HiFiMask. We attribute the poor performance of ViTranZFAS to three main reasons: 1) Lack of temporal information: VIT focuses primarily on analyzing individual images without considering temporal cues, but 3D mask detection often requires understanding the dynamic changes over time. 2) Small regions of interest: Liveness cues often involve subtle facial regions, such as the edge of facial features or muscle contractions. However, VIT's patch-basce based approach may not focus on these small regions of interest effectively, leading to challenges in capturing fine-gne iine-grained liveness indicators. 3) Training difficulty: ViT requires a pre-trained model and high computational costs. MD-FAS aims to solve the forgetting issue when learning new domain data and improve the adaptability of the model. It cannot work well with unseen target data, as shown in Table 33, In addition, the baseline SRENet selected by this scheme is region- based fine classification task, which has high requirements for the diversity of attack categories, as well as the resolution of images. When we use HiFiMask as a training set for a fair comparison, its accuracy performance remains poor.
Overall evaluation. We calculate the average AUC, ERR, and HTER of all the PAD methods in both intra-dataset and cross-dataset evaluations to compare the overall performance. The results in
7

---

AUCST
EERs 4
HTERs
RESTEN
03-
0.5
02
0.5
HOVAS
CDCH
HO-LIR
Hin
05-
CDC
MP
Ieraddateeet
Idra-dutoset
Figure 2: Comparison of both intra and cross-dataset evaluations.
Figure 2 demonstrate that simultaneously considering temporal and spatial features helps FASTEN achieve the best detection performance and generalizability.
Computational overhead. We further compare the computational cost of different PAD methods when detecting 3D masks. Since MS-LBP and CTA only need handcrafted features for detection and rPPGMs require long videos with rPPG signal extraction, we only compare CDCN+ and HRFP with our proposed method under the same inputs. The average inference time of CDCN++, HRFP and FASTEN is 28. 38.8ms, 286.5ms and 25.3ms respectively. Compared to HRFP, FASTEN reduces computational cost and saves inference time with the benefit of smaller networks. Although CDCN+ achieves a comparable time cost with FASTEN, the lack of consideration of temporal features leads to insufficient detection performance. Compared with the best rPPGMs, SrPSS which requires one second (> 25 frames) for observation, FASTEN only needs five frames as input, which can significantly reduce computational overhead.
4.3 Ablation study
Effectiveness of components. There are two main components in FASTEN, the spatial branch and the temporal branch. To verify the effectiveness of each component, we consider the following five scenarios. 1) Only spatial features considered (Spat.): the temporal branch, including optical flow prediction, frame weight assignment, and temporal transitional feature extraction, removed. 2) Only temporal features considered (Temp.): :: the spatial feature extraction and frame weight assignment are removed. 3) Equal frame weights (Spat.
(emp.(Eq.)) the frame weight assignment is replaced by equally assigning frame weights (Vi E {1. 2....... A} War }) to each frame. 4) Different frame weights (Spat. (empp(UnEE.). also FASTEN): FASTEN under its default settings. 5) The FlowNetFace is frozen when finetuning the network (Frozen FlowNetFacc). Table $ reports the intra-dataset evaluation on HiFiMask. Similar to most existing DLMs, only considering spatial information neglects the subtle transitional features and unnatural prosthetic traces, resulting in higher HTER (1%). When only temporal information considered, the trained model takes no notice of the splicing traces and unnatural gloss on the masks when the subject is closing their eyes opening their mouth. Therefore, detection performance drops dramatically (with 7.47% HTER). When all frames weighted equally, assigning lower weights frames with obvious actions (e.g.. eye-blinking: also leads to performance abatement. This also indicates that FASTEN performs better than simply concatenating temporal and spatial features.
The performance gap between the first three scenarios and FASTEN demonstrates that fully and reasonably utilizing spatial and temporal features is beneficial for improving detection accuracy. Intuitively, the network might perform better if FlowNetFace is frozen when the network finetuned, as the optical flow seems remain fixed when the network is learning the distinguishment between 3D masks and real faces. The experimental results in the last scenario nevertheless show that slightly adjusting FlowNetFace with minuscule learning rate (2e-5) can improve the detection performance. We draw a conclusion that the joint finetuning of the optical flow model and detection model is instrumental in improving the supervision of temporal features over spatial features, thereby providing more adaptive spatial information for detection.
Possible variants of FASTEN. Instead of fixing the input frame number À as 5, the number could be varied. We conduct an ablation study on HiFiMask under different input frame numbers, and the results are also reported in Table Since the total frame number of HiFiMask is 10, we set À as 2, 5, 8 and 10. We follow the same frame selection strategy as FASTEN, where the À frames are randomly selected and sorted in sequential order. When À 2, the lack of optical flow information results in
8

---

Table 4: Ablation study of FASTEN on HiFiMask.
HTER (%)
EER (%)
B@A =0.1
B@A
HTER
EER
B@A
B@A
Component
AUC
À
AUC
=0.01
(%)
(%)
=0.1
=0.01
Spat.
3.11
3,06
99.64
0.58
7.28
2
3.73
99.55
0.62
8.48
Temp.
7.47
7.51
98.15
5.56
24.8
5
2.11
2.18
99.85
0.13
4.13
Spat. Temp.(Eq.)
2.91
2.89
99.67
0.52
6.76
8
2.78
2.76
99.73
0.52
7.34
Spat. -Temp.(UnEq))
2.11
2.18
99.85
0.13
4.13
10
2.90
3.01
99.55
0.71
7.25
Frozen FlowNetFace
2.76
2.80
99.66
0.65
6.30
035
Mouth
0.7
p25
522
Distance(-)) Distunce(--)
State Score
Houthmed ve stote soore
Figure 5: Screenshots of a 3D mask attack and real face on mobile deployment.
Figure 3: Heatmap of frame weights.
Diagram of eye and mouth state score calculation.
Figure 4:
the temporal information not being able to effectively guide spatial features. Hence, the AUC (99.55) is lower than when only spatial features are considered (99.64), and the HTER is 19.9% higher. When rises to 10, although more temporal information brings about marginal improvement compared with À = 2, processing more pairs of optical flows requires a larger computational budget and longer inference time. When there are no strict requirements for accuracy or inference time, variants of FASTEN with different À values are also practicable.
Contribution of flow attention. To further demonstrate the effectiveness of flow attention, we plot the relationship between facial relative state scores and frame weights heatmap in Figure Given videos, we first get the eye/mouth state score Sú of the j-th frame in the i-th video by dividing the height of the eye/mouth by their corresponding width (as shown in Figure The eye state score averaged between both eyes. Next, we calculate the set S of relative eye/mouth state scores:
==ó (U E (
(3)
5=1
where (6.) is a cuunationnnnomaaliiao distribution of all x À values and then normalizes them [0, 1]. Larger eye/mouth relative state scores represent the relative trend of closing eyes and opening mouths. Similar to intuition, we observe that frames with larger action amplitudes tend to have higher frame weights. FASTEN pays the greatest attention to these frames with both mouth opening and eye closing (upper right corner). We also observe that the frame weight is high for frames which subjects are about to open their mouths widely or close their eyes. Considering that blinking happens very quickly and the opening of the mouth is not always fully open, the maximum weight value is not at the maximum score in these cases (upper left corner and lower right corner). Overall, flow attention can effectively help FASTEN assign larger frame weights frames with large movement.
function that retains the values between the 5% 95% %
4.4 Real-world deployment in mobile devices
To show the inference performance of our pro- posed FASTEN in resource- constraied platforms (e.g., mobile devices), we package our model into an Android application and deploy it to the mo- bile end in the real world. Figure [ shows two screenshots of mobile deployment. The test experiments are conducted on a Samsung and Xiaomi smartphone. Table [5] shows that FASTEN only consumes 9,21MB for storage, with required run-time memory of less than 14MB. Furthermore, FASTEN achieves an average inference time of less than 224ms per image. We do not use batch processing when inferring. Considering the pre-processing time for video parsing, face detection with landmark, and face alignment (no post- processing), the overall response time is about 879 ms. Due to the low computational cost and model
Table S: Mobile deployment performance.
Device
Processor
GPU
Time
Storage
Memory
Samsung S8 Snapdragon 835 Adreno 540
224ms
9.21MB
13.99MB
Xissmm
Snapdragon 845
Adrenu 630
145ms
9.21MB
10.02MB

---

parameters, quantization acceleration is unnecessary when deploying on mobile devices, so we can practically use it with almost no loss of accuracy. The overall system achieves defense accuracy of 99.3% against 3D masks, indicating that our proposed framework can realize practical 3D mask PAD.
5 Discussion on research focus
Detecting 3D masks has become a rising challenge for most existing PAD methods owing to the rapid development and maturity of 3D printing [22]. Realistic 3D masks are hard to differentiate even for human eyes. To date, effective defenses against 3D masks remain a missing piece. Therefore, our method focuses on this type of attack. As a deep learning method by extracting features, we believe that our method can also be adapted to other attack types, e.g., printing, replay, since their features are relatively simpler. Moreover, in actual face anti-spoofing security applications, it is usually an aggregation of multiple defense models. Other attacks can be mitigated separately using the corresponding tactics. Deploying our defense will enhance the overall security of the system.
Conclusion
This paper innovatively proposes 3D mask detection framework, FASTEN, which addresses the generalizability deficiency of HFMs and DLMs and the high computational complexity of rPPGMs. Mainly including facial optical flow network, FlowNetFace, flow attention module, and spatio- aggregation process, FASTEN fully utilizes spatial and temporal information to assign different frame weights to each frame. In addition, FASTEN only needs any five frames in sequential order from the RGB video as input, which can reduce computational overhead. FASTEN shows excellent superiority and generalizability compared with eight competitors in both intra-dataset and cross- dataset evaluations. Furthermore, we achieve real-world practical detection for 3D masks by releasing an application to mobile devices, which also illustrates the landing feasibility and high potential of our proposed method for practical usage.
Limitation and future work. The aforementioned contribution of flow attention has demonstrated that focusing on frames with more obvious movement details works well for detection. Actually, the distinguishability of different regions within each frame also varies. In future work, we will be dedicated to exploring inter- and intra-frame attention to further improve detection accuracy.
7 Broader Impact
Our proposed defense alleviates 3D mask spoofing attacks against face recognition systems (FRS). Such defense would be beneficial in minimizing the security concerns associated with FRS in the financial and public service sectors. However, we would also like to extend our considerations to privacy and faireess issues of the defense and its serving FRS. 1) Privacy. Our defense applies to the specific task of spoofing detection which facial information is presented to the associated FRS as credentials. To ensure that no private information is memorized, our model is trained on public datasets and does not use any private photos for training purposes. 2) Fairness. In our work, we use HiFiMask one of the main datasets. HiFiMask provides an equal number of subjects (25 each) for each ethnicity to facilitate fair artificial intelligence and mitigate bias [29]. Our training method also encourages the fairness of the trained defense. However, considering that defense is independent component in the FRS, it cannot improve the fairness of the overall FRS. Moreover, there might be fairnsss issues for newborns or people with certain skin diseases, since they do not have enough representation in the training dataset. Another possibility would be that the dataset itself contains unbalanced data faces with even one single skin tone. If this happens, incremental training after collecting obfuscated/synthetic data of the corresponding group of people can be used to mitigate the fairness issue. We also encourage future research in this area to incorporate fairness training techniques to mitigate possible issues.
8 Acknowledgement
This work was supported in part by facilities of Ping An Technology (Shenzhen) Co., Ltd., China and CSIRO's 's Data61, Australia. Minhui Xue and Derui Wang the corresponding authors.
10

---

References
M. Wang and W. Deng, "Deep face recognition: A survey," Neuroconputtigg vol. 429, pp. 215-244, 2021.
[2] C. Szegedy, Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus. "Intriguing propertics of neural networks," in Proceedings of the International Conference on Learning Representations, 2014.
[3] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, "Black-hox adversarial attacks with limited queries and information," in Proceedings of the 35th International Conference Machine Learning. PMI.R 2018, pp. 2137-2146.
[4] S. Li, A. Neupane, S. Paul, C. Song, S. V. Krishnamurthy, A. K. Roy-Chowdhury, and A. Swami, "Stealthy adversarial perturbations against real-time vidco classification systems." in Proceedings of the Symposium Network and Distributed Systems Security (NDSS), 2019.
[5] Y. Cao, X. Xiao, R. Sun, D. Wang. M. Xue, and S. Wen, "Stylefool: Fooling video classification systems via style transfer." 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 1631-1644
[6] S. Marcel, M. Nixon, Fierrez, and N. Evans, Handbook of biometric anti-spoofing: Presentation attack detection. Springer, 2019, vol. 2.
[7] R. Ramachandra and C. Busch, "Presentation attack detection methods for face recognition systems: A comprehensive survey." ACM Computing Surveys, vol. 50, no. 1. pp. 1-37, 1-37,2017. 2017.
[8] Komulainen, A. Hadid, and M. Pietikäinen, "Context based face anti-spoofing." " Proceedings of the International Conference on Biometrics: Theory, Applications and Systems, 2013.
191 Z. Boulkenafet, J. Komulainen, and A. Hadid, "Face spoofing detection using colour texture analysis," IEEE Transactions on Information Forensics and Security, vol. 11, no. 8, pp. 1818-1830, 2016.
[10] J.-W. Li, "Eye blink detection based multiple gabor response waves." in 2008 International Conference on Machine Learning and Cybernetics, vol. 5. JEEE, 2008, pp. 2852-2850
[11] Bao, H. Li, Li, and W. Jiang, "A liveness detection method for face recognition based on optical flow field," in 2009 International Conference on Image Analysis and Signal Processing. IEEF, 2009, pp- 233-236.
[12] T. de Freitas Pereira, J. Komulainen, A. Anjos, M. De Martino, A. Hadid, M. Pietikäinen, and S. Marcel, "Face liveness detection using dynamic texture," EURASIP Journal Image and Video Processing, vol. 2014, no. 1, p. 2, 2014.
[13] D. Yi, 7. Lei, Z. Zhang, and S. 7. Li, "Face anti-spoofing: Multi-spectral approach," Handbook of Biometric Snti-Spoofing Trusted Biometrics under Spoofing Attacks, pp. 83-102, 2014.
[14] X. Song, X. Zhao, L. Fang, and T. Lin, "Discriminative representation combinations for accurate face spoofing detection," " Pattern Recognition, vol. 85, pp. 220-231,72 2019.
[15] Z. Yu, C. Zhao, Z. Wang, Y. Qin, Z. Su, Li, Zhou, and G. Zhao "Searching central difference convolutional networks for face anti-spoofing"" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. pp. 5295-5305.
[16] Z. Yu, X. Li, X. Niu, J. Shi, and G. Zhao, "Face anti-spoofing with human material perception." in Proceedings of European Conference on Computer Vision, 2020.
[17] X. Yang, W. Luo, L. Bao, Y. Gao, D. Gong, Zheng, Z. Li, and W. Liu, "Face anti-spoofing: Model matters, so does data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3507-3516.
[18] Y. Atoum, Y. Liu, A. Jourabloo, and X. Liu, "Face anti-spoofing using patch and depth-based cnns," in Proceedings of IEEE International Joint Conference on Biometrics, 2017.
[19] N. Erdogmus and Marcel, "Spoofing face recognition with 3d masks." IEEE transuctions on information forensics and security, vol. 9, no. 7, pp. 1084-1097, 2014.
[20] X. Zhang, W. Wu, Y. Huang, X. Yang, and M. Gou, "Antheraea pernyi silk fibroin bioinks for digital light processing 3d printing." International Journal of Bioprinting, vol. 9, no. 5, 2023.
11

---

[21] S.-Q. Liu, X. Lan, and P. C. Yuen, "Learning temporal similarity of remote photoplellyssogrrahy for fast 3d mask face presentation attack defection," IEEE Transactions on Information Forensics and Security, vol. 17. pp. 3195-3210, 2022.
[22] Z. Yu, Y. Qin, X. Li, C. Zhao, Lei, and G. Zhao. "Deep learning for face anti-spoofing: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[23] G. Pan. L. Sun, Z. Wu, and S. Lao, "Eyeblink- based anti-spooling face recognition from a generic webcamera," in 2007 IEEE 11th international conference on computer vision. IEEE, 2007, pp. 1-8.
[24] X. Li, J. Komulainen, G. Zhao, P.-.. Yuen, and M. Pietikäinen, "Generalized face anti-spooiing by detecting pulse from face videos." in 2016 23rd International Conference on Pattern Recognition (ICPR). TEFE, 2016, pp. 4244-4249,
[25] B. Lin. X. Li, Z. Yu, and G. Zhao, "Face liveness detection by rppg features and contextual patch-based can," in Proceedings of the 2019 3rd international conference on biometric engineering and applications, 2019, pp. 61-68.
[26] Z. Yu, Peng. X. Li. X. Hong, and Zhao, "Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with vidco enhancement," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 151-160.
[27] Z. Yu, X. Li, P. Wang, and G. Zhao, "Transrppg: Remote phoooplethysmoorrphyy transformer for 3d mask face presentation attack detection," IEEE Signal Processing Letters, vol. 28, pp. 1290-1294, 2021.
[28] S.-Q. Liu, X. Lan, and P. C. Yuen. "Multi-channel remote photoplettysmoorrphhy correspondence feature for 3d mask face presentation attack detection," IEEE Transactions Information Forensics and Security. vol. 16, pp. 2683-2696, 2021.
[29] A. Liu, C. Zhao, 7. Yu, J. Wan, A. Su, X. Liu, Z. Tan, S. Escalera, J. Xing. Y. Liang al., "Contrastive context-aware learning for 3d high-fidelity mask face presentation attack detection," IEEE Transactions on Information Forensics and Security, vol. 17, pp. 2497-2507, 2022.
[30] Z. Wang, Q. Wang, W. Deng, and G. Guo, "Learning multi-granin granulartttatty temporal characteristics for face anti-spoofing," JEEE Transactions on Information Forensics and Security, vol. 17, pp. 2254-1269, 2022.
[31] T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Marcel, "I.bp- top based countermcasure against face spoofing attacks," in Computer Vision-ACCV 2012 Workshops: ACCV 2012 International Workshops, Daejeon, Korea, November 5-6, 2012, Revised Selected Popers, Part 11. Springer, 2013, pp. 121-132.
[32] J. Määuä, A. Hadid, and M. Pietikäinen, "Face spoofing detection from single images using micro-texture analysis," in 2011 international joint conference on Biometrics (IJCB). IEEE, 2011, pp. 1-7.
[33] K. Patel, H. Han, and A. K. Jain, "Secure face unlock: Spoof detection on smartphones," IEFE transactions on information forensics and security, vol. 11, no. 10, pp.
2016.
[34] K. Patel, H. Han, A. K. Jain, and G. Out, "Live face video vs. spoof face video: Use of moiré patterns to detect replay video attacks," in 2015 International Conference on Biometrics (ICB). IEEF, 2015, pp. 98-105.
[35] Y. Liu, Y. Tai. J. Li, S. Ding, C. Wang. F. Huang, D. Li. W. Qi, and R. Ji, "Aurora guard: Real-time face anti-spoofing via light reflection," arXiv preprint arXiv: 1902. 103/1, 2019.
[36] S. Bhattacharjee, A. Mohammadi, and S. Marcel, "Spoofing deep face recognition with custom silicone masks," in 2018 IEEE 9th international conference on biometrics theory, applications and systems (BTAS). IEEF, 2018, pp. 1-7.
[37] O. Grinchuk, A. Parkin, and E. Glazistova, "3d mask presentation attack detection via high resolution face parts," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 443.
[38] A. George and Marcel, "On the effectiveness of vision transformers for zero-shot face anti-spoofing." in 2021 IEEE International Joint Conference on Biometrics (IJCB). IEEF, 2021, pp. 1-8.
[39] X. Guo, Y. Liu, A. Jain, and X. Liu, "Multi-domain learning for updating face anti-spooling models." in European Conference on Compuder Vision. Springer, 2022, pp. 230-249.
[40] Z. Yu, Y. Qin, H. Zhao, X. Li. and G. Zhao, "Dual-cross central difference network for face anti- anti-spooting." in Proceedings of the International Joint Conference on Artificial Intelligence, 2021.
12

---

[41] H. Wu, D. Zeng, Y. Hu, H. Shi, and T. Mei, "Dual spoof disentanglement generation for face anti-spoofing with depth uncertainty learning," IEEE Trunsactions Circuits and Systems for Video Technology, 2021.
[42] M.-Z. Poh, D. J. McDuff, and R. W. Picard, "Non-contact, automated cardiac pulse measurements using video imaging and blind source separation." Optics express, vol. 18, no. 10, pp. 10762-10774, 2010.
[43] G. De Haan and V. Jeanne, "Robust pulse rate from chrominance-baset rppg," IEEE Transactions on Biomedical Engineering, vol. 60, 10, pp. 2878-2886, 2013.
[44] S. Liu, P. P.C. Yuen, S. Zhang, and G. Zhao, "3d mask face anti-spoofing with remote photoplethysmogrrppy."" in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14. Springer, 2016. pp. 85-100.
[45] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al., "Searching for mobilenetv3," in Proceedings of the IEEE/CVF international conference on computer vision, 2019. pp. 1314-1324.
[46] A. Dosovitskiy, Fischer, E. Ilg. P. Hausser, C. Hazirbas, V. Golkov, Der Smagt, Cremers. and T. Brox, "Flownet: Learning optical flow with convolutional networks," in Proceedings of the IEEE international conference computer vision, 2015, pp.
[47] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks," in Proceedings of the IEEE conference computer vision and pattern recognition, 2017, pp. 2462-2470.
[48] S.-Q. Liu. X. Lan, and P. C. Yuen, "Remote shotoplethymmography correspondence feature for 3d mask face presentation attack detection," in Proceedings of the European Conference Computer Vision (ECCV), 2018, pp. 558-573. 573.
[49] C. E. Shannon, "A mathematical theory of communication." The Bell system technical journal, vol. 27, no. 3, pp. 379 423, 1948.
[50] S. Umeyama. "Least-squares estimation of transformation parameters between point patterns," JEEE Transactions on Puttern Analysis & Machine Intelligence, vol. 13, no. 04, pp. 376-380, 1991.
[51] T.-T. Wong, "Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation," " Pattern Recognition, vol. 48, no. 9, pp. 2839 2846, 2015.
13


Reviews:
[{"review_id": "C2UkqprC7Q", "reviewer": "Reviewer_VV4D", "summary": "The paper presents a 3D mask detection tool called FASTEN, aimed at making face recognition systems more secure. The proposed network focuses on fine-grained details in large movements, which helps eliminate redundant spatio-temporal feature interference.  This approach lowers computational overhead and it outperforms six competitive models in both intra-dataset and cross-dataset evaluations as Table 2/3.  Real-world application on mobile devices (Table 5) shows the efficiency of the method.", "questions": "See the weakness.", "limitations": "N/A.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. FASTEN offers a solution by focusing on small details during big movements, helping to detect traces of 3D masks in fewer frames.\n2. The reduction in frames needed from 25 to just 5 means this tool is computationally more efficient and suitable for real-time applications. \n3. The fact that FASTEN can be used in real-time on mobile devices shows that it's practical for everyday use.", "weaknesses": "I am not directly working on the research topic. The contribution of this paper is more technical that combines the FlowNet and mobilenetv3 (as Fig. 2) to improve the efficiency of 3D mask detection across 5 frames. I think the authors should state the explicit insights of this paper, compared to the simple baseline of FlowNet+mobileNetv3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer VV4D:\n\nQ1: I think the authors should state the explicit insights of this paper, compared to the simple baseline of FlowNet+mobileNetv3.\n\nA1: It is intuitive to consider both spatial and temporal features when detecting 3D masks. However, as shown in the ablation study (Table 4), simply combining these two (Spat. + Temp.(Eq.)) will neglect the apparent movement changes among different frames, thus leading to average performance. Instead, we introduce frame weights obtained by temporal features to assign different significance to the spatial features in each frame. This contributes more to the detection with significant improvement, especially when the input frames include large movements, e.g., eye-blinking, mouth-opening. We will clarify more explanation in the revised manuscript.\n"}, {"review_id": "6bH2YSUMYW", "reviewer": "Reviewer_bEHS", "summary": "This paper presents an approach to 3D mask detection using a small number of face video frames. Experiments were conducted on several databases, and comparisons with other methods are presented. ", "questions": "See my questions and comments in Weakness part. ", "limitations": "The argument of using 5 frames is doubtable. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Using small networks for 3D mask detection and a small number of frames;\n\nBoth spatial and temporal features are extracted;\n\nMost parts are written clearly.", "weaknesses": "It seems that this work only focuses on 3D mask detection, without dealing with paper print, video replay, etc. This makes it quite narrow in terms of applications for face anti-spoofing.\n\nIt is not new to use both spatial and temporal features for face anti-spoofing. Many existing works have used spatial and temporal information.\n\nMore recent works use vision transformer based approaches for face anti-spoofing, which can effectively encode attention into the learning framework in a simple way, rather than using optical flow to indicate attention, as shown in the work. Further, there are no comparisons with the ViT based approaches in this paper. \n\nIt is confusing for the saying of \"only 5 frames\", since the five frames can be picked from a sequence with some fixed intervals, rather than consecutive frames in the original videos (25 or 30 frames per second), as indicated in some databases. Further, the paper argues to capture facial changes caused by pulse, how to acquire the blood changes within 5 frames? You can calculate: 5/25=0.2 seconds, which is less than a pulse period (suppose a regular person has 60 pulses per minute, then one pulse will take one second).\n\nIt mentions that the method can be developed in a mobile device, and shows the computation and memory used. But there is no accuracy or error rate reported. How accurate it can achieve in mobile device?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer bEHS:\n\nQ1: Only focusing on 3D mask detection, which is quite narrow in terms of applications for face anti-spoofing.\n\nA1: Owing to the rapid development and maturity of 3D printing, detecting 3D masks has become a rising challenging task in the field of face anti-spoofing, which threatens most existing PADs [1][2][3]. 3D masks made up of soft silicone are vivid enough, in terms of color, texture, and geometry structure, that even humans find it confusing to differentiate. This has gradually made 3D mask defense from a branch of face anti-spoofing to a relatively independent sub-topic. To date, effective defense against 3D masks remains a missing piece. Therefore, our method focuses on this type of attack (this is also where the motivation of our method comes from). As a deep learning method by extracting features, we believe that our method can also be used in other attack types e.g., printing, replay, since their features are relatively simpler. In addition, in terms of the actual application you mentioned, in actual face anti-spoofing security applications, it is usually the aggregation of multiple defense models. Other attacks can be separately mitigated by the corresponding tactics. Deploying our defense model will enhance the overall security of the system.\n\nQ2: It is not new to use both spatial and temporal features for face anti-spoofing.\n\nA2: Simply using both spatial and temporal features is not new for detection tasks, but our contribution focuses more on the frame-wise attention to aggregate the feature information of multiple frames, rather than simple addition or concatenation of the two features. We provide comprehensive experimental results in the ablation study (Table 4) to show the outperformance of our defense (Spat. + Temp.(UnEq.)) over simply combining two features together (Spat. + Temp.(Eq.)) without considering frame weights. According to our best knowledge, we are the first to consider flow attention and spatio-temporal aggregation for 3D mask detection. We will add more explanation in the revised manuscript.\n\nQ3: Recent works using ViT, can effectively encode attention into the learning framework in a simple way, rather than using optical flow to indicate attention.\n\nA3: The “attention” you mentioned here is different to the form of attention in our method. We used the optical flow to generate inter-frame attention (the contribution or weight of each frame to the detection) but not the attention on each feature map. However, we reckon this comment raised an interesting point, the attention on the spatial dimension also plays an important indication to the final detection. We therefore compare our defense with a ViT-based method, ViTranZFAS [4], which is slightly modified to adapt to 3D mask detection. Table 1 in the enclosure shows the intra-dataset and cross-dataset results on HiFiMask. FASTEN performs better than ViTranZFAS in almost all cases. We attribute it to three main reasons: 1)  Lack of Temporal Information: ViT primarily focuses on analyzing individual images without considering temporal information. Facial liveness detection often requires understanding the dynamic changes over time; ViT's static image analysis might not effectively capture these temporal cues. 2) Small Regions of Interest: Liveness cues often involve subtle facial regions like the edge of the facial features or muscle contractions; ViT's patch-based approach might not focus on these small regions of interest effectively, leading to challenges in capturing fine-grained liveness indicators. 3) Training Difficulty: ViT requires a pretrained model and high computational costs.\n\nQ4: “Only 5 frames”\n\nA4: Our method supports picking 5 frames at equal/unequal intervals. For datasets such as HiFiMask that have been pre-sampled, we directly use the preprocessed sampled data, since our method is not sensitive to how the 5 frames are selected, and just focuses on the movement between consecutive selected frames. We explained in the Statement for rPPG methods in the paper that they usually require a longer video clip (usually more than 10 seconds), and require continuous (high-frequency sampling, by Nyquist–Shannon Sampling Theorem) signal inputs. So “5-frame input” is for our method, not rPPG methods.\n\nQ5: How accurately can it be achieved in mobile devices?\n\nA5: With less computational cost and model parameters, quantization acceleration is unnecessary when deploying on mobile devices, so we can practically use it with almost no loss of accuracy. The overall system achieves a defense rate (accuracy) of 99.3% against 3D masks.\n\nReference:\n\n[1] Liu S Q, Lan X, Yuen P C. Learning Temporal Similarity of Remote Photoplethysmography for Fast 3D Mask Face Presentation Attack Detection[J]. IEEE Transactions on Information Forensics and Security, 2022, 17: 3195-3210.\n\n[2] Yu Z, Qin Y, Li X, et al. Deep learning for face anti-spoofing: A survey[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(5): 5609-5631. \n\n[3] Erdogmus N, Marcel S. Spoofing face recognition with 3D masks[J]. IEEE transactions on information forensics and security, 2014, 9(7): 1084-1097.\n\n[4] George A, Marcel S. On the effectiveness of vision transformers for zero-shot face anti-spoofing[C]//2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 2021: 1-8.\n"}, {"review_id": "uqVkGZlP8i", "reviewer": "Reviewer_U1im", "summary": "This work proposes a 3D mask detection system, successfully deployed on mobile devices. In addition, the number of frames required for making predictions is minimal, improving the latency of responses. Experiments,  performed with face mask datasets showed improved performance compared to existing techniques.", "questions": "1. How would motion affect performance of the proposed method? You stated that ‘flow attention can effectively help FASTEN assign larger frame weights to frames with large movement.’ but what type of the movement is acceptable?\n2. Average processing time has been provided, but what’s the latency of responses? Do you use batch processing at the inference time?", "limitations": "Limitations and ideas for future work improvement were clearly explained and make sense.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The introduced method is very interesting. \n2. Combination of facial optical flow for inter-frame characteristics with flow attention and spatio-temporal aggregation (to the best of my knowledge) hasn’t been applied for 3D face mask detection yet. \n3. The use of only 5 frames and potential of deploying such solution on mobile devices is very promising. \n4. The method is well described with clear comparison to hand-crafted features, deep learning methods and remote vital estimation approaches. \n5. Experimental analysis is clear and thorough with results supporting claims well.", "weaknesses": "1. The method has only been tested using MobileNet backbone. It’s understandable that experimental analysis has to have some limitations, but it would be interesting to explore other maybe even faster classifiers.\n2. Minor comment: Font in tables is very small, please increase readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer U1im:\n\nQ1: More backbones.\n\nA1: Thanks for your advice. We use MobileNet due to its excellent accuracy-efficiency trade-off. Replacing it with other backbones may accelerate the speed but the performance improvement is limited. We will consider this advice in our future work. \n\nQ2: Fonts in tables.\n\nA2: We will check the fonts in all tables and figures and adjust them for better readability. \n\nQ3: Large movement.\n\nA3: One of our work’s insights is that the motion characteristics difference between the mask and that of the face, due to that the mask is not perfectly fit for the face. In practical applications, we consider movements such as nodding, shaking the head, and opening the mouth. \n\nQ4: Latency and batch processing.\n\nA4: The time mentioned in Table 5 refers to the time it takes for a single image to be processed using a mobile device. Considering the pre-processing time for video parsing and other necessary steps in actual usage, the overall response time is about 879 ms. We do not use batch processing at the inference time.\n\n"}, {"review_id": "SzPYVPtRJR", "reviewer": "Reviewer_iaxD", "summary": "In this paper, To enhance the accuracy of face presentation attack detection by effectively incorporating temporal information，the author proposes a novel 3D mask detection framework. The architecture integrates: 1) a facial optical flow network to obtain non-RGB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation to aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms six competitors for both intra-dataset and cross-dataset evaluations in terms of multiple detection metrics.", "questions": "none", "limitations": "see weakness", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The author employs a simple model structure that enables deployment on end devices. It is hoped that the author can further compare the Flops and Params of this model with other models.\n2.The author simplifies the structure of the optical flow calculation model specifically for facial features, thereby reducing computational burden.\n3.The model only requires five images as input, reducing the requirement for the length of the input sequence compared to the rPPG method.", "weaknesses": "1.The structure proposed in this article is not sufficiently innovative, as it only uses simplified optical flow networks and MobileNetV3-Small. The fusion of the spatial features and optical flow features is achieved by employing the \"calculate the frame weights\" method.\n2.Due to this paper has achieved results beyond previous works by using a simple structure, it is hoped that the author will open-source the code for validation purposes.\n3.Although the model exhibits significant improvement in accuracy compared to rPPG-based models, which is attributed to the longer temporal information required for measuring rPPG signals, the improvement in accuracy compared to \"Deep-Learning-based Methods\" from two years ago is limited. It is requested that the author compare their approach with recent algorithms developed in the past two years.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer iaxD:\n\nQ1: Novelty.\n\nA1: Simply using both spatial and temporal features is out of intuition and not new for detection tasks, but our contribution focuses more on the frame-wise attention to aggregate the feature information of multiple frames, rather than simple addition or concatenation of the two features. We provide comprehensive experimental results in the ablation study (Table 4) to show the outperformance of our defense (Spat. + Temp.(UnEq.)) over only combining two features together (Spat. + Temp.(Eq.)) which neglects the apparent movement changes among different frames. Assigning different frame weights in our method contributes more to the detection with significant improvement, especially when the input frames include large movements, e.g., eye-blinking, mouth-opening. As mentioned by Reviewer U1im, and also according to our best knowledge, we are the first to consider flow attention and spatio-temporal aggregation for 3D mask detection. We will add more explanation in the revised manuscript.\n\nQ2: Open source.\n\nA2: We feel sorry that we currently cannot open-source our code for public disclosure since it may involve confidentiality. We will open-source a demo code and model structure once the paper is accepted.\n\nQ3: It is requested that the author compare their approach with recent algorithms developed in the past two years.\n\nA3: We additionally compare our method with one ViT-based method, ViTranZFAS [1], and one deep-learning-based method, MD-FAS [2]. \nTable 1 in the enclosure shows the intra-dataset and cross-dataset results on HiFiMask. ViTranZFAS performs worse than FASTEN. We attribute it to three main reasons: 1) Lack of Temporal Information: ViT primarily focuses on analyzing individual images without considering temporal information. Facial liveness detection often requires understanding the dynamic changes over time, such as blinking, movement, or facial expressions. ViT's static image analysis might not effectively capture these temporal cues. 2) Small Regions of Interest: Liveness cues often involve subtle facial regions like the edge of the facial features or muscle contractions. ViT's patch-based approach might not focus on these small regions of interest effectively, leading to challenges in capturing fine-grained liveness indicators. 3) Training Difficulty: ViT requires a pretrained model, which requires high computational costs.\n\nMD-FAS aims to solve the forgetting issue when learning a new domain data and improve the adaptability of the model. It cannot perform well with unseen target data, as shown in the cross-dataset evaluation. In addition, the baseline SRENet selected by this scheme is a region-based fine classification task, which has high requirements for the diversity of attack categories, as well as the resolution of images. When we use HiFiMask as the training set for a fair comparison, its accuracy performance remains poor.\n\nReference:\n\n[1] George A, Marcel S. On the effectiveness of vision transformers for zero-shot face anti-spoofing[C]//2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 2021: 1-8.\n\n[2] Guo X, Liu Y, Jain A, et al. Multi-domain Learning for Updating Face Anti-spoofing Models[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 230-249.\n"}, {"review_id": "brmzrsXnRN", "reviewer": "Reviewer_jUZR", "summary": "This paper presents a framework called FASTEN (Flow-Attention-based Spatio-Temporal Aggregation Network) for 3D mask presentation attack detection. In previous works, as a recent technology rPPG addresses some of the limitations but also sensitivity to noise and high computational overhead. \n\nTo overcome these challenges, FASTEN is designed to focus on fine-grained details in large movements, aiming to eliminate redundant spatio-temporal feature interference and capture splicing traces of 3D masks using fewer frames. \n\nIt comprises three key modules: a facial optical flow network for obtaining inter-frame flow information, flow attention for assigning different significance to each frame, and spatio-temporal aggregation for combining high-level spatial features and temporal transition features.", "questions": "The proposed method seems to focus on fine details in large movements. However, I am very curious about how the model will perform if the detected object does not have large movements.\n\nIn line 44, As far as I know, the majority of rPPG methods involve analyzing facial videos captured by a camera and do not necessitate the use of additional photodetectors. \n", "limitations": "It might be interesting if we could see more discussions about the different regions.\n\nIt would be useful to see more experiments and discussions in real-world scenarios. For example, a comparison of performance with previous methods in real-world scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Through extensive experiments, FASTEN demonstrates a good performance compared to six competing methods in both intra-dataset and cross-dataset evaluations using multiple detection metrics. \n\nThe three datasets used for the evaluation are frequently used in the field and are publicly available.\n\nThe authors mention the deployment of FASTEN on real-world mobile devices for real-time 3D mask detection. This demonstrates the practicality and applicability of the proposed framework in real-world scenarios, validating its potential for deployment in various practical applications and settings.", "weaknesses": "The proposed method by the authors seems to have limited novelty as a substantial portion of its components is derived from existing work. \n\nThe experiments and discussions in real-world scenarios are very limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer jUZR:\n\nQ1: Limited novelty as a substantial portion of its components is derived from existing work.\n\nA1:  Simply using both spatial and temporal features is out of intuition and not new for detection tasks, but our contribution focuses more on the frame-wise attention to aggregate the feature information of multiple frames, rather than simple addition or concatenation of the two features. We provide comprehensive experimental results in the ablation study (Table 4) to show the outperformance of our defense (Spat. + Temp.(UnEq.)) over only combining two features together (Spat. + Temp.(Eq.)) which neglects the apparent movement changes among different frames. Assigning different frame weights in our method contributes more to the detection with significant improvement, especially when the input frames include large movements, e.g., eye-blinking, mouth-opening. According to our best knowledge, we are the first to consider flow attention and spatio-temporal aggregation for 3D mask detection. We will add more explanation in the revised manuscript.\n\nQ2: The experiments and discussions in real-world scenarios are very limited.\n\nA2: Our work proves that the accuracy is almost unaffected when the model is deployed in real-world scenarios. Considering the development cost and time constraints, we devote more effort to comparing and discussing experimental data.\n\nQ3: Large movements and different regions.\n\nA3: Our method performs better for input videos with larger motions, but since we also have a spatial branch, when the motion range is small, our method will rely more on the spatial branch, with the temporal branch providing assistance. Our experimental results show the average results of the whole dataset, which includes both large movements and trivial movements. The sound results can provide effective support for our method's effectiveness in inputs with subtle movements. We think it’s not scalable to conduct a comparison experiment to test the defense performance solely in small movements of the dataset since it is hard to quantitatively define the boundary between large and trivial movements.\n\nQ4: Error in line 44.\n\nA4: We admit this is a factual error. We will modify it in the revised version. \n"}]

Author rebuttal:
Thanks for your feedback! We summarize our responses by questions in this rebuttal. Please see more details in the responses to each reviewer.

* Ethics [Ethics Reviewer HeTH, Nmac]

We acknowledge there might be social impacts associated with our work. We will accordingly add a section to the paper to discuss the impact.

Our defense alleviates 3D mask spoofing attacks against face recognition systems (FRS). Such defense would be beneficial for minimizing security concerns associated with FRS in financial and public service sectors. However, we would also like to extend our considerations to privacy and fairness issues of the defense and its serving FRS. 1) Privacy. Our defense applies to the specific task of spoofing detection in which the facial information is presented to the associated FRS as credentials. To ensure no private information is memorized, our model is trained on public datasets and does not use any private photos for the training purpose. 2) Fairness. In our work, we use HiFiMask as one of the main datasets. HiFiMask provides an equal number of subjects (25 each) for each ethnicity to facilitate fair artificial intelligence and mitigate biases [1]. Our training method also encourages the fairness of the trained defense. However, considering the defense is an independent component in the FRS, it cannot improve the fairness of the overall FRS. Moreover, there might be fairness issues for newborns or people with certain skin diseases since they do not have enough representation in the training dataset. Another possibility would be the dataset itself contains unbalanced data or faces with even only one skin tone. If this happens, incremental training after collecting obfuscated/synthetic data of the corresponding group of people can be used to mitigate the fairness issue. We also encourage future research in this area to incorporate fairness training techniques to mitigate possible issues.

* Novelty [Reviewer VV4D, bEHS, iaxD, jUZR]

We emphasize that our contribution focuses more on the frame-wise attention to aggregate the feature information of multiple frames, rather than simple addition or concatenation of the two features. We provide comprehensive experimental results in the ablation study (Table 4) to show the outperformance of our defense (Spat. + Temp.(UnEq.)) over only combining two features together (Spat. + Temp.(Eq.)) which neglects the apparent movement changes among different frames. We are the first to consider flow attention and spatiotemporal aggregation for 3D mask detection.

* Research focus [Reviewer bEHS]

Detecting 3D masks has become a rising challenge for most existing PADs owing to the rapid development and maturity of 3D printing [2]. Realistic 3D masks are hard to differentiate even for human eyes. To date, effective defense against 3D masks remains a missing piece. Therefore, our method focuses on this type of attack (this is also where the motivation of our method comes from). As a deep learning method by extracting features, we believe that our method can also be adapted to other attack types e.g., printing, replay, since their features are relatively simpler. Moreover, in actual face anti-spoofing security applications, it is usually an aggregation of multiple defense models. Other attacks can be separately mitigated by the corresponding tactics. Deploying our defense will enhance the overall security of the system.

* More competitors [Reviewer bEHS, iaxD]

We additionally compare our method with ViTranZFAS[3], and one recent deep-learning-based method, MD-FAS[4]. Table 1 in the enclosure shows the intra-dataset and cross-dataset results on HiFiMask. Please refer to the responses to each reviewer for more analyses.

* Frames [Reviewer bEHS]

Our method supports picking five frames at equal/unequal intervals. For datasets such as HiFiMask that have been pre-sampled, we directly use the preprocessed sampled data, since our method is not sensitive to how the frames are selected, and just focuses on the movement between consecutive selected frames. We explained in the Statement for rPPG methods in the paper that they usually require a longer video clip (usually more than 10 seconds), and require continuous (high-frequency sampling, by Nyquist–Shannon Sampling Theorem) signal inputs. So “5-frame input” is for our method, not rPPG methods.

* Real-world scenarios [Reviewer bEHS, U1im, jUZR]

bEHS: With less computational cost and model parameters, quantization acceleration is unnecessary when deploying on mobile devices, so we can practically use it with almost no loss of accuracy. The overall system achieves a defense rate (accuracy) of 99.3% against 3D masks.

U1im: The time mentioned in Table 5 refers to the time it takes for a single image to be processed using a mobile device. Considering the pre-processing time for video parsing and other necessary steps in actual usage, the overall response time is about 879 ms. We do not use batch processing at the inference time.

jUZR: Our work proves that the accuracy is almost unaffected when the model is deployed in real-world scenarios. Considering the development cost and time constraints, we devote more effort to comparing and discussing experimental data.
* Open source [Reviewer iaxD]

We feel sorry that we currently cannot open-source our code for public disclosure since it may involve confidentiality. We will open-source a demo code and model structure once the paper is accepted.

Reference:

[1] Liu A, et al. Contrastive context-aware learning for 3d high-fidelity mask face presentation attack detection. TIFS, 2022.

[2] Yu Z, et al. Deep learning for face anti-spoofing: A survey. TPAMI, 2022. 

[3] George A, et al. On the effectiveness of vision transformers for zero-shot face anti-spoofing. IJCB, 2021. 

[4] Guo X, et al. Multi-domain Learning for Updating Face Anti-spoofing Models. ECCV, 2022.


Meta-review: This paper presents an approach to 3D mask detection. Experiments on several databases demonstrated the effectiveness of the proposed approach. The author rebuttal provided feedback to the review comments. Three positive reviewer were satisfied with the rebuttal. The other two reviewers were negative and did not provide further comments after the rebuttal. The AC checked the comments from Reviewer iaxD and the feedback. The critical one is the third one, comparison to the recent algorithms is needed. The authors provided additional results, which are convincing. Regarding the comments from Reviewer bEHS, the AC checked the feedback and most comments were well addressed. One issue is about the accuracy for the mobile device. As the rebuttal said in the discussion with the reviewer, the authors will modify the description about the real-time issue. Please do make modification.
Decision: Accept (poster)