FM-CLIP: Flexible Modal CLIP for Face Anti-Spoofing

Authors: ajian  liu
junze  zheng
haocheng  yuan
xiaoyuan  yu
yanyan  liang
sergio  escalera
jun  wan
  casia
hui  ma
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 11:43:38

Abstract: Flexible modal Face Anti-spoofing (FAS) aims to aggregate all the available training modalities’ data to train a model, and enables flexible testing of any given modal samples. Previous works introduce shared cross-modal transformers (attentions) to facilitate the learning of modality-agnostic features, which inevitably leads to the distortion of feature structures and achieves limited performance. In this work, borrowing a solution from the large-scale vision-language models (VLMs) instead of directly removing modality-specific signals from visual features, we propose a novel Flexible Modal CLIP (\textbf{FM-CLIP}) for flexible modal FAS, that can utilize text features to dynamically adjust visual features to be modality independent. In the visual branch, considering the huge visual differences of the same attack in different modalities, which makes it difficult for classifiers to flexibly identify subtle spoofing clues in different test modalities, we propose Cross-Modal Spoofing Enhancer (\textbf{CMS-Enhancer}). It includes a Frequency Extractor (\textbf{FE}) and Cross-Modal Interactor (\textbf{CMI}), aiming to map different modal attacks in a shared frequency space to reduce interference from modality-specific signals and enhance spoofing clues by leveraging cross modal learning from the shared frequency space. In the text branch, we introduce a Language-Guided Patch Alignment (\textbf{LGPA}) based on the prompt learning, which further guides the image encoder to focus on patch level spoofing representations through dynamic weighting by text features. Thus, our FM-CLIP can flexibly test different modal samples by identifying and enhancing modality-agnostic spoofing cues. Finally, extensive experiments show that FM-CLIP is effective and outperforms state-of-the-art methods on multiple multi-modal datasets.

Main paper: 
 introduction: 
 The task of Face Anti-spoofing (FAS) is to protect face recognition systems from physical media-based presentation attacks, such as print [53, 54] , replay [3] and mask [4, 21] . With the increasing advancement of attack mediums and spoofing methods, FAS algorithms [14, 20, 29, 30, 50, 51] designed based on RGB modality have become challenging for realistic spoofing clues. The multi-modal algorithms [8, 9, 17, 53] target these clues by leveraging the complementary advantages of multi-modal samples, which insist that the same attack is difficult to evade detection from multiple spectra simultaneously. However, these multi-modal fusion methods need to provide samples of the same modality during training and testing. When any modality disappears during testing, these methods will not be able to differentiate between real and fake faces, resulting in poor performance. Due to hardware cost and space constraints, it is not always possible to provide consistent modal samples in practical applications, which makes these systems difficult to deploy widely. Recently, flexible modality FAS [16, 19, 47, 48] has attracted widespread discussion in the community because it can be flexibly deployed in any given test modality environment without providing the same modal type as in the training stage. A type method in this field is FM-ViT [19] , which sets up an independent encoder for each modality. After learning individual modality features from different branches, a shared cross-modal block is established between them to guide each branch to learn potential and modality agnostic active features by summarizing multi-modal information as shown in Fig. 1 (a) . Due to the significant differences in the same attack type in different modalities, classifiers are highly susceptible to interference from modality-related information. Inspired by CLIP [36] , in this work, borrowing a solution from the large-scale vision-language models (VLMs) instead of directly removing modality-specific signals from visual features, we propose a novel Flexible Modal CLIP (FM-CLIP) for flexible modal FAS as shown in Fig. 1 (b), that can utilize text features to dynamically adjust visual features to be modality independent. In the visual branch, considering the huge spatial domain differences of the same attack in different modalities, which makes it difficult for classifiers to flexibly identify subtle spoofing clues in different test modalities, we propose Cross-Modal Spoofing Enhancer (CMS-Enhancer). Specifically, it includes a Frequency Extractor (FE) and Cross-Modal Interactor (CMI). FE converts visual features into the frequency domain through discrete cosine transform (DCT) and further adaptively adjusts the distribution of spatial domain features to mine essential spoofing clues. CMI conducts cross-modal mutual learning of effective cues in each modality to enhance the features of a single modality. FE and CMI work together to map different modal attacks in a shared frequency space to reduce interference from modality-specific signals and enhance spoofing clues by leveraging cross-modal learning from the shared frequency space. In the text branch, considering the alignment between image features and text feature embeddings can lead to non-trivial generalization improvements, so we introduce a Language-Guided Patch Alignment (LGPA) based on the prompt learning, which further guides the image encoder to focus on patch level spoofing representations through dynamic weighting by text features. Specifically, we follow CoOp [57]  
related work: 
 Single-Modal FAS Methods. With the rise of deep learning frameworks, Some CNN-based methods [13, 31] design a unified framework for feature extraction and classification in an end-to-end manner. Another works [29, 38, 45, 51] utilize physical-based depth information as a supervisory signal instead of binary classification loss, improving the model's ability to perceive and understand scene depth more effectively. Although these algorithms have achieved astonishing results in intra-datasets experiments, their performance deteriorates severely when faced with unknown domains.To solve these limits, domain generalization-based FAS is increasingly receiving attention from researchers. Some Domain Adaptation-based methods [25, 26, 42, 44, 52] aim to minimize the distribution discrepancy between the source and target domain by leveraging the unlabeled target data. Another Domain Generalization based methods [10, 14, 20, 23, 24, 37, 39-41, 43, 50] can conquer this by taking the advantage of multiple source domains. MFAE [56] randomly masks the low-frequency spectrum of images and reconstructs the images for self-supervised pre-training of Vision Transformers, and also integrates an auxiliary contentregularization decoder to further enhance the model's insensitivity to low-frequency features. CIFAS [27] addresses the challenge of domain bias by introducing causal intervention techniques. Liu et al. [28] propose UDG-FAS, the first Unsupervised DG framework for FAS. CFPL-FAS [20] targets DG FAS via textual prompt learning for the first time, which utilizes two lightweight models to learn the different semantic prompts conditioned on content and style features. CA-MoEiT [14] introduces professional experts and super expert to solve DG FAS.Multi-Modal Fusion Methods. Multi-modal FAS has gained attention due to the increasing sophistication of high-quality 2D attacks, such as those present in datasets like OULU-NPU [1] , SiW [29] , CelebA-Spoof [55] as well as high-fidelity mask attacks, including MARsV2 [22] , and HiFiMask [21] . These attacks exhibit realistic color, and texture making it challenging to detect spoofing clues using only the visible spectrum. To address this issue, multi-modal fusion methods [8, 9, 17, 18, 53] have proven effective by leveraging different modalities that may reveal distinct properties of fake faces. However, these multi-modal fusion-based algorithms require the testing phase to provide the same modal types as the training phase, limiting their deployment scenarios.Flexible-Modal Methods. To alleviate the limitation of consistency between testing and training modalities, flexible modalitybased methods [8, 12, 16, 19, 47, 48] aim to improve the performance of any single modality by leveraging available multi-modal data. George et al. [8] presents a framework for PAD that uses RGB and depth channels supervised by the proposed cross-modal focal loss (CMFL), which makes it possible to train models using all the available channels and to deploy with a subset of channels. MA-ViT [16] adopts the early fusion to aggregate all the available training modalities' data and enables flexible testing of any given modal samples with a Modality-Agnostic Transformer Block. Yu et al. [47, 48] introduce a flexible-modal benchmark aimed at training a unified model capable of deployment across various modality scenarios. FM-ViT [19] retains a specific branch for each modality to capture different modal information and introduces the Cross-Modal Transformer Block, which consists of two cascaded attentions named Multi-headed Mutual-Attention and Fusion-Attention. MMDG [12] proposes a multi-modal domain generalization framework MMDG, which addresses modal unreliability and imbalance issues through uncertainty-guided U-Adapter and ReGrad strategy. 
method: 
 CLIP. We adopt CLIP [36] as the pre-trained model with a ViT image encoder and transformer text encoder, respectively. Given an image 𝑥 ∈ R 3×𝐻 ×𝑊 , the vision encoder converts it into a 𝐷dimensional image feature 𝑓 𝑖𝑚𝑔 ∈ R (1+𝑁 ) ×𝐷 , where 1 represents the CLS token and 𝑁 denotes the patch tokens. Meanwhile, the text encoder 𝑔 (.) takes text descriptions 𝑡 and generates the text feature 𝑓 𝑡𝑒𝑥𝑡 ∈ R 𝐾 ×𝐷 from the appended EOS tokens, where 𝐾 denotes the number of classes. Formally, let cls token 𝑓 (0) 𝑖𝑚𝑔 extracted for a testing image x, and given the text prompts 𝑃, we have the predicted similarity of class 𝑖 ∈ {0, 1}, where 0 represents 'real' and 1 is 'fake':EQUATION)where 𝜏 is a temperature coefficient, and y denotes the label.To cope with the difficulty of classifiers in identifying subtle spoofing clues caused by the huge difference in the representation of the same attack type in different modalities, we introduce a Cross-Modal Spoofing Enhancer (CMS-Enhancer) to bridge the adjacent ViT stage in the image encoder as shown in Fig. EQUATIONEQUATIONwhere Conv1 and Conv3 represent the 1x1 and 3x3 convolution kernel respectively; 𝐹( 𝑗 )𝑖𝑛𝑝𝑢𝑡 and 𝐹( 𝑗 )𝑆𝐸_𝑜𝑢𝑡𝑝𝑢𝑡 represent the vanilla transformer layer features and output features extracted by the SE module in j-th ViT stage respectively; F ( 𝑗 ) 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 represents the spatial feature; ⊕ represents element-wise summation. Frequency Extractor (FE). As is known, RGB images and depth maps contain various information. Depth maps contain more contours, while RGB images convey rich detailed information, such as texture and color. Huge differences in representations of different modalities can cause classifiers to overly focus on modal content information. In the frequency domain, an image is represented as a superposition of its different frequency components. Inspired by FcaNet [35] , we construct a lightweight Frequency Extractor (FE) in the frequency domain, comprising Discrete Cosine Transform (DCT), two convolution layers, and one GELU layers, as followsEQUATIONEQUATIONwhere 𝜎 is the sigmoid function, and ⊗ represents element-wise multiplication. The 2D DCT is mathematically defined as follows:EQUATIONViT where H and W are the height and width of the input feature map. Different modal features are mapped in a shared frequency domain, and the Cross-Modal Interactor module is used to complementarily enhance the spoofing-related representation of each modality. Cross-Modal Interactor (CMI). Inspired by [46] and [5] , we use a gate network to extract discriminative information from each modal feature. Specifically, the input frequency featureF ( 𝑗 ) 𝑓 𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦is fed into a gate consisting of a 3x3 convolutional layer and a Sigmoid activation function, and then we can obtain the value probability gate maps M 𝐹𝑟𝑒𝑞.𝑅𝐺𝐵 ∈ [0, 1] and M 𝐹𝑟𝑒𝑞.𝐷𝑒𝑝𝑡ℎ ∈ [0, 1]. Since the calculation of each ViT stage is the same, in the subsequent formulas, we omit the ViT stage mark ( 𝑗). The formulation isEQUATIONThe larger the value in this map, the more discriminative information the feature vectors in 𝐹 𝐹𝑟𝑒𝑞.𝑅𝐺𝐵 have at their corresponding positions. On the contrary, the smaller the value, the smaller the amount of valuable information. Furthermore, the uninformative gate maps can be represented as M 𝐹𝑟𝑒𝑞.𝑅𝐺𝐵 and M 𝐹𝑟𝑒𝑞. 𝐷𝑒𝑝𝑡ℎ as follows:EQUATIONTwo unimodal features F 𝐹𝑟𝑒𝑞.𝑅𝐺𝐵 and F 𝐹𝑟𝑒𝑞.𝐷𝑒𝑝𝑡ℎ are obtained by:F 𝐹𝑟𝑒𝑞. 𝑅𝐺𝐵 = M 𝐹𝑟𝑒𝑞. 𝑅𝐺𝐵 ⊗ F 𝐹𝑟𝑒𝑞. 𝑅𝐺𝐵 ; F 𝐹𝑟𝑒𝑞. 𝐷𝑒𝑝𝑡ℎ = M 𝐹𝑟𝑒𝑞. 𝐷𝑒𝑝𝑡ℎ ⊗ F 𝐹𝑟𝑒𝑞.𝐷𝑒𝑝𝑡ℎ , (9) Through such calculations, the model can effectively learn useful information from features, thereby suppressing the interference of useless information on the model. To utilize the cross-modal complementary relationships, we perform the same operation between the uninformative gate maps ( M 𝐷𝑒𝑝𝑡ℎ ) to get complimentary cross-modal features:EQUATIONPlease note that the complementary information from the deep modality is selected by the uninformative gate map of the RGB modality, which means it propagates to positions with little information in the RGB features. In addition, we adopt residual connections to preserve the original feature of each modality. Therefore, we can capture the complementary features across modalities. So we can obtain these two features in the following way:EQUATIONFurthermore, the frequency domain enhanced features and spatial domain features corresponding to each modal sample are fusedEQUATIONwhere F 𝐸 𝑅𝐺𝐵 and F 𝐸 𝐷𝑒𝑝𝑡ℎ represent enhanced features; F 𝑆𝑝𝑎𝑡 .𝑅𝐺𝐵 and F 𝑆𝑝𝑎𝑡 .𝐷𝑒𝑝𝑡ℎ represent the spatial feature extracted by SE model.In the text branch, we introduce a Language-Guided Patch Alignment based on prompt learning, which further guides the image encoder to focus on patch-level spoofing cues through dynamic weighting by text features. In a bit more detail, prompt learning initializes learnable content embeddings, which are fed into a text encoder along with handcrafted categories to produce text supervision signals. The LGPA aligns image patches with text embedding features. Finally, we combine patch-based and CLS token-based alignment to supervise model training. Prompt learning. We follow CoOp [57] to avoid prompt engineering to further enhance the migration capabilities of CLIP models. Different from the zero-shot transfer that used a fixed hand-craft prompt, we construct and fine-tune a set of M continuous context vectors 𝒗 = {𝒗 1 , 𝒗 2 , ..., 𝒗 𝑀 } as the turntable prompt. Specifically, the prompt 𝒕 𝑖 = {𝒗 1 , 𝒗 2 , ..., 𝒗 𝑀 , 𝒄 𝑖 } combines the learnable context vectors 𝒗 and the class embedding 𝒄 𝑖 , and is fed to the text encoder 𝑔(•). The probability for 𝑦-th class is obtained asEQUATIONLanguage-Guided Patch Alignment (LGPA). We propose a Language-Guided Patch Alignment, which aligns the image patch with text features to guide visual networks to attend to spoofing cues adaptively inspired by [33] . Specifically, given the text prompt embeddings 𝑓 𝑡𝑒𝑥𝑡 ∈ R 𝐾𝑥𝐷 from text encoder and image patch tokens 𝑓(1:𝑁 ) 𝑖𝑚𝑔 ∈ R 𝑁 𝑥𝐷 , our Language-guided calculates the similarity matrix 𝑆 between them byEQUATION)where • represents matrix multiplication, and 𝑆 ∈ R 𝑁 𝑥𝐾 . We fuse textual features with similar visual representations in image patches,EQUATIONwhere f (1:𝑁 )denotes the language-guided image patch tokens. For the loss calculation, we combine visual patch-based alignment and visual CLS token-based alignment to supervise model training. For visual CLS token-based alignment, we apply Eq. 13 to calculate probability. For visual patch-based alignment, we calculate image patch tokens probability, as followsEQUATIONIn this paper, we adopt the cross entropy loss on Eq. 13 and Eq. 16 with label 𝑦 ∈ {0, 1} to calculate loss, as followsEQUATIONthe total loss is as followsEQUATION)where L 𝑃 and L 𝐶 represent patch loss and CLS loss respectively. 
experiments: 
 Datasets & Protocols. We use three commonly used multi-modal FAS datasets for experiments, including CASIA-SURF (SURF) [53] , CASIA-SURF CeFA (CeFA) [17] , WMCA [9] . SURF [53] consists of 1, 000 subjects with 21, 000 videos and each sample has 3 modalities, and we follow a protocol to evaluate the performance against unknown attack types. CeFA [17] covers 3 modalities, 1, 607 subjects, and provides five protocols. We select the Protocols 1, 2, and 4 for experiments. WMCA [9] contains a wide variety of presentation attacks, which introduces 2 protocols: the grandest protocol emulates the "seen" attack scenario and the "unseen" protocol evaluates the generalization of an unseen attack.Test Scenario Settings&Evaluation Metrics. We follow the test scenarios of FM-ViT [19] . The first is a commonly used setting where the test modalities need to be consistent with the training stage. The second is a flexible modal test scenario, which means the user can provide any single-modal sample. Attack Presentation Classification Error Rate (APCER), Bonafide Presentation Classification Error Rate (BPCER), and ACER [11] are used for the metrics. Implementation Details. We utilize ViT-B/16 as an image encoder, whose output embedding dimension is 768. Meanwhile, we employ a transformer-based text encoder. In the training stage, the input images are cropped and resized to 224 x 224 x3. We apply random cropping and random horizontal flipping at training, while center cropping at testing, both with no other augmentations. We use an Adam optimizer with an initial learning rate of 1e-6 and weight decay of 1e-6. We train all methods with a maximum of 200 epochs, and FM-CLIP is trained with a batch size of 12. Unless otherwise stated, all ablation experiments are conducted with WMCA ('seen') data in the flexible modality test scenario.Fixed Modal Scenario Evaluations. The fixed modal scenario setting evaluates the fusion ability of the FM-CLIP framework approach to multi-modal information. We verified the performance of the CMS-Enhancer and the FM-CLIP overall framework. (SURF): As shown in Tab. 1, after adding CMS-Enhancer to the visual network ViT, compared to FM-ViT, the ACER was reduced from 0.45 to 0.44. Further, we added LGPA to form the final FM-CLIP, and the ACER was further reduced to 0.43. VisioinLabs used additional datasets, Method TPR APCER BPCER ACER @FPR=10 -2 @FPR=10 -3 @FPR=10 -4 MS-SEF [53] 99 LGPA achieved APCER, BPCER, and ACER of 2.24, 1.81, and 2.10 respectively; Finally, the combination of Prompt Learning and LGPA achieved 2.05, 1.73, and 1.89 respectively.Model parameter analysis. We conducted a complexity analysis on the FM-CLIP framework as shown in Tab. 9. Specifically, the FLOPs of FM-ViT [19]  
conclusion: 
 In this paper, we borrowed large-scale visual language models (VLM) to propose a novel Flexible Modal CLIP (FM-CLIP) for FAS, which can utilize text features to dynamically adjust visual features to make them independent of modality. In the vision branch, we propose a cross-modal spoofing enhancer (CMS-Enhancer), including a frequency extractor (FE) and a cross-modal interactor (CMI), aiming to map different modal attacks into a shared frequency space to perform cross-modal learning and suppress large differences between different modalities, allowing the model to effectively focus on subtle spoofing cues. In the text branch, we introduce languageguided patch alignment (LGPA) based on prompt learning, which further guides the image encoder to focus on patch-level spoofing representation through dynamic weighting of text features. Extensive experiments show that FM-CLIP is effective and outperforms SOTA methods on multiple multi-modal datasets. 
 
 Appendix:  


Decision: The paper proposes a novel Flexible Modal CLIP (FM-CLIP) approach for face anti-spoofing (FAS). The introduction of text modality features for Face Anti-Spoofing is a novel and promising direction worth exploring. The method demonstrates excellent performance in both multi-modal and flexible modality FAS protocols. After one round of rebuttal, all reviewers agreed to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper proposed a cross-modal feature enhancer implemented with adapter for FAS, as well as a learning approach based on text modality-invariant characteristics to reduce the learning of modality-specific features. The proposed method achieves nearly SOTA performance across various protocols.
strength_weakness: strength_weakness: - The introduction of text modality features for Face Anti-Spoofing is a novel and promising direction worth exploring.

- The method demonstrates excellent performance in both multi-modal and flexible modality FAS protocols.
limitations: limitations: - Flexible-modal Face Anti-Spoofing typically evaluates performance across various modality combinations, such as RGB+D and RGB+IR. However, this paper does not assess these combinations, focusing instead on the performance of single modalities (R, IR, D).
- Does the introduction of text modality information rely on external models? If so, might this lead to some unfairness when comparing with baseline methods? It is recommended to report the performance when only using the cross-modal feature enhancer.
- The article uses the frequency domain as the modality-shared feature space but lacks an explanation of the motivation behind this choice. Utilizing some visualization methods could help to demonstrate the validity of this approach.
- For the CMS module in the article, when training with three modalities together, is it necessary to perform feature enhancement pairwise?
paper_summary: paper_summary: The paper proposes a novel Flexible Modal CLIP (FM-CLIP) approach for face anti-spoofing (FAS). Unlike previous methods that suffer from modality-specific signal interference, FM-CLIP leverages text features to dynamically adjust visual features, making them modality-independent. The approach includes a Cross-Modal Spoofing Enhancer (CMS-Enhancer) and a Language-Guided Patch Alignment (LGPA) mechanism to enhance feature generalization and focus on spoofing cues. The method is validated through extensive experiments on multiple multi-modal datasets, demonstrating superior performance compared to state-of-the-art methods.
strength_weakness: strength_weakness: 1. FM-CLIP introduces a novel way to handle multi-modal face anti-spoofing by leveraging text features to make visual features modality-independent. This is a significant departure from traditional methods.

2. The paper includes extensive experimental validation on multiple datasets (SURF, CeFA, WMCA), showing the method's effectiveness and superior performance over existing state-of-the-art techniques.
limitations: limitations: 1. The proposed FM-CLIP method, especially with the inclusion of CMS-Enhancer and LGPA, introduces additional complexity and computational overhead. This could limit its practical deployment in real-time systems or environments with limited computational resources.

2. The paper lacks a detailed discussion on how the proposed method can be effectively integrated into real-world face recognition systems.

3. While ablation studies are conducted, more detailed analysis and comparison of the contributions of each component (e.g., CMS-Enhancer, LGPA) could provide deeper insights. Better understanding of each component's role would help in optimizing the model and potentially simplifying it without significant loss in performance.
paper_summary: paper_summary: This paper proposes a novel Flexible Modal CLIP (FM-CLIP), which borrows solutions from large-scale visual-language models (VLMs) to dynamically adjust visual features using text features to make them modality independent. In the vision branch, a CMS-Enhancer is proposed, including a frequency extractor (FE) and a cross-modal interactor (CMI), aiming to map attacks of different modalities into a shared frequency space to reduce interference from modality-specific signals and enhance deception cues. In the text branch, prompt-learning based LGPA is introduced to further guide the image encoder to focus on spoofing representation through dynamic weighting of text features. Through extensive experiments demonstrate the effectiveness of FM-CLIP on multiple multi-modal datasets.
strength_weakness: strength_weakness: 1. The motivation and method design of this article are novel, and it contributes a unique solution to the flexible modal FAS task, providing reference value for the community's subsequent research.
2. This article applies the multi-modal model to the flexible-modal FAS task for the first time. This article guides the model to learn generalization features by drawing on the characteristics of CLIP multi-modal large model text features that can adaptively weight visual features.
3. The writing of the article is clear and easy to understand, and the views expressed by the author can be understood very intuitively. The article also conducted a large number of comparative experiments and ablation experiments to analyze the independent contribution of each component.
4. CMS-Enhancer is used to allow multi-modal features to interact with each other, avoiding the design of multiple backbones in traditional methods. LGPA guides visual token features through text features, and the design idea of model constraining tokens is novel.
limitations: limitations: 1. When the loss is finally calculated, will all tokens in the visual features be subject to loss constraints, will it cause overfitting? Are the labels used for patch token supervision the same as the labels for CLS token features?

2. FM-CLIP may perform well on a specific data set, but its performance on unseen data sets, such as testing across scenarios, requires further verification.

3. For Frequency Extractor, DCT is used to obtain the frequency domain features of the image, and then convolution and sigmoid are used to multiply the original input. The paper mentioned the shared frequency domain space. How to understand the feature of different modes in the shared frequency domain space?
paper_summary: paper_summary: This paper presents language-guided multi-modal face anti-spoofing, which consists of two main strategies: Cross-Modal Spoofing Enhancer (CMS-Enhancer) and Language-Guided Patch Alignment (LGPA). CMS-Enhancer aims to capture spoofing clues by leveraging cross-modal learning from different modalities. LGPA aims adopt prompt learning to focus on patch-level spoofing representations. The authors claim that the proposed method is able to tackle the issue of missing modal scenario.
strength_weakness: strength_weakness: The authors conduct comprehensive experiments to evaluate their proposed method under fixed modal and missing-modal scenarios for multimodal face anti-spoofing.
limitations: limitations: Paper Weaknesses
1. The novelty of this paper is limited. The proposed method is composed of content from several previous methods, but the author did not cite or discuss these methods.
In Section 3.2, the concept of exploring spoof cues between RGB and frequency domains is not new and has already been proposed in a previous FAS method [A] (Please refer to Figure 2 of [A]). Next, the design Cross-Modal Interactor is highly similar to Cross-Modality Feature Fusion proposed in [B] (Please refer to Figure 2 of [B]). In Section 3.3, the proposed Language-Guided Patch Alignment is also not new. The similar idea about patch alignment learning already proposed in [C] (Please refer to Section 5 Patch Aligned Contrastive Learning of [C]). Therefore, the novelty of the proposed method appears to be incremental.
2. In Lines 389-450, the authors claim that the useful information can be selected by the uninformative gate map from other modalities. However, it is unclear why the selected information from other modalities would be useful to the current modality.
3. In Section 3.3 Vision-Language Alignment, it is unclear why the authors proposed to incorporate the text feature into the image features, as indicated by Equation (15).
4. Unclear description in experiment.
In Table 4, the authors demonstrate that the proposed method is capable of addressing the single modal scenario (e.g., only RGB or Depth). Since the proposed Cross-Modal Spoofing Enhancer extracts features from two different modalities, it remains unclear how the proposed method would operate when the input consists of a single modality.
5. The proposed method adopts a language model to guide the FAS model. The authors should discuss and compare it with recent language-guided methods, such as [D], [47], and [16].

[A] Learnable Multi-level Frequency Decomposition and Hierarchical Attention Mechanism for Generalized Face Presentation Attack Detection, WACV, 2022.

[B] Learning Polysemantic Spoof Trace: A Multi-Modal Disentanglement Network for Face Anti-spoofing, AAAI, 2023.
[C] Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning, CVPR, 2022.

[D] VL-FAS: Domain Generalization via Vision-Language Model For Face Anti-Spoofing, ICASSP, 2024.

[16] CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing, CVPR, 2024.

[47] Visual Prompt Flexible-Modal Face Anti-Spoofing, TMM, 2023.