# Modal-Guided Multi-Domain Inconsistency Learning for Face Forgery Detection
## 1 Introduction  
The development of deepfake techniques (such as Deepfakes [1] and StarGAN [2]) involves a range of modalities of manipulated face-related content, including images, videos, and audio. Of these, tampered video footage with corresponding audio is the most convincing and threatening. Therefore, researching modal-guided face forgery detection is becoming an increasingly urgent task. A variety of works have focused on deepfake detection, which can be divided into unimodal-based and bimodal-based detection. Some researchers [3, 4, 5, 6] have attempted to achieve deepfake detection based on unimodal methods, i.e., only extracting spatial domain artifacts from the fake image or detecting frequency-domain-manipulated cues in tampered audio. However, the development of manipulation methods may produceextremely realistic forged images, and the subtle spatial artifacts are difficult to perceive in static images. Though deepfake techniques have achieved great success in image forgery, they struggle to eliminate the unnatural facial inconsistency in the temporal dimensions because they alter the video footage frame by frame. As illustrated in the top row \(I\) of Figure 1, artifacts may not be obvious from unimodal information, but when investigating multi-modal spatial, temporal, and frequency domain information, the unmatched fake cues can be observed. Therefore, inconsistency in the temporal domain can serve as an efficient cue for detecting deepfakes. Other works [7; 8; 9] have utilized bimodal information to identify input video as \"real\" or \"fake\". Most of them have integrated spatial domain cues with temporal inconsistency, voice features, depth information, etc., to improve the performance in deepfake video detection. The problem is that these methods only detect input samples with fixed and known modalities, while in practical applications, the modalities of face-related content that have been manipulated are unknown. In view of the previous methods, we argue that it is essential to take diverse modal inputs into consideration during deepfake detection. Therefore, we develop an MGDL-Net (Modal-Guided Domain Learning Network), which consists of a spatial branch, a temporal branch, and a frequency branch. The MGDL-Net is introduced to adaptively select corresponding domain inconsistency extraction branches according to the modality of the input content. This modal-guided mechanism not only allows the MGDL-Net to identify face-related inputs with diverse modalities but also equips our model with the ability to perceive both intra- and inter-domain traces of tampering. Another issue is that current deepfake detection methods extract features with a large amount of useless interference information while ignoring significant intra- and inter-domain inconsistencies from the spatial-temporal frequency perspectives. The intra-domain inconsistency used in most of the current methods is also an essential discriminative clue for face forgery detection. For instance, in Figure 1, the inter-domain consistency is broken when any of the modalities are modified. The inter-modal information encodes the visual signals together with audio cues to capture spatial-temporal frequency domain inconsistencies, facilitating accurate detection. Therefore, the ability to exploit inconsistent cross-domain relations in forged inputs is the key to face forgery detection. However, to extract more general domain inconsistency, several challenges have yet to be addressed, with the most notable being extracting a unified and general representation from inconsistent features with different distributions. To address the above issues, we propose using heterogeneous inconsistency learning (HIL) to capture powerful intra-domain inconsistencies  
Figure 1: Observations in multi-modal face forgery data. Rows annotated with \(I\), \(V\), and \(A\) from top to bottom denote the image modality, video modality, and audio modality.