# Modal-Guided Multi-Domain Inconsistency Learning for Face Forgery Detection
## 3 The Proposed Method
### The Architecture  
**Problem formulation:** Suppose the face-related input with different combinations of modalities is denoted by \(m_{x},x\in\{i,v,a,iv,iva\}\), where \(i,v\), and \(a\) represent the image, video, and audio modalities, respectively. Similarly, \(m_{iv}\) denotes bimodal data, while \(m_{iva}\) represents trimodal input. We denote the triplet branches as \(B_{1}\) (2D CNN), \(B_{2}\) (3D CNN), and \(B_{3}\) (the audio feature extraction net). Each branch consists of an encoding network and a classification network. This design empowers the MGDL-Net to independently  
Figure 2: The architecture of the proposed framework. extract intra-domain forgery clues through individual branches, while simultaneously enhancing its ability to capture comprehensive inter-domain tampered information by integrating multiple branches. Specifically, the encoder \(EC_{n}\), where \(n\in 1,2,3\), extracts the discriminative embedding \(f_{n}\) from the input \(m_{x}\). Subsequently, the classifier \(FC_{n}\) maps the feature representation \(f_{n}\) to binary predictions \(p_{n}\) (fake or real). **Unimodality inference:** In this case, the face-related input \(m_{x}\) is composed of unimodal data, such as audio, image, or video. Based on the modality of \(m_{x}\), the MGDL-Net adaptively transfers it to the corresponding branch to generate predictions \(p_{n}\). We use the audio input \(m_{a}\) as an example to illustrate the MGDL-Net pipeline for unimodal face-related content. The inference can be formulated using Equation (1), where L is the \"Librosa\" audio processing library, which converts the original audio input \(m_{a}\) into an MFCC spectrogram. The encoder \(EC_{3}\) extracts the frequency domain features from the MFCC spectrogram, and the classifier \(FC_{3}\) maps these features to a predictive logit \(p_{3}\) (fake or real). The inference process for the image and video modalities is similar to that for the audio modality. \[p_{3}=FC_{3}(EC_{3}(L(m_{a}))) \tag{1}\]  
**Bimodality inference:** In a bimodal scenario, we assume that the input to the MGDL-Net is \(m_{ir}=\{i,v,p\}\), where \(p\in[0,1]\) serves as the prediction indicating whether \(m_{ir}\) is forged. The image \(i\) is extracted from the current video clip \(v\), and then the MGDL-Net activates the spatial branch \(B_{1}\) and the temporal branch \(B_{2}\) to reorganize a new combined network for extracting bimodal forgery clues. The inference process can be summarized as follows:  
\[f_{1}\in R^{\frac{T}{n},C,H,W}=\delta(EC_{1}(m_{i})) \tag{2}\]  
\[f_{2}\in R^{T,C,H,W}=EC_{2}(m_{v}) \tag{3}\]  
\[p_{2}=FC_{2}(\varphi(f_{1},f_{2})) \tag{4}\]  
where \(\varphi\) indicates spatial-temporal feature fusion, and \(\delta\) donates a stacking operation. The encoder \(EC_{1}\) in branch \(B_{1}\) adopts a 2D CNN backbone to encode the finer-grained spatial features \(f_{1}\) of image \(i\) with a shape of \([C,H,W]\). Subsequently, branch \(B_{1}\) stacks the sequence of image \(i\) along the temporal dimension to endow \(f_{1}\) with the capacity for temporal perception. In contrast, the encoder \(EC_{2}\) in the \(B_{2}\) branch employs a 3D CNN backbone to capture the long-range temporal representations _retained_\(f_{2}\) of the video clip, which has a shape of \([T,C,H,W]\). The feature representations from the 2D CNN and 3D CNN are then transformed into the same dimensions \([C,H,W]\) to facilitate subsequent spatiotemporal feature aggregation. The classifier \(FC_{2}\) predicts \(p_{2}\) for the bimodal input \(m_{ir}\) by leveraging the fused features from the spatial-temporal branches. The inference process for the bimodal input \(m_{ir}\) is similar to that for \(m_{ir}\). **Multi-modality inference:** We have the face-related input \(m_{iora}=\{i,v,a,p_{v},p_{a},p\}\), which comprises both the video and audio modalities, along with their respective predictions, denoted as \(p_{v}\) and \(p_{a}\). Moreover, \(p\) indicates the authenticity of the entire input and is labeled as \"fake\" if either modality has been tampered with.