# Modal-Guided Multi-Domain Inconsistency Learning for Face Forgery Detection
## 3 The Proposed Method
### Heterogeneous Inconsistency Learning  
The MGDL-Net coarsely aggregates cross-domain features with different distributions from separate branches using either \"concatenation\" or \"summation\". However, this feature fusion operation overlooks the dependency relationships among the spatial, temporal, and frequency domains, potentially leading the network to learn irrelevant facial content from the input. While this content may be useful for facial recognition, it is ineffective for deepfake detection. To address this issue, we propose heterogeneous inconsistency learning (HIL), which captures more comprehensive inconsistent features from a hierarchical perspective, including intra-frame spatial inconsistencies (A-HIL), inter-frame temporal inconsistencies (E-HIL), and cross-modal visual-audio inconsistencies (M-HIL). **Intra-frame-inconsistent feature learning:** The structure of A-HIL is shown in Figure 3a. As defined in Section 3.1, the 2D embedding of the image group is \(f_{1}\). We first apply convolution and batch normalization to \(f_{1}\) to generate a new feature map \(f_{tmp1}\), as described in Equation (9), where \(\sigma\) and \(F_{conv}\) represent the operations of batch normalization and convolution, respectively. \[f_{tmp1}=F_{conv}(\sigma(F_{conv}(\sigma(f_{1})))),f_{1}\in R^{C,H,W} \tag{9}\]  
\[e\in R^{C,1}=\varsigma(\sigma(F_{conv}(\rho(f_{tmp1})))) \tag{10}\]  
Then, we design the inconsistency attention, constructed using a global pooling layer, a \(1\times 1\) convolution layer, a batch normalization layer, and a sigmoid layer on the feature map \(f_{tmp1}\), as delineated in Equation (10), where \(\varsigma\) represents the sigmoid layer and \(\rho\) denotes the global pooling layer. Finally, to activate the ability of A-HIL to capture the intra-frame inconsistency, we multiply the input feature \(f_{1}\) and the inconsistency attention weight \(e\), as formulated in Equation (11). A-HIL focuses attention on the finer spatially inconsistent features, which enhances the ability of the network to capture intra-frame differences. \[f_{1}^{\prime}\in R^{C,H,W}=e*f_{1} \tag{11}\]  
**Inter-frame-inconsistent feature learning:** To exploit the temporal dependencies across adjacent frames, we design an inter-frame inconsistency learning module (E-HIL), the structure of which is revealed in Figure 3b. We first feed the embedding \(f_{2}\in R^{\mathbb{C}_{1},T,H,W}\) into a 3D pooling layer with a kernel size of \([T,1,1]\) and then fuse its output with \(f^{{}^{\prime}}_{1}\) to encode both the temporal and spatial inconsistencies into \(f_{tmp2}\in R^{\mathbb{C}+\mathbb{C}_{1},H,W}\). Then, two convolutional layers are applied to \(f_{tmp2}\) to regenerate a feature map \(f^{{}^{\prime}}_{tmp2}\in R^{\mathbb{C},H,W}\). Convolutional operations are crucial for E-HIL, as they mitigate the semantic discrepancies arising from different branches with diverse distributions. Subsequently, we transform the feature map \(f^{{}^{\prime}}_{tmp2}\) into a one-dimensional vector \(Q\in R^{\mathbb{C},Z}\), as described in Equation (14). \[f_{tmp2}\in R^{\mathbb{C}+\mathbb{C}_{1},H,W}=\rho(\rho(f_{2}),f^{{}^{\prime}} _{1}) \tag{12}\]  
\[f^{{}^{\prime}}_{tmp2}\in R^{\mathbb{C},H,W}=F_{conv}(F_{conv}(f_{tmp2})) \tag{13}\]  
\[Q\in R^{\mathbb{C},Z}=Reshape(f^{{}^{\prime}}_{tmp2}\in R^{\mathbb{C},H,W}) \tag{14}\]  
Afterward, we apply a softmax operation to the correlation between the vector \(Q\) and its transpose vector \(Q^{T}\) to generate an inconsistency indicator \(Q^{\prime}\) (Equation (15)), which represents the difference across the temporal features.