# Modal-Guided Multi-Domain Inconsistency Learning for Face Forgery Detection
## 4 Experiments
### Discussion and Analysis  
In this section, we first discuss the visualizations generated by Grad-CAM [52] to reveal evidence that is determinant of the final decision. Then, we focus on the representation distributions processed using heterogeneous inconsistency learning and analyze the ability of our method in multi-modal learning via the t-SNE visualization [53]. **Grad-CAM visualization:** Our framework makes decisions based on feature maps encoding heterogeneous inconsistencies. The Grad-CAM visualizations of the spatial and temporal branches help us understand the decision-making mechanism of spatial branching better. The images in Figure 5c indicate that Grad-CAM reversely maps a weighted heat map to the original image. The region in deeper red is more important to the final decision of the spatial branch. Through the Grad-CAM visualization results, it can be found that the image branch mainly focuses on regions with intra-frame spatial inconsistency features, such as facial blending distortion and unnatural textures. Obviously, the spatial branch concentrates on the facial region exhibiting pronounced traces of forgery, which are easily detected using CNN backbones in the spatial domain. To elucidate the decision-making mechanism of our model further, the local regions that the temporal branch concentrates on to make the final prediction (\"real\" or \"fake\") are visualized in Figure 6. For each video clip, we generate a Grad-CAM visualization for the adjacent frames, and this can reflect how the inter-frame temporal inconsistency features affect the final decision-making using the proposed method. From Figure 6, we can observe that our model tends to focus on areas where modal inconsistency is likely to arise, such as the eyes and mouth, because the modal consistency is broken when movements \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{4}{c}{**HIL Element**} & \multirow{2}{*}{**Accuracy (All)**} \\ \cline{2-2} \cline{4-5}  & **A-HIL** & & **E-HIL** & & **M-HIL** \\ \hline \(M_{6}\) & \(\times\) & \(\times\) & \(\times\) & 91.57 \\ \(M_{7}\) & \(\checkmark\) & \(\times\) & \(\times\) & 92.28 \\ \(M_{8}\) & \(\times\) & \(\checkmark\) & \(\times\) & 92.14 \\ \(M_{9}\) & \(\times\) & \(\times\) & \(\checkmark\) & 92.33 \\ \hline \(M_{10}\) & \(\checkmark\) & \(\checkmark\) & \(\times\) & 92.81 \\ \(M_{11}\) & \(\times\) & \(\checkmark\) & \(\checkmark\) & 92.76 \\ \hline \(M_{12}\) & \(\checkmark\) & \(\checkmark\) & \(\checkmark\) & 93.34 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Ablation studies on the components of heterogenous inconsistency learning (HIL), where \(\checkmark\) indicates the component is included and \(\times\) indicates it is excluded. of the lips or eyes do not match the emotions reflected by the corresponding audio. This demonstrates that the features captured by the spatial and temporal branches compensate for each other to form more powerful representations. Compared with the spatial branch, the temporal branch searches for clues of manipulation primarily fine-grained areas of emotional expression, e.g., the mouth and eyes. Just as we have speculated, the temporal domain serves as a complementary to the other two modalities, and they contribute to each other via heterogeneous inconsistency learning to exploit comprehensive forgery cues. Figure 5: Visual explanations of predictions made by the spatial branch: (**a**) the original image; (**b**) a weight map; (**c**) the Grad-CAM visualization results. Figure 6: Visual explanations of predictions made by the temporal branch. **t-SNE visualization:** To investigate the feature distributions from different models, the t-SNE visualizations are presented in Figure 7, where four models are delineated, including the spatial inconsistency learning model (Figure 7a), the spatial-temporal inconsistency learning model (Figure 7b), the proposed method with single labels (Figure 7c), and the proposed method with multi-labels (Figure 7d).