# Modal-Guided Multi-Domain Inconsistency Learning for Face Forgery Detection
## 4 Experiments
### Ablation Studies  
To examine how different combinations of modalities and different levels of inconsistent features contribute to the performance of the proposed method, we conduct a series of ablation studies in this section. A total of 12 variants of our model (\(M_{1}-M_{12}\)) are implemented, and two groups of comparisons are executed on the DFDC dataset. The first group (\(M_{1}-M_{6}\)) is used to verify the effect of different combined modalities on the accuracy of face forgery detection. The other group (\(M_{6}-M_{12}\)) reveals the contributions of inconsistent features to HIL. **Effectiveness of the modalities:** To examine the impacts of the modalities, we compare the proposed method encoding different combinations of modalities, including three unimodality models (image (\(M_{1}\)), video (\(M_{2}\)), and audio(\(M_{3}\)) ), bimodality versions (image-video(\(M_{4}\)), video-audio(M5)), and a model jointly trained on multi-modalities but with the simple \"concat\" fusion (without HIL). The results are described in Table 7. It is clear from the table that the single image modality model \(M_{1}\) cannot achieve a satisfying performance since it leaves temporal and frequency cues out of consideration, and nor does \(M_{2}\). In contrast, the integration of the video modality with the image modality (\(M_{4}\)) upgrades the accuracy by a margin of 1.46% and 1.09% compared to the independently trained single-modality models \(M_{1}\) (image modality) and \(M_{2}\) (video modality), respectively. The proposed framework achieves the best accuracy of 91.57% with spatial-temporal-frequency multi-domain modeling, which demonstrates the encoding of multiple modalities. Also, the feature extraction networks for image and video capture finer spatial information and temporal relationships, respectively. The audio feature extraction network obtains frequency domain cues, which assists our framework in learning complementary and comprehensive inconsistency features. However, the fusion of the heterogeneous features from three different domain sources is not simple, so simple concatenation may not be the optimal option because it does not take sparse inconsistencies into account. **Effectiveness of heterogeneous inconsistency learning:** We summarize the ablation studies on HIL in Table 8. HIL incorporates three components: A-HIL, E-HIL, and M \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{3}{c}{**Domain**} & \multicolumn{3}{c}{**Accuracy**} \\ \cline{2-7}  & **Image** & **Video** & **Audio** & **Vis** & **Aud** & **All** \\ \hline \(M_{1}\) & ✓ & \(\times\) & \(\times\) & 89.07 & - & - \\ \(M_{2}\) & \(\times\) & ✓ & \(\times\) & 89.46 & - & - \\ \(M_{3}\) & \(\times\) & \(\times\) & ✓ & - & 96.79 & - \\ \hline \(M_{4}\) & ✓ & ✓ & \(\times\) & 90.53 & - & - \\ \(M_{5}\) & \(\times\) & ✓ & ✓ & 89.58 & 96.71 & 90.65 \\ \hline \(M_{6}\) & ✓ & ✓ & ✓ & 90.60 & 96.82 & 91.57 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Ablation studies on the components of the proposed network, where ✓ indicates the component is included and \(\times\) indicates it is excluded. HIL. The model without any inconsistency learning is \(M_{6}\), which achieves an accuracy of 91.57%. Based on this model, A-HIL is added to \(M_{7}\), and this boosts the accuracy to 92.28%. A-HIL allows the model to learn intra-frame inconsistency signals at the pixel level, which is beneficial for exploiting spatial inconsistency features. E-HIL and M-HIL encourage the model to learn inconsistent features from temporal and frequency perspectives, which boosts the accuracy by gains of 0.57% and 0.76%, respectively. Then, we add E-HIL to \(M_{7}\), and the performance is increased from 92.28% to 92.81%. This is because the model \(M_{10}\) not only encodes the temporal inconsistencies but also reserves rich spatial inconsistency clues. It captures the modified traces from a new perspective, which brings a performance improvement.