# Analysis of Score-Level Fusion Rules for Deepfake Detection
## 4 Experimental Results
### Results
#### 4.2.2 Cross-Dataset Scenario  
The cross-dataset analysis is of the utmost importance because it allows evaluating whether the fusion of multiple classifiers can increase the ability to generalize over \"never seen before\" manipulations. Since the detectors are trained on the same dataset, the contribution of the fusion in moderating the individual matching score distributions can be better isolated without referring to complementarity induced by other causes, such as the training data employed. Similarly to the previous section, we reported the cross-dataset results using the performance metrics of Table 3.  
As in the intra-dataset scenario, we observe a similar trend in terms of performance of the single models, with the EfficientNetB4 ST showing the best values in terms of the AUC, EER and Cohen's d. However, this is not confirmed for all the decision thresholds since the  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Model/Fusion Method** & **AUC** & **Cohen’s d** & **EER** & **FPR@FNR1\%** & **FNR@FPR1\%** \\ \hline EfficientNetAutoAttB4 ST & 0.688 & 0.614 & 0.365 & 0.968 & 0.918 \\ \hline EfficientNetAutoAttB4 & 0.692 & 0.625 & 0.367 & 0.963 & 0.889 \\ \hline EfficientNetB4 ST & 0.697 & 0.635 & 0.357 & 0.965 & 0.907 \\ \hline EfficientNetB4 & 0.684 & 0.571 & 0.371 & 0.969 & 0.898 \\ \hline Xception & 0.632 & 0.428 & 0.384 & 1.000 & 0.936 \\ \hline ResNet & 0.398 & 0.303 & 0.579 & 1.000 & 1.000 \\ \hline Avg & 0.690 & 0.636 & 0.357 & 0.983 & 0.908 \\ \hline Bayes & 0.626 & 0.421 & 0.389 & 1.000 & 0.931 \\ \hline Prod & 0.629 & 0.289 & 0.389 & 1.000 & 0.931 \\ \hline Max & 0.448 & 0.011 & 0.554 & 0.984 & 1.000 \\ \hline Min & 0.620 & 0.334 & 0.389 & 1. EfficientNetAutoAttB4 performed better for decision thresholds in which one of either the False Negative Rate or the False Positive Rate was equal to 1%. Among the non-parametric fusion rules, the best was still the simple Average, highlighting a noticeable difference from the other non-parametric fusion rules, especially in terms of Cohen's d. However, in this scenario, the simple Average was no better than the best of the detectors, thus, highlighting the limitations of non-parametric methods. Regarding the approaches based on weighted averages, we notice that the correlation-based average was able to provide the best results (even better than the ones related to the single detectors) followed by the MI-based one, which showed similar performance. In contrast to the intra-dataset scenario, Complement NB was the best of the fusion methods based on machine-learning models. The results in the cross-dataset scenario show a general improvement with respect to the single models through the parametric fusion methods. In particular, the best fusion methods were the correlation-based average and the Complement NB. Instead, the non-parametric fusion rules were not able to improve the performance compared to the most performing detection models. In Figure 6, the ROC curves related to the cross-dataset scenario (on the DFDC dataset) are shown for the three adopted fusion typologies, together with the ones related to the single models (Figure 6a-c). The most performing methods of each fusion approach are compared with the single model showing the highest AUC value (Figure 6d). By visual inspection, it is possible to confirm the slight improvement through the parametric approaches with respect to the single model and the deterioration of performance through the non-parametric one. Figure 6: ROC curves on the DFDC dataset comparing the single deepfake detectors with their fusion through non-parametric approaches (**a**), the approaches based on weighted averages (**b**), the ones based on machine-learning models (**c**) and comparing the most performing methods of each category with respect to the obtained AUC (**d**). The analysis of the distributions of scores resulting from the investigation in the cross-dataset scenario (Figure 7) allows us to add some further insights for a designer.