## 4 Experiments
### Training Evaluation
#### 4.2.3 Evaluation Metrics  
It is extremely useful for measuring Recall, Precision, F1-score, and most importantly AUC-ROC curves. It can be calculated through the confusion matrix shown in Table 1.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Predicted**} \\ \cline{3-4}  & & **Negative** & **Positive** \\ \hline \multirow{2}{*}{Actual} & Negative & True Negative (TN) & False Positive (FP) \\ \cline{2-4}  & Positive & False Negative (FN) & True Positive (TP) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Confusion matrix.  
Each column in the confusion matrix represents instances in a predicted class, while each row represents instances in an actual class. The positive class refers to the original and unmanipulated images, while the negative class represents the manipulated ones. True positive (TP) represents the outcome where the model correctly predicts the positive class. Similarly, true negative (TN) represents the outcome where the model correctly predicts the negative class. False positive (FP) represents the outcome where the models incorrectly predict the positive class. And false negative (FN) represents the model incorrectly predicts the negative class.  
Precision measures the percentage of positive identifications was actually correct. Precision is defined as (5):  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{5}\]  
Recall measures the percentage of actual positives was identified correctly. Recall is defined as (6):  
\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{6}\]  
F1-score is the harmonic mean of precision and recall. It is difficult to compare two models with low precision and high recall or vice versa. F1-score helps to measure Recall and Precision at the same time. F1-score is defined as (7):  
\[\text{F1-score}=\frac{2*\text{Precision}*\text{Recall}}{\text{Precision}+ \text{Recall}} \tag{7}\]  
True positive rate (TPR) is a synonym for recall and is defined as (8):  
\[\text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{8}\]  
False positive rate (FPR) is defined as (9):  
\[\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}} \tag{9}\]  
Accuracy measures the percentage of number of correct predictions. Accuracy is defined as (10):  
\[\text{Accuracy}=\frac{\text{Number of correct predicitons}}{\text{Total number of predictions}}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}} \tag{10}\]  
An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model. This curve plots with two parameters: true positive rate (TPR) and false positive rate (FPR). AUC (area under the ROC curve) measures the two-dimensional area under the ROC curve. AUC provides an aggregate measure of performance across all possible classifications. With AUC, we can measure how well predictions are ranked, rather than their absolute values since AUC is scale-invariant.  
Training results of the classifier network are shown in Figures 6 and 7. This step basically becomes a simple case of classification model with two classes (\"Real\" and \"Fake\"), therefore it doesn't take a long time in this section. Most of the time will be spent on the distillation set extraction. The training outputs of the Celeb-DF v2 dataset is shown in Figure 6 and DFDC dataset is shown in Figure 7. The performance of the training set is usually better than the validation set, for example, the accuracy of the training set is approximately equal to \"1\". The accuracy of the validation set is not too much lower than the training set, detailed values are listed in Table 2. This demonstrates that the model has partially solved the overfitting problem mentioned in Section 3.1.  
We can also measure frame-by-frame quality using feature sets at the output of the classifier network. However, we are primarily interested in DeepFake detection for video, so the results of classifier network will be used in the combination and classification module.