# FF-MR: A DoH-Encrypted DNS Covert Channel Detection Method Based on Feature Fusion
## 5 Experimental Evaluation
### Performance Evaluation
#### 5.3.2 Results and Evaluation  
**Baselines. ** To measure the improvements achieved by FF-MR, we reproduce four baselines:  
* LightGBM [13] is a framework for implementing the Gradient Boosting Decision Tree (GBDT) algorithm, which supports efficient parallel training and has a faster training speed, lower memory consumption, and better accuracy. The method takes the statistical features of flow as input and outputs one of the five labels as a prediction;
* RF [8] is based on the Random Forest classifier, which adopts the same input and output as LightGBM [13]. The above two methods use the same statistical features as in Table 2;
* HAST-II [42] takes the first 4096 bytes of the session as input and combines CNN with LSTM to learn the spatial features and temporal features of the session bytes, respectively; it then uses a softmax classifier to perform five classifications on the spatio-temporal features;
* The input of CENTIME [43] is the same as ours: both use the statistical features and the first \(n\) bytes of the session. The difference is that they use the self-encoder to reconstruct the statistical features and the residual neural network with the same structure as ours to extract the byte sequence features and then concatenate the two inputs to the fully connected network for classification. **Results. ** As shown in Table 5, FF-MR achieved scores of 99.72%, 99.73%, and 0.9978 on Macro_P, Macro_R, and Macro_F1, respectively, demonstrating that the detection performance is better than the other four methods both in macro-averaging and metrics of identification of each category. Next, we will present a detailed comparative analysis of the experimental results though Table 5 and Figure 10.  
**Evaluation. ** As shown in Figure 10, comparing the results of LightGBM and RF, HAST-II, and LightGBM and RF, which use statistical features alone, it can be seen that these methods have higher recall and their overall performance is similar; HAST-II, which uses session bytes as input, has a higher precision. Although the results of the above three methods reflect the different advantages of the two features, the macro-averaging metrics are poor, and their F1-scores are only about 0.96. In contrast, the detection performance of FF-MR and CENTIME, which combine the two features, is better than the other three methods. Figure 9: Visualization of t-SNE dimension reduction in traffic features. As shown in Table 5, in terms of the metric Macro_F1, FF-MR improves over LightGBM and RF by 4.56% and 4.35%, respectively, and 3.62% over HAST-II. The results of FF-MR in the five classifications are also significantly higher than the other three methods, especially in the identification of encrypted DNS covert channel traffic generated by the iodine and dnscat2 tools. FF-MR improves F1-Score when using iodine and dnscat2 by 6.06% and 8.22% over LightGBM, by 5.81% and 8.12% over RF, and by 5.67% and 11.32% over HAST-II, respectively, indicating the important role of the combined use of features for encrypted DNS covert channel identification. Our analysis is that LightGBM and RF belong to traditional machine learning algorithms, which are variations of decision tree algorithms. Therefore, we can infer that the decision tree algorithm does not achieve accurate classification and that decision tree integration algorithms cannot simply improve the performance of detection by optimizing the node splitting algorithm, i.e., XGBoost in LightGBM, or by increasing the number of decision trees (RF). The reason is that non_DoH, benign_DoH, and the three kinds of malicious DoH traffic are closer in the hyperplane, and multiple decision trees and the  
Figure 10: FF-MR vs. other methods on macro-averaging Metrics.