# Unknown Traffic Recognition Based on Multi-Feature Fusion and Incremental Learning
## 4 Evaluation
### The Percentage of Unknown Traffic Classes  
In order to further demonstrate that the proposed method also performs well with different numbers of unknown classes, the following experiments were performed. We introduced the concept of _openness_[51] to characterize the ratio of unknown traffic to known traffic in the testing dataset. Openness is mathematically defined as:  
\[openness=1-\sqrt{2\times\frac{C_{T}}{C_{R}+C_{E}}} \tag{28}\]  
where \(C_{T}\) is the number of the known classes used in the training dataset, \(C_{E}\) is the number of all traffic classes in the testing dataset, and \(C_{R}\) is the number of traffic classes to be recognized. In this experiment, \(C_{R}\) was equal to \(C_{E}\) because of the need to classify all traffic classes in the testing dataset.  
As can be seen from the above equation, openness reflects the amount of knowledge available for training the detection model. An openness of 0 indicates a completely known classification problem, while a high openness means that we have less knowledge of what is known for training, and distinguishing known traffic from unknown traffic is more difficult. To investigate the effect of openness on model performance, we conducted experiments under different openness settings. Figures 11-13 demonstrate the performance of several unknown traffic classification methods mentioned in the previous section under different openness, with the measures of _Acc_, _FTF_, and \(F\)1.  
Figure 11: Accuracy of unknown traffic classification under different openness.  
Figure 12: FTF of unknown traffic classification under different openness values.  
Figure 13: F1 score of unknown traffic classification under different openness.  
From these figures, it can be seen that our proposed method significantly outperformed the other three methods at high openness. To be specific, the accuracy basically tended to decrease as the openness increased in all four scenarios, but the decreasing trend of our proposed model was slower and even slightly increased in scenario C. In other words, while the accuracy of the four methods under low openness did not much differ, the accuracy of our proposed model under higher openness was much higher than that the other methods, especially for the detection of unknown attacks in scenario C and scenario D. In the test of FTF metrics, our method achieved the best performance at all openness values in the four scenarios, which also means that the method is able to balance the true positive rate of unknown traffic and the false positive rate of known traffic. As for the comparison of F1 score, the scores of OPEN-CNN and of our proposed MI-UTR under scenario A were both relatively close for different openness values; OPEN-CNN performed better at low openness for scenario B, but the performance rapidly decreased with increasing openness; the CNN-GR model under scenario C experienced a similar problem. The MI-UTR model in scenario D always maintained an advantage under all openness values. In summary, our proposed MI-UTR model has strong robustness in various scenarios and for various openness values.