# DGA Domain Name Classification Method Based on Long Short-Term Memory with Attention Mechanism
## 5 Experimental Evaluation
### Work Comparison and Discussion  
[PERSON] et al. [8] proposed a DGA domain name detection method. This work was inspired by the observation that the difference in character distribution between the normal domain name and the DGA domain name is quite large. They used the characters in the domain name and the distribution characteristics of the 2-gram character set, and finally used the edit distance and Jaccard distance [13] algorithm. Although [PERSON] et al. [16] proposed a method for DGA detection and classification using LSTM algorithm in 2016, they only detected and classified the strings of domain names, and it has been verified by experiments to have very good detection and classification effects. Compared  
Figure 4: Confusion Matrix for the long short-term memory (LSTM) model with attention mechanism. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Domain Type** & **Precision** & **Recall** & \(\mathbf{F_{1}}\) **Score** & **Support** \\ \hline nymaim & 0.3988 & 0.1115 & 0.1743 & 601 \\ ranbyus & 0.4672 & 0.8455 & 0.6018 & 1346 \\ murofet & 0.7641 & 0.7207 & 0.7418 & 1443 \\ pykspa & 0.8972 & 0.7207 & 0.7994 & 1393 \\ lucky & 0.0000 & 0.0000 & 0.0000 & 574 \\ shiotob & 0.9751 & 0.9251 & 0.9494 & 1268 \\ banjori & 0.9998 & 1.0000 & 0.9999 & 43,808 \\ necurs & 0.6651 & 0.1722 & 0.2735 & 3241 \\ Cryptolocker & 0.1000 & 0.0018 & 0.0035 & 562 \\ simda & 0.9264 & 0.9669 & 0.9462 & 1418 \\ dyre & 1.0000 & 1.0000 & 1.0000 & 797 \\ Post & 0.9994 & 0.9998 & 0.9996 & 6644 \\ tinba & 0.9259 & 0.9920 & 0.9578 & 6498 \\ qakbot & 0.7862 & 0.5013 & 0.6122 & 1973 \\ ramnit & 0.4688 & 0.7525 & 0.5777 & 4856 \\ Alex & 0.9898 & 0.9956 & 0.9927 & 91,119 \\ \hline avg/total & 0.9505 & 0.9514 & 0.9458 & 167,541 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental results of our method. with the method in this paper, the attention mechanism is lacking. We compare our method with [PERSON] et al. [8] and [PERSON] et al. [16], and the results are shown in Table 4:  
It can be seen from Table 4 that our method has a much higher precision, recall, and \(F_{1}\) score compared with the work of [PERSON] et al. [8]. Compared with the work of [PERSON] et al. [16], the improvement of the method is not obvious. Therefore, we conducted a 10-fold cross-validation experiment to compare the method of [PERSON] et al. [16] with our method. The sample set was divided into 10 parts on average, and one of them was taken as a test set each time. One of the remaining nine samples was used as a verification set, and the remaining eight were used as a training set. The ratio of the final training set, verification set, and test set is 8:1:1. Each experiment was carried out for 20 epochs of training. After training, the test set was used to test the classification model. The results of 10 experiments are shown in Figure 5. As can be seen in Figure 5, the precision of our method is superior than [PERSON]'s method [16] in each experiment. This proves that the weights of different characters in different positions in the DGA domain name are different, and the attention mechanism is necessary for DGA classification.