# Novel Feature Extraction Method for Detecting Malicious MQTT Traffic Using Seq2 Seq
## 7 Experimental Results
### Analysis  
In this section, we analyze the effects of not using each feature. First, when the source port index was not used, the accuracy of detection was reduced by 0.2% to 99.77%, as shown in Figure 14.  
As shown in Table 5, DoS and brute-force attacks were not effectively detected. Therefore, we demonstrated the significance of a source port index in detecting DoS and brute-force attacks because clients use many source ports. Second, when the TCP length was not used, the accuracy of detection was reduced by 0.24% to 99.73%, as shown in Figure 15. In addition, flood attacks were not detected, as shown in Table 6. We demonstrated the significance of the TCP length in detecting flood attacks. Third, when the message type was not used, the accuracy of detection was reduced by 0.04% to 99.93%, as shown in Figure 16. As shown in Table 7, 13 malformed attacks were classified as DoS attacks. Therefore, we demonstrated the significance of the message type in detecting malformed attacks. Fourth, when keep alive was not used, the accuracy of detection was reduced by 0.01% to 99.96%, as shown in Figure 17. As shown in Table 8, DoS attacks were classified as SlowITe attacks, and malformed attacks were classified as SlowITe attacks. We demonstrated that keep alive is important for detecting SlowITe attacks because it is used in SlowITe attacks. Figure 14: Accuracy when a source port index is not used. \begin{table}
\begin{tabular}{c c c c c c c} \hline  & & \multicolumn{6}{c}{**Predicted**} \\ \cline{3-8}  & & **Normal** & **BF** & **DoS** & **Flood** & **SlowITe** & **Malformed** \\ \hline \multirow{4}{*}{Actual} & Normal & 4419 & - & - & - & - & - \\ \cline{2-8}  & BF & 10 & 588 & - & - & - & - \\ \cline{2-8}  & DoS & 12 & 3 & 18,637 & - & - & 1 \\ \cline{2-8}  & Flood & - & - & 9 & 53 & - & - \\ \cline{2-8}  & SlowITe & - & - & - & - & 408 & - \\ \cline{2-8}  & Malformed & 7 & - & 7 & - & - & 726 \\ \hline \end{tabular}
\end{table} 
Table 5: Confusion matrix when a source port index is not used. Fifth, when a connection ACK was not used, the accuracy remained at 99.97%, as shown in Figure 18. However, as shown in Table 9, DoS and malformed attacks were classified as brute-force attacks. Therefore, we demonstrated the significance of a connection ACK in detecting brute-force attacks. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Predicted**} \\ \cline{2-7}  & \multicolumn{1}{c}{**Normal**} & **BF** & **DoS** & **Flood** & **SlowTle** & **Malformed** \\ \hline \multirow{4}{*}{Actual} & Normal & 4419 & - & - & - & - & - \\ \cline{2-7}  & BF & - & 606 & - & - & - & - \\ \cline{2-7}  & DoS & - & - & 18,653 & - & - & - \\ \cline{2-7}  & Flood & 62 & - & - & - & - & - \\ \cline{2-7}  & SlowTle & - & - & - & - & 408 & - \\ \cline{2-7}  & Malformed & 2 & - & 2 & - & 1 & 735 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Confusion matrix when the TCP length is not used. Figure 16: Accuracy when the message type is not used. Figure 15: Accuracy when the TCP length is not used. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & & \multicolumn{6}{c}{**Predicted**} \\ \cline{2-8}  & & **Normal** & **BF** & **DoS** & **Flood** & **SlowTe** & **Malformed** \\ \hline \multirow{4}{*}{Actual} & Normal & 4419 & - & - & - & - & - \\ \cline{2-8}  & BF & - & 606 & - & - & - & - \\ \cline{2-8}  & DoS & - & - & 18,652 & - & - & 1 \\ \cline{2-8}  & Flood & - & - & - & 62 & - & - \\ \cline{2-8}  & SlowTe & - & - & - & - & 408 & - \\ \cline{2-8}  & Malformed & 1 & - & 13 & - & 2 & 724 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Confusion matrix when the message type is not used.