## 4 Evaluation
### The Effectiveness of Penalty in Packet2 vec Preprocessing by Autoencoder  
This section discusses the impact of penalty values \(\eta\) on the performance of IDS in Packet2 vec-AE. Different penalty values \(\eta\) may affect the random walk range of the source network packet, and ultimately affect the accuracy of the local proximity features of the source network packet. Therefore, it is necessary to evaluate the impact of different penalty values \(\eta\) in Packet2 vec-AE on IDS performance based on experiments. The IDS performance of the Packet2 vec-AE penalty values of \(-0.01\), \(-0.03\), \(-0.05\), and \(-0.1\) was evaluated. Table 6 is a statistical value of the similarity relationship between network packets in \(G^{\prime}\). The statistical value shows that the average of the weights in \(G^{\prime}\) is 0.93. Table 7 shows the IDS performance with different penalty values in Packet2 vec-AE. It can be concluded from Table 7 that the packet2 vec-AE based IDS achieves optimal performance when the penalty value is \(-0.03\). When the penalty values are \(-0.01\) and \(-0.1\), the performance of the IDS is relatively poor. Figure 6 shows the Packet2 vec-AE visualization results for different penalty values. It can be concluded from Figure 6 that the IDS can accurately distinguish DDoS attacks, and it is difficult to accurately distinguish the other four types of network packets, when the penalty values are \(-0.01\) and \(-0.1\); When the penalty value is \(-0.05\), the IDS has the ability to distinguish the following types \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Actual Class: True** & **Actual Class: False** \\ \hline Predicted Class: True & TP & FP \\ \hline Predicted Class: False & FN & TN \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Cofusion Matrix. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Preprocessing Algorithm**} \\ \cline{2-9}
**Length \(I\)** & \multicolumn{3}{c}{**(a) Preprocessed with Node2 vec**} & \multicolumn{3}{c}{**(b) Preprocessed with Packet2 vec**} \\ \cline{2-9}  & **ACC** & **DR** & **Precision** & \(\mathbf{F_{1}}\) & **ACC** & **DR** & **Precision** & \(\mathbf{F_{1}}\) \\ \hline
10 & 73.8 & 73.7 & 75.2 & 74.4 & 89.8 & 87.9 & 88.0 & 87.9 \\
80 & **87.4** & **81.1** & **87.3** & **84.1** & **94.7** & **90.9** & **94.3** & **92.6** \\
1000 & 65.8 & 63.5 & 67.9 & 65.6 & 90.1 & 87.3 & 88.7 & 88.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of IDS performance of Node2 vec and Packet2 vec preprocessed under different random walk lengths (%). of network packets more accurately, including normal, Brute Force SSH attacks and DDoS attacks; When the penalty value is \(-0.03\), the IDS has the ability to distinguish five class of network packets more accurately. One possible explanation is that the penalty value \(\eta\) determines the range of local proximity samples for network packets. Experiment shows that the penalty value calculated by the empirical formula, i.e., (the pruning threshold \(\varepsilon\approx\) the mean of weights in \(G^{\prime}-depth^{\prime}\) * penalty value \(\eta\)), can achieve the optimal effect. In this experiment, the value of the pruning threshold \(\varepsilon\) is \(0.86\). The random walk range of the source network packet is within the low-order proximity, that is, the maximum value of \(depth^{\prime}\) is \(2\). It uses an empirical formula, i.e., (the pruning threshold \(\varepsilon\approx\) the mean of weights in \(G^{\prime}-depth^{\prime}\) * penalty value \(\eta\)), for approximate calculation. When the penalty value is \(-0.03\), the Packet2 vec-AE algorithm achieves optimal performance. At the same time, the experimental results in Table 7 and Figure 6 show that the Packet2 vec-AE algorithm achieves optimal performance when the penalty value is \(-0.03\). In addition, when the penalty value is \(-0.1\), the penalty for the weight is too large, which leads to too much constraint on the local proximity sampling range.