## 1 Introduction
The proposed algorithm based on graph representation by considering the relationship between network packets, and then uses deep Auto-encoder for an intrusion detection system. The system is named by Packet2 vec-AE. Compared with the existing preprocessing algorithms for extracting the features of network packets, our proposed packet2 vec learning algorithm combines the original features of the network packet and the local proximity features that characterize the similarity between network packets. Therefore, the accuracy of unsupervised intrusion detection is improved. (2) In this algorithm, we construct a relational graph \(G^{\prime}\) by using each packet as a node, calculate the cosine similarity between packets as edges, and then explore the low-order proximity of each packet via the penalty-based random walk in \(G^{\prime}\). We use the above algorithm as a preprocessing method to enhance the accuracy of unsupervised IDS by retaining the local proximity features of packets maximally. This is the first time in the intrusion detection to extract features using the graph representation leaning algorithm. (3) In this paper, the penalty is added on the basis of the existing graph representation algorithm node2 vec [10].This method can increase the probability of selecting nodes within the local proximity, and the closer the source network packet is, the greater the probability of being selected. Therefore, this method has the ability to extract the local proximity features of the network packet more accurately, and thus more accurately describe the similarity relationship between the network packets. (4) The local proximity features are obtained to accurately characterize the similarity relationship between network packets from the similarity relational graph. The penalty term is used to limit the random walk range of node2 vec [10] to \(k\)-order proximity. In detail, \(k\) is a positive integer that can be customized. This approach has the ability to accurately characterize the neighbor structure of network packets. Experiments have shown that using packet2 vec-AE with penalty has the ability to achieve better performance than packet2 vec-AE without penalty. In the best case, the accuracy, the detection rate, the precision, and the \(F_{1}\) of packet2 vec-AE are up to 94.7%, 90.9%, 94.3%, and 92.6%, respectively. The proposed algorithm achieves the best performances regarding the accuracy, the precision, and the \(F_{1}\) exceeding those of the other state-of-the-art algorithms by 11.6%, 11.9% and 8.7%, respectively. In the worst case, the accuracy, the detection rate, the precision, and the \(F_{1}\) of packet2 vec-AE reached 87.4%, 81.1%, 87.3% and 84.1%, respectively. In the worst case, the proposed algorithm achieves the good performances regarding the accuracy, the precision, and the \(F_{1}\) exceeding those of the other state-of-the-art algorithms by 4.3%, 4.9% and 0.2%, respectively. (5) An empirical formula, i.e., the pruning threshold \(\epsilon\approx\) the mean of weights in \(G^{\prime}-depth^{\prime}\) * penalty value \(\eta\), is designed to calculate the approximately optimal penalty value. The experimental results show that the penalty value calculated by this formula has the ability to obtain better intrusion detection performance. Section 2 describes related work. Section 3 introduces the IDS based on packet preprocess using packet2 vec-Autoencoder. Section 4 introduces the experiment. Section 5 discusses the results. Section 6 concludes the paper.