# A Payload Based Malicious HTTP Traffic Detection Method Using Transfer Semi-Supervised Learning
## 2 Related work  
### Malicious Network Traffic Detection  
The existing malicious network traffic detection methods could be divided into two categories according to the exploited information, i.e., statistic based methods and payload based methods. Statistic based methods extract flow based statistical features (e.g., duration of flow, inter arrival time, number of packets, etc.) and apply rule based or machine learning based techniques to detect malicious network traffic [1; 2; 3; 4; 5]. Although statistic based methods could adapt to data privacy and encryption, they cannot detect malicious network traffic that does not exhibit significant differences in flow patterns (e.g., SQL injection). In addition, they require a relative long period of monitoring time to extract statistical features, so they are usually designed for network traffic based on protocols such as TCP and UDP, and not suitable for malicious network traffic detection from single HTTP requests. Payload based methods detect malicious network traffic by exploiting the content of the packets (e.g., URL, file path, request body, etc.). Most state-of-the-art payload based methods adopt traditional machine learning techniques and deep learning techniques for malicious network traffic detection. Given a set of network traffic samples as training data, learning based methods train a detection model, which could classify a new network traffic sample as malicious or benign. The key factor for the learning based methods is to extract good features to represent the network traffic samples. Traditional machine learning techniques explore features from various aspects, including lexical features (e.g., N-gram features, bag-of-word features, etc.) [8; 9; 18; 19] and statistical semantic features (e.g., number of characters, number of parameters, count of sensitive keywords, etc.) [10; 11; 20]. However, these features should be manually designed based on domain knowledge and require constant adjustment to accommodate changes in malicious network traffic patterns.  
In recent years, thanks to the superior capability of automatically extracting representative features from raw data (especially the unstructured data), deep learning techniques have been exploited for malicious network traffic detection [16]. For example, [PERSON] et al. [21] proposed eXpose, which is a malicious payload detection model based on character-level embedding and CNN (convolutional neural network). [PERSON] et al. [12] proposed a malicious HTTP message detection model by using an autoencoder with character-level binary image transformation. [PERSON] et al. [13] proposed a joint approach of CNN, LSTM, and attention mechanism to detect malicious HTTP traffic by exploiting character-level lexical features and host features. However, character-level features cannot capture the latent semantics of payloads, resulting in low generalization ability. Aiming at this problem, more recent researches design detection model based on word-level features. For example, [PERSON] et al. [22] modeled a HTTP traffic sample as a natural language sequence under predefined vocabulary using BiLSTM (bidirectional long short-term memory) with attention mechanism. [PERSON] et al. [14] proposed URLNet, an end-to-end deep learning framework to learn a non-linear URL embedding by applying CNN to both characters and words. [PERSON] et al. [15] designed a convolutional GRU (gated recurrent unit) for malicious URL detection based on a predefined keyword library. However, deep learning techniques require a large number of labeled samples to train the models, while it is almost impossible to collect adequate labeled samples for each individual system, especially the malicious samples.