# A Payload Based Malicious HTTP Traffic Detection Method Using Transfer Semi-Supervised Learning
## 1 Introduction  
Web applications are the most popular applications on Internet, as they require a smaller amount of client side resources, and are relaxed to deliver and maintain. Therefore, a large number of attackers put web applications as their primary target. Among all the network protocols, HTTP (Hypertext Transfer Protocol) is used by most of the web applications for communication between web browsers and web servers, and a large majority of attacks against web applications are launched by using malicious HTTP requests, e.g., SQL injection, XSS (Cross-Site Scripting). Therefore, malicious HTTP traffic detection systems play an essential role in establishing a secure and reliable web application. In practice, the malicious HTTP traffic detection models are usually deployed on a WAF (Web Application Firewall) in front of the web servers. The existing malicious network traffic detection methods could be divided into two categories according to the exploited information, i.e., statistic based methods and payload based methods. Statistic based methods [1; 2; 3; 4; 5] extract statistical features (e.g., duration of flow, inter arrival time, number of packets, etc.) from network traffic analyzer (e.g., NetFlow [6]), and then apply rule based or learning based techniques to detect malicious network traffic. These statistical features are payload-independent, and thus the statistic based methods can address specific issues such as data privacy and traffic encryption [7]. However, statistic based methods have the following limitations. First, useful information such as suspicious keywords in payloads are ignored, so it is difficult to detect malicious network traffic that does not exhibit significant differences in flow patterns (e.g., SQL injection). Second, most statistical features require a relative long period of monitoring timebefore they can be appropriately extracted, and thus they are usually used for representing network traffic based on protocols such as TCP and UDP, and not suitable for malicious network traffic detection from single HTTP requests. On the other hand, payload based methods [8; 9; 10; 11; 12; 13; 14; 15] detects malicious network traffic by analyzing the content of the packets (e.g., URL, file path, request body, etc. ), and thus they can be used for single HTTP request. Therefore, these two types of detection methods can be used to complement each other. In this paper, we focus on the payload based malicious HTTP traffic detection method. Most state-of-the-art payload based methods adopt learning techniques for malicious network traffic detection, including traditional machine learning techniques [8; 9; 10; 11] and deep learning techniques [12; 13; 14; 15; 16]. They train detection models by exploiting a corpus of correctly labeled samples (including both normal and various types of malicious network traffic samples), and then utilize the detection models for real-time malicious network traffic detection. Although learning based methods could achieve high detection accuracy, they still suffer from the following problem: the learning techniques (especially the deep learning techniques) require a large number of labeled samples to train the models. However, it is very difficult to collect adequate labeled samples in practice, especially the malicious network traffic samples. Although this problem could be alleviated by training the detection models on a public dataset with a relatively large number of labeled samples and then applying the trained models to the target systems, there is a significant pattern shift between the samples collected from different systems (called the train-test gap problem), due to the unique characteristic of each individual system [17]. Applying the models trained on the public dataset to a new system usually results in performance degradation. To address the above problems, we propose DeepPTSD, a payload based malicious HTTP traffic detection method based on deep transfer semi-supervised learning technique.