# A Payload Based Malicious HTTP Traffic Detection Method Using Transfer Semi-Supervised Learning
## 3 Methodology  
### Preliminary  
First, we elaborate the data used for payload based malicious HTTP traffic detection. Malicious information of web attacks is usually contained in the content of the request URL (if a GET request is sent) or the request body (if a POST request is sent). If a HTTP traffic sample is a GET request, its payload is the path of the URL. If a HTTP traffic sample is a POST request, its payload is the path of the URL and the body of the request (i.e., the request parameters). To this end, a HTTP traffic sample \(s\) is defined as a tuple \(s=(x,y)\) where \(x\) denotes the payload extracted from \(s\) (in the form of a string), and \(y\) denotes the label of \(s\) (i.e., normal or the malicious type). Note that \(y\) is unknown if \(s\) is a testing sample. We give an example of HTTP traffic sample in Example 1.  
**Example 1**.: _Given a GET request with the URL as: [[http://www.test.com/scripts/index.php?id=1](http://www.test.com/scripts/index.php?id=1)]([http://www.test.com/scripts/index.php?id=1](http://www.test.com/scripts/index.php?id=1))' union select 1,2,3 from admit#, the obtained HTTP traffic sample is \(s=(x,y)\), where \(x\) is the path of the URL (i.e., /scripts/index.php?id=1' union select 1,2,3 from admit#) and \(y\) is SQL injection (i.e., a type of malicious HTTP traffic)._  
Second, we define the problem in this paper as follows. Given a large public labeled dataset \(D^{\text{PU}}\) and a small dataset from the target system \(D^{\text{TS}}\) (including both labeled and unlabeled samples), the malicious HTTP traffic detection problem can be considered as learning a mapping \(f:\mathcal{X}\rightarrow\mathcal{Y}\) using \(D^{\text{PU}}\) and \(D^{\text{TS}}\). For any unseen sample from the target system \(s^{\text{TS}}=\left(x^{\text{TS}},y^{\text{TS}}\right),f\big{(}x^{\text{TS}} \big{)}\) and \(y^{\text{TS}}\) should be as similar as possible. Here, \(\mathcal{X}\) represents the input space and \(\mathcal{Y}\) represents the set of labels.  
Third, Figure 1 gives the architecture of DeepPTSD, which consists of three major steps, i.e., model initialization, data augmentation, and transfer semi-supervised learning. First, the model initialization step trains the initial model, with DPU as training data and parameters being randomly initialized. Here, we treat malicious HTTP traffic detection as a special text classification problem and utilize TextCNN as the underlying deep neural network (detailed in Section 3.2). Second, the data augmentation step enhances the dataset of the target system by generating a diverse set of samples from \(D^{\text{TS}}\). Here, we propose a HTTP payload data augmentation method based on a keyword avoidance perturbation mechanism for this task (detailed in Section 3.3). Third, the transfer semi-supervised learning step adapts the initial model to the target system by fine-tuning it on the augmented dataset of \(D^{\text{TS}}\). In this paper, we use the same deep neural network (i.e., TextCNN) for both the initial model and the model of the target system (i.e., target model). Here, we design a learning framework by integrating a semi-supervised learning framework (i.e., the UDA framework detailed in Section 3.4) into a transfer learning structure, to leverage the augmented samples.  
In practice, the malicious HTTP payload detection model can be deployed as follows. A user consults a web application and sends a request to the webserver. Before the request arrives at the server, it will be processed by a WAF (Web Application Firewall), which takes as input all the HTTP requests and processes them using the proposed malicious HTTP payload detection model. If the request is malicious then it will be rejected automatically, else it will be sent to the webserver.