# A Payload Based Malicious HTTP Traffic Detection Method Using Transfer Semi-Supervised Learning
## 3 Methodology
### Data Augmentation  
Since the dataset of the target system DTS is usually limited in size and of low degree of diversity, retraining or fine-tuning the initial detection model on such dataset would easily lead to overfitting. Data augmentation is a process of generating more data with higher degree of diversity by using the available limited data, so as to address the overfitting issue [35]. Since payloads are in the form of textual data, it is intuitive to apply textual data augmentation methods for the payload data augmentation task. The existing textual data augmentation methods include back translation [36; 37], word replacement [38; 39], and sentence generation [35; 40] Back translation is a procedure of translating into another language and then back to the original language. It can generate diverse paraphrases while preserving the semantics of the original sentence. However, payloads are not human language, and thus cannot be translated. Word replacement generates new sentences by replacing selected words in the original sentence with new words, and the most intuitive strategy is replacing with synonym. However, a large portion of tokens in payloads have particular meaning. For example, if we replace \"select\" in the payload of Example 1 with \"choose\", the new payload /scripts/index.php?id=1'union choose 1,2,3 from admin# would lose the ability of web attack, because \"select\" is a particular keyword for querying the database. Sentence generation uses generative model such as VAE (variational autoencoder) and GAN (generative adversarial network) to generate sentences from a latent space. However, it is difficult to learn a conditional token distribution from the payloads, since they do not have strong structure and grammar patterns as compared with the natural human language. Based on the above analysis, the existing textual data augmentation methods cannot be directly used for payload data augmentation task. Aiming at this problem, we propose a HTTP payload data augmentation method based on a keyword avoidance perturbation mechanism. The idea is to add noises to the original payloads while keeping the particular keywords unchanged. The proposed HTTP payload data augmentation method consists of two steps, i.e., keyword mining and payload perturbation. Figure 2: The architecture of TextCNN. **Keyword mining**: A simple approach is to manually build a keyword library for HTTP payloads [12; 14]. However, this approach has the following limitations. First, manual building is tedious and costly, and it usually has a very low recall rate. Second, attackers could use obfuscation operation to change the keywords in a payload into meaningless substrings, in order to bypass the malicious keyword detection system. As shown in Example 3, it is extremely difficult to manually collect these obfuscated keywords. **Example 3**. : _The keywords \"select\" and \"union\" could be changed into \"#!50000%53 elect*/\" and \"uni%0 bon\", respectively. Then, the payload in Example 1 can be transformed into: [[http://www.test.com/scripts/index.php?id=1](http://www.test.com/scripts/index.php?id=1)]([http://www.test.com/scripts/index.php?id=1](http://www.test.com/scripts/index.php?id=1))'uni%0 bon /#!50000%53 elect*/1,2,3 from admin#._  
Therefore, we use a TF-IDF based method for keyword mining. First, given a malicious HTTP payload dataset \(MD\) and a benign HTTP payload dataset \(BD\), where each payload has been segmented based on the method in Section 3.2, we calculate the TF-IDF score \(tis(w)\) for each token \(w\) based on Equation (1), where \(n_{MD}(w)\) is the number of \(w\) in \(MD,n_{MD}\) is the number of all tokens in \(MD\), and \(d_{j}\) is the \(j\) th payload in \(BD\). According to Equation (1), a token that frequently appears in malicious payloads but seldom or never appears in benign payloads would be given a high TF-IDF score. Second, for each token \(w\), if its TF-IDF score \(tis(w)\) is larger than a predefined threshold \(\sigma\), it will be defined as a keyword.