# A VPN-Encrypted Traffic Identification Method Based on Ensemble Learning
## 2 Methodology
### VPN-Encrypted Traffic FL-XGB Identification Model
#### 2.2.2 FL-XGB-VPN-Encrypted Traffic Identification Model  
The basic Xgboost ensemble learning model suffers from data imbalance when identifying VPN-encrypted traffic data. Various types of training data are not evenly distributed. Among them, classes with more data will take up a larger proportion of the dataset, whereas classes with less data will have a smaller proportion of the dataset. To solve the problem, the Xgboost model is improved with the Focal Loss function based on the cross-entropy loss function. The method can solve the problem of imbalance in the number of difficult and easy samples identified during the training process. Focal Loss is proposed in target detection to solve the problem of imbalance of difficult and easy samples. A parametric balancing factor \(\alpha\) is added in front of the cross-entropy loss function to balance the unequal proportions of difficult and easy samples. The parameter \(\gamma\) adjusts the rate at which the weight of simple samples decreases. When \(\gamma\) is 0, it is the cross-entropy loss function. When \(\gamma\) increases, the influence of the factor also increases. The Focal Loss function is shown in Figure 3.  
The framework of the FL-XGB-VPN-encrypted traffic identification model is shown in Figure 4.  
The basic principle of the FL-XGB-VPN-encrypted traffic identification algorithm is as follows. First, the training set \(D=\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{m},y_{m})\}\) start training the model initializes \(t\) trees, and when \(t=1\), initialize the sample class label predicted value \(y_{i}\). XGBoost is a boosted tree model, where an initial tree predicts a value, then gets the deviation of that value from the actual value, and then adds a tree to learn the deviation. The objective function is a combination of the loss function and penalty function. One part is to calculate the difference between the predicted and true values, and the other part is the regularization term \(\Omega(f_{t})\). The objective function is \(Obj=FL+\Omega(f_{t})\), and the Focal Loss function with weight \(\alpha\) is shown in Equation (9). \[FL=-(\alpha y_{i}logp)+(1-y_{i})log(1-p) \tag{9}\]  
In Equation (9), \(\gamma\) is the focus parameter for focusing on hard-to-identify samples. \(i\) is the number of training samples and \(p\) is the prediction probability of the label. When \(\gamma=2\) and \(p\) is close to 1, the value of \((1-p)^{2}\) will be close to 0, and the more accurate the prediction is. When \(p_{t}\) is close to 0, \((1-p)^{2}\) will be close to 1, and the more inaccurate the prediction is. In summary, easily identifiable samples will have smaller weights and hard-to-identify samples will have larger weights. The first derivative \(g_{i}\) and the second derivative \(h_{i}\) of the Focal Loss function are shown in Equation (10). \[g_{i}=-\big{(}y_{i}+(-1)^{y_{i}}p\big{)}^{\gamma}\big{[}(y_{i}-p)+\gamma(1-y_ {i}-p)+log\big{(}1-y_{i}-(-1)^{y_{i}}p\big{)}\big{]} \tag{10}\]  
Figure 4: The framework of the FL_XGB model. Figure 3: Focal Loss function diagram. Before deriving Equation (10), make:  
\[\left\{ \begin{array}{c}\beta_{1}=\big{(}y_{i}+(-1)^{y_{i}}p\big{)}^{\gamma}\\ \beta_{2}=(y_{i}-p)\\ \beta_{3}=\gamma(1-y_{i}-p)\\ \beta_{4}=log\big{(}1-y_{i}-(-1)^{y_{i}}p\big{)}\end{array} \right. \tag{11}\]  
\[h_{i}=-[\gamma\big{(}y_{i}+(-1)^{y_{i}}p\big{)}^{\gamma-1}p(1-p)(\beta_{2}+ \beta_{3}+\beta_{4})+\beta_{1}(-p(1-p)-\gamma p(1-p)\beta_{4}-\frac{1}{1-y_{i} (-1)^{y_{i}}p}p(1-p))] \tag{12}\]  
According to \(g_{i}\) and \(h_{i}\), the tth tree \(f_{t}(x_{i})\) is trained. The \(t\)th tree \(f_{t}(x_{i})\) together with the previous \(t-1\) trees \(\hat{y}_{i}^{(t-1)}\) form the strong learner \(\hat{y}_{i}^{(t)}=\hat{y}_{i}^{(t-1)}+f_{t}(x_{i})\). \(\hat{y}_{i}^{(t)}\) is used to predict the samples in dataset \(D\) and obtains the dataset class label predictions \(y_{i}\). The training set is fed into the strong learner \(\hat{y}_{i}^{(t)}\), i.e., the FL-XGB model, to obtain the identification results.