# TMVDPatch: A Trusted Multi-View Decision System for Security Patch Identification
## 5 Experiments and Results
### Hyperparameters Tuning  
This subsection mainly discusses the problem of hyperparameter setting in the model. In order to achieve better tuning parameters, we use Optuna for hyperparameter search. Optuna [45] is a hyperparameter optimization tool that optimizes tree-based hyperparameter search and relies on Bayesian probability to iteratively adjust the search for hyperparameters. The specific details are shown in Table 2. This subsection mainly adjusts the 8 hyperparameters listed in the Table 2. Except the learning rate is sampled by logarithmic uniform distribution, other hyperparameters are sampled by step. At the same time, we maps the 200 trials to the state points in the three-dimensional space in the form of \"X-Recall, Y-Precision, Z-Accuracy\", and the red series of points are the trials that are judged be the best trials. The color of the points only represents the trials ID. The specific results are shown in Figure 4, where values represent the accuracy, precision and recall values respectively, and params is the parameter setting of the current best trial. In addition, Figure 5 draws a parallel coordinate graph based on the \(F1\) evaluation metric to highlight the relationship between hyperparameters in the search space. Among them, the vector dimension may affect the security patch identification performance of the model. If the dimensions are too large, the model may be so complex that become overfit. However, too small dimension will also lead to poor model learning ability. Therefore, how to select the optimal parameter combination in the hyperparameter search space is critical to the model performance. As shown in the Figure 5, for the code diff view, both the LSTM hidden layer and the embedding layer require a higher vector dimension to achieve better model performance. But for the message commit view, it needs to achieve better patch identification results by keeping the embedding layer dimension high and the LSTM hidden layer dimension low. The main reason is that the model is more complicated in the message commit view. Because of this, the dropout layer parameter in the LSTM self-attention mechanism is set to 0.3, thereby improving the generalization ability of the model and preventing the model from overfitting. In the same way, in the code diff view, the global dropout layer \begin{table}
\begin{tabular}{c c c c} \hline \hline
**View** & **Hyper-Parameter** & **Search Space** & **Best Value** \\ \hline \multirow{4}{*}{Code Diff} & codediff\_embedding & [40,120] & 100 \\  & codediff\_global\_dropout & [0.1,0.5] & 0.2 \\  & codediff\_lstm & [32,128] & 128 \\  & codediff\_lstm\_dropout & [0.1,0.5] & 0.4 \\ \hline \multirow{4}{*}{Commit Message} & commit\_dropout & [0.1,0.5] & 0.3 \\  & commit\_embedding & [40,120] & 120 \\  & commit\_lstm & [32,128] & 64 \\ \hline \hline Overall & lr & [0.00001, 0.1] & 0.000557 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The hyperparameters tuning details. Figure 4: The hyperparameters tuning trials in the three-dimensional space. parameters are set lower than the dropout layer parameters in the LSTM self-attention mechanism. Moreover, the F1 scores of the 200 trials in Figure 5 are mostly concentrated between 0.8 and 0.9, which further indicates that the model has strong robustness to hyperparameter settings. In order to more intuitively explain the influence of parameters on each evaluation metric, we evaluate the importance of hyperparameters based on Optuna. The specific results are shown in Figure 6, where the sum of the importance of each parameter is 1 under each metric. It is not difficult to see that for all evaluation metrics, the learning rate setting is extremely critical. At the same time, the hyperparameters in the code diff view are slightly more important to accuracy and F1 than the message commit view. Moreover, the F1 score is used as a comprehensive measure of the precision and recall of the model, and the importance of each parameter is distributed evenly.