# SATFuzz: A Stateful Network Protocol Fuzzing Framework from a Novel Perspective
## 4 Experimental Analysis
### Test Case Filtration  
In Section 3.4, we generalized the input and output of the test case filter, and the parameter names and their corresponding values are shown in Table 4. Each byte component of P has only two values, 0 and 1; thus, the input size was set to 2. The classification process has only two results, valid and invalid; thus, the output size was set to 2. The default values of the number of hidden layers, bidirectional flag, dropout rate, and learning rate were 2, True, 0.4, and 0.001, respectively. Moreover, the cross-entropy loss function was used to improve the classification performance of the test case filter.  
By fixing the other parameters, we compared the influences of different hidden layer sizes on the classification performance of our approach. The experimental results are shown in Table 5. As classification performance requires specific evaluation indicators and a single precision rate or recall rate cannot meet the actual evaluation needs, we took the weight-equivalent harmonic mean of the two, the F1 value, as the measurement standard, and its definition is given as follows:  
\[\text{F1}=2\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{ Recall}}=\frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}} \tag{1}\]  
In the formula: Precision denotes the precision rate; Recall denotes the recall rate; P denotes invalid test cases; N denotes valid test cases; TP denotes true-positive cases; FP denotes false-positive cases; and FN denotes false-negative cases.  
The experimental results show that the corresponding relationship between the classification performance and hidden layer size varies slightly for different test objects. For LightFTP, when the hidden layer size is 4, the F1 score reaches its highest value, and the test case filter has the best classification performance. In contrast, for Live555, the F1 score reaches its highest value, and the test case filter has the best classification performance when the hidden layer size is 2. Figure 6 visually shows the difference between the experimental results of the two test case filters.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter Name** & **Parameter Value** \\ \hline Input Size & 2 \\ Output Size & 2 \\ Hidden Layer Size & 4(LightFTP) \\
2(Live555) \\ Hidden Layer Number & 2 \\ Bidirectional Flag & True \\ Epoch & 500 \\ Dropout & 0.4 \\ Learning Rate & 0.001 \\ Optimization Algorithm & Adam \\ Loss Function & Cross-Entropy \\ \hline \hline \end{tabular}
\end{table}
Table 4: The parameter settings of the test case filter.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Hidden** & **Test** & & & & & & \\
**Layer Size** & **Object** & & & & & \\ \hline \multirow{2}{*}{2} & LightFTP & 73 & 1604 & 7026 & 319 & 87.96\% \\  & Live555 & 37 & 833 & 6710 & 707 & 89.71\% \\ \multirow{2}{*}{4} & LightFTP & 49 & 921 & 6861 & 403 & 91.20\% \\  & Live555 & 62 & 1207 & 6095 & 532 & 87.52\% \\ \multirow{2}{*}{8} & LightFTP & 67 & 1511 & 7316 & 358 & 88.67\% \\  & Live555 & 92 & 1839 & 7067 & 370 & 86.74\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Classification performance of the test case filter under different hidden layer sizes.