# Squill: Testing DBMS with Correctness Feedback and Accurate Instantiation
## 5 Evaluation
### Contribution of Correctness-Guided Mutation  
To understand the contribution of different factors in correctness-guided mutation, we disable each factor to perform unit tests in MySQL and measure various aspects of the fuzzing process. In addition to the capabilities of bug finding and path exploration, we also compare the correctness of the input. Figure 8 shows the result, where Squill\({}_{\textit{lib}}^{\textit{ised}}\) means we disable both correctness-focused seed selection and mutation with interesting material library, and Squill\({}_{\textit{lib}}\) means we only disabled interesting material library. Since the implementation of the mutation with interesting material library relies on correctness-focused seed selection, we do not disable the latter and keep the former.  
**Correctness of Inputs.** Figure 8a shows the valid rate of inputs when fuzzing, which means the proportion of valid inputs in all inputs. Higher valid rate of inputs when fuzzing is better, because we want the input to pass the validity check of the DBMS. The result is Squill \(\approx\) Squill\({}_{\textit{lib}}>\) Squill\({}_{\textit{lib}}^{\textit{ised}}\), where Squill\({}_{\textit{lib}}\) is 6% higher than Squill\({}_{\textit{lib}}^{\textit{ised}}\). The result shows  
Figure 8: Contributions of correctness-guided mutation. The experiment is performed on MySQL.  
that the correctness-focused seed selection improves the ability of the fuzzer to generate more valid inputs because of its strategy to prioritize mutating seeds which is valid.  
**Path Exploration.** Figure 8b shows the number of paths explored by each fuzzer. Squill, Squill\({}_{lib}\), and Squill\({}_{lib}^{send}\) are almost equal in the number of paths, and Squill\({}_{lib}^{send}\) is slightly higher than the other two. Due to that Squirrel\({}_{lib}^{send}\) generates more syntactically and semantically incorrect inputs, as shown in Figure 8a, its inputs are executed faster, leading to more paths. In addition, we count the number of valid seeds generated during fuzzing, as shown in Figure 8c. A seed represents a path since only if an input triggers a new path will it be saved into the seed queue as a seed. Therefore, the number of valid seeds reflects the capability of exploring the path which passes the syntactic and semantic check of DBMS. The result is Squill > Squill\({}_{lib}\) > Squill\({}_{lib}^{send}\), where Squill is approximately 9% higher than Squill\({}_{lib}^{send}\), and Squill\({}_{lib}\) is about 3% higher than Squill\({}_{lib}^{send}\). The result shows that both two mechanisms can help fuzzing in path exploration.  
**Bug Finding.** Figure 8d shows the number of bugs found by Squill with each setting, where the original Squill achieves the best results. Squill and Squill\({}_{lib}\) found 15 and 14 bugs in MySQL, while Squill\({}_{lib}^{send}\) only found 10. The results show that the correctness-guided mutation plays an important role in bug finding.  
Overall, both mechanisms of correctness-guided mutation improve the effectiveness of Squill in path exploration and bug finding, where correctness-focused seed selection improves the ability of Squill to generate more valid inputs and mutation with interesting material library helps Squill explore more DBMS states after the syntactic and semantic check.