# Squill: Testing DBMS with Correctness Feedback and Accurate Instantiation
## 5 Evaluation
### Comparison with Existing Tools  
We evaluate Squill and Squirrel on three real-world DBMSs, MySQL, MariaDB, and OceanBase, to help us better understand the performance of Squill. As shown in Figure 7, we compare the capability of bug finding and path exploration between the two tools. The number details are listed in Table 2. More program paths explored and more bugs found per unit of time means better fuzzer performance. We also compared the type of bugs they found, which represents how harmful the bug is. Since Squirrel will drop long inputs with multiple subqueries, we disable the length and the subquery check of Squirrel, denoted as Squirrel\({}_{check}\). In statistics, we deduplicate crashes to the corresponding bug since a bug often causes hundreds of crashes and summarize the number of bugs by the hour. Due to the multithreading feature of DBMSs, the unique crash mechanism of AFL is hard to deduplicate DBMS crashes accurately. For MySQL and MariaDB, we deduplicate crashes according to the report output by ASan [19]. For OceanBase, we use GDB [20] to debug each crash after fuzzing and deduplicate according to the information, such as the call stack of functions, at the time of the DBMS crash. **Path Exploration. ** Figure 7a-c show the number of paths explored by [PERSON], Squirrel, and Squirrel\({}_{check}\) over time in MySQL, MariaDB, and OceanBase. As we can see, [PERSON] explored more paths than [PERSON] and Squirrel\({}_{check}\). It is because Squirrel drops long inputs with multiple subqueries for their low semantics-correct rate after instantiation. The input generated by Squirrel is very short and simple and with a fast execution speed. However, \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**DBMS**} & \multicolumn{2}{c}{**Squill**} & \multicolumn{2}{c}{**Squirrel**} & \multicolumn{2}{c}{**Squirrel\({}_{check}\)**} \\ \cline{2-7}  & **Paths** & **Bugs** & **Paths** & **Bugs** & **Paths** & **Bugs** \\ \hline MySQL & 32,827 & 15 & 46,232 & 3 & 26,387 & 6 \\ MariaDB & 33,904 & 17 & 62,465 & 7 & 23,835 & 10 \\ OceanBase & 13,081 & 2 & 16,684 & 0 & 10,630 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The number of paths and bugs explored by each fuzzer in 24 h.  
with semantics-aware instantiation, we do not need to limit the number of subqueries in inputs generated by Squill. Thus Squill can generate long and complex inputs, which means a slow execution speed. Faster execution usually means more paths. So we tested Squirrel\({}_{clcheck}\), which can also generate long and complex inputs, to evaluate Squill more comprehensively. Compared with Squirrel\({}_{clcheck}\), Squill explored 24% more paths in MySQL, 40% in MariaDB, and 23% in OceanBase. Moreover, Squill and Squirrel found many more paths on MySQL and MariaDB than OceanBase. We think this may be caused by the feature of OceanBase as a distributed database and the bad grammar compatibility of the fuzzer with OceanBase. **Bug Finding. ** Figure 7d-f show the number of bugs found by Squill, Squirrel, and Squirrel\({}_{clcheck}\) over time in MySQL, MariaDB, and OceanBase. In total, Squill found 3.4x and 2x more bugs than Squirrel and Squirrel\({}_{check}\), which shows the effectiveness of Squill in bug finding. Note that Squill and Squirrel\({}_{check}\) found more bugs than Squirrel in MySQL and MariaDB. The result proves that there is no fundamental reason that maximizing the number of paths (or seeds) is directly connected to finding bugs [21]. Figure 7g,h show the type of bugs found by Squill, Squirrel, and Squirrel\({}_{check}\). The main types of bugs are assertion fails and SEGV. It shows that Squill found a total of four buffer-related errors, while Squirrel and Squirrel\({}_{check}\) only found one. Figure 7: Comparison with existing tools. Overall, Squill outperforms Squirrel in finding memory error bugs of real-world DBMSs. Because Squill has the ability to generate valid complex SQL while Squirrel cannot.