# Squill: Testing DBMS with Correctness Feedback and Accurate Instantiation
## 5 Evaluation
### Contribution of Semantics-Aware Instantiation  
In this section, we evaluate semantics-aware instantiation introduced in Section 3.2. We perform instantiation method of Squill and Squirrel to instantiate SQL statements of the same dataset and input the SQL statements with semantics to DBMS. We evaluate the instantiation of Squill by comparing the correctness of these inputs. The higher valid rate of input after instantiation is, the better instantiation method is. In the end, we illustrate the advantages of Squill instantiation through a practical example. The dataset contains all valid seeds in one MySQL fuzzing of Squill because we want to compare the two methods' capability to instantiate some critical inputs and ensure that these inputs can be correctly instantiated. We normalize the seeds before adding them to the dataset, that is, removing the semantics in them. Due to the design of translating AST to string, the input generated by Squill has a very tiny probability that it cannot be parsed by itself (same with Squirrel). Moreover, there are differences between the grammar of Squill and Squirrel, and Squirrel cannot parse some inputs of Squill. So we remove the seeds that both Squill and Squirrel cannot parse. The evaluation results are shown in Table 3.  
The results show that the instantiation of Squill (92.5% valid rate) outperforms Squirrel's (58.24% valid rate). In addition, we make separate statistics according to the file size of the seeds. The file size of seeds corresponds to the length of the SQL statement, which we think is positively related to the complexity of the SQL statement. Long SQL statement \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Fuzzer** & **Seed Size** & **Valid** & **Invalid** & **Total** & **Valid Rate** \\ \hline \multirow{4}{*}{**Squill**} & **\textless{1 kb}** & 5246 & 373 & 5619 & 93.36\% \\  & **1\textless{1.5 kb}** & 15,810 & 1328 & 17,138 & 92.25\% \\  & **\textgreater{1.5 kb}** & 8280 & **676** & 8956 & 92.45\% \\  & **total** & 29,336 & 2377 & 31,713 & 92.5\% \\ \hline \multirow{4}{*}{**Squirrel**} & **\textless{1 kb}** & 4304 & 1315 & 5619 & 76.6\% \\  & **1\textless{1.5 kb}** & 9747 & 7391 & 17,138 & 56.87\% \\ \cline{1-1}  & **\textgreater{1.5 kb}** & **4419** & **4537** & 8956 & **49.34\%** \\ \cline{1-1}  & **total** & 18,470 & 13,243 & 31,713 & 58.24\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The comparison between instantiation of Squill and Squirrel. usually means more complicated dependencies between nodes and more nested structures, such as subquery. With the increased complexity of SQL statements (file size), the valid rate of Squirrel's instantiation is significantly reduced, while the correct rate of Squill's instantiation changes less. This shows the advantage of Squill's instantiation in processing complex SQL statements. Because of the randomness in semantics filling, the valid rate of Squill's instantiation is not 100%, though the input was instantiated correctly before. Listing 3 is a PoC of ID 23 in Table 1. It can be seen that there is a nested structure containing subqueries in the SELECT statement in Line 3. This kind of nested structure is pervasive in SQL statements generated by mutation, which may be closely related to overflow vulnerabilities. It is difficult for Squirrel to instantiate such type of structure since Squirrel is hard to build correct dependencies between subqueries, such as the semantics of the first two v4 positions in Line 3. **Listing 3**. A PoC of ID 23 in Table 1.  
**CREATE TABLE v0 ( v1 NUMBER UNIQUE, v2 BIGINT ) ; CREATE TABLE v3 ( v4 INT, v5 INT ) ; SELECT 1 FROM v3 GROUP BY ( SELECT v4 FROM ( SELECT v4 FROM ( SELECT v4 FROM v3 UNION SELECT v1 FROM v0 ) AS v7 WHERE (v4 = 0 AND v4 = -1 AND v4 = 67 ) ) AS v9);**  
**6. Discussion**  
In this section, we discuss several limitations of our current implementation and possible future directions. **Universality of Fuzzer. ** In this paper, the instantiation of Squill is based on the grammar of MySQL, which has low universality.