## 5 Evaluation
### Coverage Improvement  
We run each application with QSYM and SHFuzz respectively under the same experimental setup for 12 h. Except for the input and branch scheduling part, the other experimental setups are the same. We set the time-out for each concolic execution process for one input as 90 s, the same as the default configuration of QSYM. Therefore, our evaluation can show fairly how our selective approach will improve the performance of hybrid fuzzing. The overall results of coverage and the number of crashes are shown in Table 2. \begin{table}
\begin{tabular}{l l r r l} \hline \hline
**Name** & **Version** & **Src** & **Size (KB)** & **Type** \\ \hline boringssl & 2016-01-12 & & 2244 & server program \\ c-ares & CVE-2016-5180 & & 14 & asynchronous DNS resolves \\ guetzli & 2017-3-30 & & 3505 & image processing \\ harfuzz & 1.3.2 & & 3357 & text type processing \\ json & 2017-02-12 & & 254 & json file processing \\ libarchive & 2017-01-04 & & 1435 & compression library \\ libjpeg-turbo & 07-2017 & & 663 & image processing \\ libpng & 1.2.56 & & 429 & image processing \\ libxml2 & v2.9.2 & & 4771 & XML C parser \\ llvm-libcxxabi & 2017-01-27 & & 513 & C++ compile tool \\ openssl & 1.0.1f & & 2014 & server program \\ openssl & 1.0.2d & & 291 & server program \\ pcre2 & 10.00 & & 808 & regular expression \\ proj4 & 2017-08-14 & & 3395 & cartographic projections library \\ re2 & 2014-12-09 & & 4551 & regular expression \\ sqlite & 2016-11-14 & & 895 & database \\ vorbis & 2017-12-11 & & 320 & audio file processing \\ woff2 & 2016-05-06 & & 1725 & font format \\ \hline nm & 2.34 & & 4248 & executable file information \\ readelf & 2.34 & & 1946 & executable file information \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Types and other information of the 20 evaluated real-world programs. \begin{table}
\begin{tabular}{l l r r r} \hline \hline  & **Method** & **Coverage** & **Crash** & **Bug Found Time** \\ \hline c-ares- CVE-2016-5180 & & QSYM & 42 & 7 & 0:29:12 \\  & SHFuzz & 42 & 11 & 0:01:27 \\ guetzli-2017-3-30 & & QSYM & 3741 & 1 & 7:15:46 \\  & SHFuzz & 4300 & 18 & 3:16:16 \\ \hline json-2017-02-12 & & QSYM & 3186 & 17 & 0:05:30 \\  & SHFuzz & 3149 & 24 & 0:01:42 \\ \hline llvm-libcxxabi-2017-01-27 & & QSYM & 10,559 & 182 & 0:15:40 \\  & SHFuzz & 11,458 & 243 & 0:11:29 \\ \hline openssl-1.0.2d & & QSYM & 3276 & 35 & 0:54:50 \\  & SHFuzz & 3226 & 84 & 0:28:10 \\ \hline pcre2-10.00 & & QSYM & 23,121 & 23 & 0:47:59 \\  & SHFuzz & 25,633 & 570 & 0:18:01 \\ \hline nm-2.34 & & QSYM & 6969 & 22 & 8:39:31 \\  & SHFuzz & 6845 & 39 & 0:22:11 \\ \hline libxml2-v2.9.2 & & QSYM & 5513 & 0 & - \\  & SHFuzz & 5833 & 6 & 0:18:31 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Coverage, crash results, and bug found time of running QSYM and SHFuzz for 12 h on 20 applications. From Table 3, we can see the increasing number and percentage of the results by SHFuzz compared with the QSYM original configuration. As for the coverage part, we can see clearly that 16 out of 20 applications show an obvious increase in the number of coverage. Though the other four show negative results on coverage, they still can find more bugs compared with QSYM. From the positive results, we can see that sqlite-2016-11-14, pcre2-10.00, llvm-libcxxabi-2017-01-27, and guetzli-2017-3-30 show the best improvement by SHFuzz, each increasing the coverage by 3522 (48.16%), 2512 (10.86%), 899 (8.51%), and 559 (14.94%). We also calculate the average improvement on coverage, which is 497 (8.40%) more lines of coverage. These data also include negative results, so we believe they show that for most applications, our SHF approach can help greatly with increasing coverage compared with the original configuration in hybrid fuzzing. We show the fuzzing process figures of four applications with the best improvement on coverage in Figure 5, which shows why SHFuzz works. In each sub-figure, the X-axis shows the time range, while the Y-axis shows the coverage hybrid fuzzing has reached at each hour.