# FirmHunter: State-Aware and Introspection-Driven Grey-Box Fuzzing towards IoT Firmware
## 5 Evaluation
### Evaluation of FirmHunter
#### 5.2.2 Comparison with FirmFuzz and ZAP(Q2)  
In order to evaluate the efficiency of FirmHunter in detecting real-world vulnerabilities, we compare it with a state-of-the-art IoT fuzzer Firmfuzz and a popular web application scanner ZAP.  
To configure the other two tools, we manually modified the crawler in FirmFuzz so that it can automatically traverse the web application. Furthermore, for ZAP, we set username and password in credentials for authentication, used the automated scan with both traditional spider and ajax spider. Likewise, FirmFuzz applied the crawler traversing the WEB interface to obtain initial messages and ZAP collected websites in automated scan mode within 1 h.  
The discovered vulnerabilities and used time are listed in Table 5.  
For FirmFuzz, it only detected five vulnerabilities due to the neglect of dependencies between messages. Furthermore, for ZAP, the front-end web application scanner, it could just find 2 XSS vulnerabilities.  
The results show that FirmHunter not only finds more vulnerabilities than other tools, but also takes less time to find them in almost all cases.  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Exploited ID** & **Vendor** & **Device** & **Type** & \begin{tabular}{l} **FirmHunter** \\ **-NULL** \\ \end{tabular} & \begin{tabular}{l} **FirmHunter** \\ **-MSQ** \\ \end{tabular} & \begin{tabular}{l} **FirmHunter** \\ **-MSQ** \\ \end{tabular} & \
\begin{tabular}{l} **FirmHunter** \\ **-MSQ** \\ \end{tabular} &
\begin{tabular}{l} **Improvement** \\ **-MSQ** \\ \end{tabular} \\ \hline CVE-2016-1555 & Netgear & WNAP320 & BO & 8 min 10 s & 21 min 11 s & 9 min 42 s & 54\% \\ CVE-2016-1558 & DLink & DAP2695 & BO & 51 min 43 s & 6 min 39 s & 6 min 43 s & \(-\)1\% \\ CVE-2017-13772 & TPLink & WR940N & BO & NA & 16 min 2 s & 11 min 59 s & 25\% \\ CVE-2018-19240 & Trendnet & IP110 wn & BO & 1 h 15 min & 2 h 39 min & 1 h 21 min & 49\% \\ CVE-2019-11400 & Trendnet & tew-652 BRP & BO & NA & 1 h 56 min & 1 h 3 min & 46\% \\ CVE-2019-7298 & DLink & DSL-3782 & BO & NA & 18 min 10 s & 12 min 25 s & 32\% \\ CVE-2021-40284 & DLink & DSL-3782 & BO & NA & 35 min 3 s & 15 min 27 s & 56\% \\ CVE-2018-17990 & DLink & DSL-3782 & CI & NA & 10 h 22 min & 6 h 19 min & 39\% \\ CVE-2019-11399 & Trendnet & tew-652 BRP & CI & NA & 3 h 34 min & 2 h 12 min & 38\% \\ Unknown2 & Netgear & WNDR3700 & CI & NA & 9 h 24 min & 2 h 22 min & 75\% \\ Unknown3 & Netgear & WNDR3700 & CI & NA & 14 h 51 min & 5 h 27 min & 63\% \\ CVE-2018-6529 & DLink & DIR-865L & XSS & NA & 18 min 31 s & 13 min 48 s & 25\% \\ CVE-2018-17989 & DLink & DSL-3782 & XSS & NA & 10 h 20 min & 6 h 18 min & 39\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistic on vulnerability detection against FirmHunter-NULL and FirmHunter-MSQ.  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Exploited ID** & **Vendor** & **Device** & **Type** & **FirmHunter** & **FirmFuzz** & **ZAP** \\ \hline CVE-2016-1555 & Netgear & WNAP320 & BO & 9 min 42 s & 12 min 55 s & NA \\ CVE-2016-1558 & DLink & DAP2695 & BO & 6 min 43 s & NA & NA \\ CVE-2017-13772 & TPLink & WR940N & BO & 11 min 59 s & NA & NA \\ CVE-2018-19240 & Trendnet & IP110 wn & BO & 1 h 21 min & 2 h 12 min & NA \\ CVE-2019-11400 & Trendnet & tew-652 BRP & BO & 1 h 3 min & 1 h 32 min & NA \\ CVE-2019-7298 & DLink & DSL-3782 & BO & 12 min 25 s & NA & NA \\ CVE-2021-40284 & DLink & DSL-3782 & BO & 15 min 27 s & NA & NA \\ CVE-2018-17990 & DLink & DSL-3782 & CI & 6 h 19 min & NA & NA \\ CVE-2019-11399 & Trendnet & tew-652 BRP & CI & 2 h 12 min & 3 h 15 min & NA \\ Unknown2 & Netgear & WNDR3700 & CI & 2 h 22 min & NA & NA \\ CVE-2018-6529 & DLink & DIR-865L & XSS & 13 min 48 s & 36 min 5 s & 45 min 56 s \\ CVE-2018-17989 & DLink & DSL-3782 & XSS & 6 h 18 min & NA & 5 h 56 min \\ \hline \hline \end{tabular}
\end{table}
Table 5: Statistic on vulnerability detection against FirmFuzz and ZAP.