## 7 Evaluation
### Effectiveness in Exploit Generation  
To answer RQ2, we compare CanaryExp with REX [3], a well-known AEG system developed by the Shellphish team. We provide the same initial seed corpus that contains input samples that can leak the canary at runtime to the two engines, test if they can generate analysis time exploit and complete the exploitation on the vulnerable programs. Note that we use the canary check stripped version of the benchmark programs as a test suite in experiments on analysis time exploit generation and the original version of the benchmark programs as a test suite in runtime exploitation session.  
Table 6 shows the results. We can see that although REX can generate analysis time exploit successfully, as CanaryExp proved with the patched programs. As shown in the column 'Exploit Generation' depicted in Table 6, it generally fails in exploitability evaluation on the original versions of the benchmarks, while CanaryExp, on the other hand, can finish the tasks, as depicted in column 'PWN ability' in Table 6. We believe the novel runtime canary relocation technique we bring out in this paper should account for this difference. As the runtime canary relocation technique can effectively bridge the difference between the analysis time canary and runtime canary, CanaryExp can bypass the sanity check imposed by StackGuard in a completely automatic way, gaining superiority on this issue over REX.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Program**} & \multicolumn{4}{c}{**Canary Leaking Input Generation Time (s)**} & \multicolumn{4}{c}{**Incremental POC Generation Time (s)**} \\ \cline{2-10}  & **CanaryExp** & **AFL** & **AFL++** & **S2E** & **QSYM** & **CanaryExp** & **AFL** & **AFL++** & **S2E** & **QSYM** \\ \hline stack1 & 141 & - & - & - & - & 414 & 906 & 1034 & 2135 & 923 \\ \hline intovf & 158 & - & - & - & - & 453 & 843 & 936 & 1785 & 793 \\ \hline fmtstr1 & 276 & - & - & - & - & 436 & 903 & 857 & 1890 & 917 \\ \hline fmtstr2 & 303 & - & - & - & - & 321 & 878 & 745 & 1905 & 671 \\ \hline heap-vul1 & 341 & - & - & - & - & 509 & 862 & 913 & 2413 & 874 \\ \hline heap-vul2 & 323 & - & - & - & - & 537 & 1047 & 1132 & 2321 & 908 \\ \hline babyrop2 & 415 & - & - & - & - & 424 & 764 & 613 & 1621 & 598 \\ \hline baby\_canary & 421 & - & - & - & - & 383 & 601 & 596 & 1422 & 523 \\ \hline canary & 517 & - & - & - & - & 395 & 503 & 497 & 874 & 488 \\ \hline microwave & 523 & - & - & - & - & 374 & 465 & 544 & 916 & 421 \\ \hline babypie & 325 & - & - & - & - & 344 & 541 & 566 & 1212 & 497 \\ \hline scv & 443 & - & - & - & - & 284 & 412 & 445 & 842 & 372 \\ \hline Mary\_Morton & 513 & - & - & - & - & 245 & 378 & 332 & 674 & 405 \\ \hline babystack & 414 & - & - & - & - & 304 & 417 & 456 & 786 & 433 \\ \hline proftpd 1.3.1 & 516 & - & - & - & - & 483 & 561 & 577 & 1231 & 584 \\ \hline iwconfig v26 & 464 & - & - & - & - & 421 & 514 & 498 & 1314 & 501 \\ \hline dnsmag 2.77 & 515 & - & - & - & - & 547 & 786 & 775 & 1874 & 694 \\ \hline rsync 2.5.7 & 432 & - & - & - & 442 & 678 & 661 & 1143 & 578 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Time cost of canary leaking input generation and incremental POC generation.