## 7 Evaluation
### Effectiveness in Canary Leaking Input and Incremental POC Input Generation  
To answer RQ1, we compare CanaryExp against well-known engines including AFL, AFL++, S2E and QSYM. To guarantee computation fairness, We allocate the same quota on resources for computation, which is composed of two logical processors, 64 GB physical memory and a 2 h time quota. Note that we set the logical processor count to 2 because CanaryExp takes exactly two logical processors during calculation-one for canary leaking input generation and the other canary leakage persistent POC input generation. We list the specific configurations for each engine in Table 4.  
We conduct two experiments as follows. To compare CanaryExp with other test case generation engines on canary leaking input generation, we launch them on the same initial input corpus containing POC samples that can lead the execution trigger some vulnerability, with an introspection process launched at the same time checking whether the generated test cases can leak the desired canary, with the time cost recorded at the same time. To compare CanaryExp with other test case generation engines on incremental POC input generation, we launch them on the same initial input corpus containing canary leakage samples, and provide the sanity check stripped programs instead of the original StackGuard-protected versions. We also calculate the time cost in generation through dynamic monitoring of the engines' output at the same time. Table 5 shows the results. We can see that CanaryExp is able to generate canary leaking input at low time cost while other engines generally fail in the given time quota. The root cause of this, we believe, lies in the uniquely proposed canary leakage forger CanaryExp. Through introspecting upon the output stream, this component dynamically transforms some execution state into a canary leaking state when identifying the output context satisfying the necessary transformation condition, therefore establishing the advantage for CanaryExp in this facet over other known test case generation engines. We can also conclude that CanaryExp is able to generate incremental POC samples for the benchmark programs more effectively than other engines. We believe this is due to CanaryExp's unique canary sensitivity. As for coverage-guided fuzzing engines like AFL and AFL++, they generally distinguish test cases by their branch coverage feedbacks. Under this paradigm, a specific branch coverage map generally groups an equivalent set of test cases, with the first test case exhibiting this branch coverage selected as the representative; other test cases subsequently generated following the same branch coverage would simply discarded in future test case generation processes. However, this is not the case for canary leaking input generation. As we can see from Figure 1, the path taken by the canary leakage sample can be easily reached through simple fuzzing. Therefore, though carrying evident canary leakage feature, the canary leaking input would simply not be deemed as interesting and discarded immediately as some cheap representative already exists. For S2E and QSym, the symbolic executor part would also ignore the canary leaking context information during path space exploration. Even though the input corpus provided to S2E and QSym already implies some canary leakage event, they only enforce blind path exploration obtaining poor efficiency in incremental POC generation. CanaryExp, \begin{table}
\begin{tabular}{c c} \hline
**Engine** & **Configuration Setup** \\ \hline CanaryExp & 1 instance of AFL++ slave mode, 1 instance of S2E \\ \hline AFL & 1 instance for AFL master mode, 1 instance for AFL slave mode \\ \hline AFL++ & 1 instance for AFL++ master mode, 1 instance for AFL++ slave mode \\ \hline S2E & 2 instances of S2E, working in parallel \\ \hline QSYM & 1 instance of a PIN-based concolic executor, 1 instance for AFL slave mode \\ \hline \end{tabular}
\end{table} 
Table 4: Engine configuration.