## 4 Evaluation
### Effectiveness  
In this section, we use both the mix binary dataset and ubuntu binary dataset to measure the effectiveness of BBDetector from three metrics: precision (P), recall (R) and F1-score (F1). For mix binaries; there are a total of 323 real reuses. Based on these reuse labels, we compare BBDetector with B2 SFinder and ModX and summarize the experiment results of the mix binary dataset in Table 3. Since ModX does not open its source code, we present the original experiment results in the paper [11].  
In Table 3, we can observe that BBDetector outperforms B2 SFinder and ModX in terms of these three metrics. B2 SFinder only considers basic syntactic features. To compare with it, BBDetector obtains an F1-score of 63.5%, 11.21% higher than that of B2 SFinder, demonstrating the necessity and usefulness for also taking into account fine-grained function-level semantic characteristics. ModX considers the FCG graph and features of functions accessing the same data, which leads to a huge time consumption when considering the expensive graph matching process and is also based on syntactic feature when considering function features. To compare with it, BBDetector obtains a 1.11% higher F1-score than that of ModX, proving the BBDetector validity of choosing anchor functions for semantic matching in BBDetector, even without considering the expensive graph matching.  
In Table 4, we list the detailed TPL detection results for the ubuntu binary dataset. The first column presents the specific target binary file name and the second column presents the number of reused third-party libraries in each of them. The rest columns present the number of TP (true positives), FP (false positives) and FN for BBDetector, B2 SFinder and ModX, respectively. As shown in Table 4, BBDetector achieves the highest F1-score of 77%, 14.93% higher than that of B2 SFinder and 1.32% higher than that of ModX.  
Moreover, we analyze and evaluate the effectiveness against some vulnerable version of libraries. In our TPL database, there are vulnerable vesion of libraries, some of which is as shown in Table 5. For example, library libbsd.so.0 contains vulnerabilities CVE-2019-20367 and CVE-2016-2090. Because the detection results of ModX do not give the contained specific library information, we only briefly analyze the result of BBDetector and B2 SFinder on some vulnerable version of libraries. The target binary program curl reuses the vulnerable library libcurl.so.4. Since curl only partially reuses this library and the ratio of the number of reused functions is only 11.28% (number of common functions: 787, total number of functions in the library: 6975), the number rate of reused synacit features is small (the common string percent is only 4.3%) and B2 SFinder does not identify this reuse relationship. However, BBDetector also considers function semantic features and measures the average similarity score (0.83, which is greater than the threshold) of the anchor functions between them; thus, BBDetector can correctly detect the reuse relationship between them.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **TP** & **FP** & **FN** & **Precision** & **Recall** & **F1-Score** \\ \hline BBDetector & 162 & 25 & 161 & 86.6\% & 50.2\% & 63.5\% \\ B2 SFinder & 138 & 22 & 185 & 86.3\% & 42.7\% & 57.1\% \\ ModX & / & / & / & 85.6\% & 49.6\% & 62.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: TPL detection on mix binary dataset.