# Empowering LLMs with Toolkits: An Open-Source Intelligence Acquisition Method
## 5 Results and Discussion
### Impact on LLMs' Response Accuracy  
After conducting a series of experiments based on the experimental setup outlined in Section 4.3, we have tallied the response accuracy for each experiment. The results are summarized in Table 2 below, with bolded values highlighting instances where the agents' accuracy surpassed that of the corresponding baseline models. From the experimental results, it can be observed that the base models of the Qwen series have low accuracy when answering questions related to the CVE and domain name. The accuracy of ChatGPT-3.5 is also not ideal in this context. However, after incorporating our framework, the corresponding agents of these base models, by utilizing tools, significantly improve the accuracy of their answers. This demonstrates that our method can effectively assist LLMs in acquiring OSINT. When comparing the models' performance on questions pertaining to CVE and domain names versus license-related issues, a striking disparity emerges. Regardless of their parameter size, the baseline models display markedly higher accuracy in answering questions about licenses. This observation suggests that, through their training, the models have acquired a substantial body of knowledge concerning licenses. The contrasting capabilities of LLMs in handling license information and CVE information can be attributed to the dynamic nature of vulnerability databases. Unlike license information, which tends to remain relatively stable over time, CVE information undergoes frequent updates as new vulnerabilities \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **CVE Questions** & **License Questions** &
\begin{tabular}{c} **Domain Name** \\ **Questions** \\ \end{tabular} & **Other** \\ \hline Qwen-7B-Chat & 13.3\% & 94.0\% & 48.0\% & 87.3\% \\ Qwen-7B-Chat+ 1  & **82.7\%** & 82.0\% & **53.3\%** & 83.3\% \\ Qwen-14B-Chat & 1.3\% & 88.0\% & 1.3\% & 92.7\% \\ Qwen-14B-Chat+ & **86.7\%** & **91.3**\% & **63.3**\% & 90.7\% \\ Qwen-72B-Chat & 10.0\% & 90.0\% & 12.7\% & 94.0\% \\ Qwen-72B-Chat+ & **96.0**\% & **98.0**\% & **44.7**\% & **95.3**\% \\ ChatGPT-3.5 & 50.0\% & 98.0\% & 7.3\% & 94.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Response accuracy for question queries. are discovered and cataloged. Similarly, the Whois database, which contains domain name registration information, also undergoes real-time updates as domains are registered, transferred, or expire. The rapid pace of these updates means that both the CVE and the Whois databases must continuously evolve to maintain their completeness and timeliness. Given that LLMs' knowledge is limited to their training data, they are unable to accurately answer questions that arise after their knowledge base has been updated. To access the most recent information, they need to rely on external tools. To assess the impact of our method on the performance of LLMs, we conducted a statistical analysis focusing on the change in response accuracy when the framework was applied. The results are summarized in Table 3, where \(\Delta_{1}\) represents the difference in response accuracy between the agent and its baseline model, and \(\Delta_{2}\) represents the difference in response accuracy between the agent and ChatGPT-3.5. Through the analysis of the data in Table 3, it can be observed that, after integrating our framework, there is a slight decrease in the accuracy of Qwen-7B-Chat and Qwen-14B-Chat when handling license-related questions and other categories of questions. This phenomenon points to potential perturbations introduced by our framework that may affect the models' performance in certain scenarios. However, this downward trend significantly diminishes with the increase in the model parameter size. This suggests that our framework has a minimal impact on the understanding and problem-solving capabilities of LLMs for other tasks and does not significantly interfere with the models' existing problem-solving abilities.