# Empowering LLMs with Toolkits: An Open-Source Intelligence Acquisition Method
## 5 Results and Discussion
### Tool Utilization and Parameter Transmission  
Based on the deliberation process of LLMs before providing the final answer, we have compiled statistics on their usage of tools when answering questions, as well as the parameter transfer scenarios during tool utilization. Table 4 presents the tool usage patterns of each agent during the problem-solving process, Table 5 outlines the selection of tools, and Table 6 depicts the parameter transfer situation. These data serve to provide insights into the strategies and methods employed by LLMs in problem-solving, as well as how parameters are passed during tool utilization. An analysis of this data enables a deeper understanding of the behavior and decision-making processes of LLMs during OSINT acquisition. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **CVE Questions** & **License Questions** &
\begin{tabular}{c} **Domain Name** \\ **Questions** \\ \end{tabular} & **Other** \\ \hline Qwen-7B-Chat+ 1  & 97.3\% & 89.3\% & 94.7\% & 19.3\% \\ Qwen-14B-Chat+ & 98.7\% & 99.3\% & 98.7\% & 15.3\% \\ Qwen-72B-Chat+ & 100.0\% & 100.0\% & 99.3\% & 6.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Tool utilization rate. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & \begin{tabular}{c} **CVE Questions** \\ \(\mathbf{\Delta_{1}}\) \\ \end{tabular} & \begin{tabular}{c} **License Questions** \\ \(\mathbf{\Delta_{2}}\) \\ \end{tabular} & \
\begin{tabular}{c} **Domain Name Questions** \\ \(\mathbf{\Delta_{1}}\) \\ \end{tabular} &
\begin{tabular}{c} **Other Questions** \\ \(\mathbf{\Delta_{2}}\) \\ \end{tabular} \\ \hline Qwen-7B-Chat & \(\uparrow\)\(\downarrow\)69.4\% & \(\uparrow\)32.7\% & \(\downarrow\)\(\downarrow\)12.0\% & \(\downarrow\)16.0\% & \(\uparrow\)5.3\% & \(\uparrow\)46.0\% & \(\downarrow\)14.0\% & \(\downarrow\)10.7\% \\ Qwen-14B-Chat & \(\uparrow\)85.3\% & \(\uparrow\)36.7\% & \(\downarrow\)3.3\% & \(\downarrow\)6.7\% & \(\uparrow\)62.0\% & \(\uparrow\)56.0\% & \(\downarrow\)2.0\% & \(\downarrow\)3.3\% \\ Qwen-72B-Chat & \(\uparrow\)86.0\% & \(\uparrow\)46.0\% & \(\uparrow\)8.0\% & 0.0\% & \(\uparrow\)32.0\% & \(\uparrow\)37.4\% & \(\uparrow\)1.3\% & \(\uparrow\)1.3\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Framework impact on response accuracy. By analyzing the data from Tables 2 and 4, it can be observed that, compared to answering other types of questions, the baseline LLMs of the Qwen series have significantly lower accuracy when addressing questions related to CVE and domain names. Interestingly, the corresponding agents show the high usage of tools, and the tool usage rates increase with the size of the model parameters. This indicates that our method can effectively assist the baseline LLMs in addressing problems beyond their inherent capabilities, especially in the acquisition of OSINT. According to the design of our toolkit, the priority of the tools available for agents to utilize varies depending on the type of problem that they are addressing. When tackling CVE-related questions, the vulnerability information search tool holds the highest priority, whereas, for domain name queries, the Whois query tool is prioritized. To monitor the agents' tool usage, we tabulated their tool selection circumstances during problem-solving, as shown in Table 5. An analysis of the data in Table 5 reveals that the agents demonstrate a high level of accuracy in selecting appropriate tools for specific problems, with a notable trend indicating that larger model parameters correlate with higher accuracy in tool selection. This finding substantiates the efficacy of our framework in assisting the agents to effectively address targeted problems, thereby affirming its capability to optimize problem-solving strategies in a context-specific manner. Regarding the responses to questions related to licenses, there were instances where the agents based on smaller-parameter models exhibited lower response accuracy compared to their baseline models. This can largely be attributed to the imprecision in vector database retrieval.