# Empowering LLMs with Toolkits: An Open-Source Intelligence Acquisition Method
## 6 Conclusions  
Open-source intelligence (OSINT) holds a crucial position in today's information society. It aids decision-makers in gaining insights into global trends, predicting and countering potential threats, and plays an irreplaceable role in various fields, such as cybersecurity, market analysis, crisis management, and academic research. Particularly in the rapidly evolving cyberspace, OSINT has become a key method in identifying cyberattacks, tracking malicious actors, and strengthening defense strategies. However, the current mainstream methods of obtaining OSINT primarily involve using scanning tools or querying relevant websites and platforms, which require substantial time and human resources. To address this issue, this paper proposes an intelligent OSINT acquisition method based on LLMs, providing an efficient and convenient way to obtain open-source security intelligence. We propose a framework incorporating chain-of-thought technology, aimed at empowering LLMs to utilize tools to obtain OSINT. Based on this framework, an agent was constructed. To overcome the knowledge limitations and timeliness issues of LLMs that arise from the reliance on training data, we developed a toolkit for dynamic use by the agent. The tools within the toolkit address the deficiencies of LLMs in acquiring real-time and in-depth knowledge in vertical domains. Users can interact with the agent by asking questions to obtain the required OSINT. To validate the effectiveness of our method on the LLMs' performance in OSINT acquisition, we designed a test dataset comprising four different types of OSINT questions and established four evaluation metrics. These metrics assess the performance in terms of question answering, tool usage, parameter passing, and the framework's impact on model performance. Comprehensive testing and the statistical analysis of the experimental results demonstrated that our method significantly improved the LLM's answer accuracy on the test dataset, while minimally disrupting the model's existing problem-solving capabilities. This indicates that our method can accurately obtain the required OSINT. In conclusion, this paper presents a novel framework for OSINT acquisition that integrates chain-of-thought reasoning with tool utilization to address the limitations of LLMs in knowledge coverage and timeliness. By enabling logical reasoning and dynamic tool access, the proposed approach demonstrates significant potential to improve the efficiency of OSINT acquisition. However, this work has several limitations. First, while the framework is designed to be flexible and adaptable--allowing the agent to dynamically access and utilize appropriate tools based on the problem at hand--its functionality is constrained by the tools provided. The initial setup, which involves defining tool functionalities and integrating them with the agent, requires manual effort. This reliance on manual configuration may limit its adaptability and scalability, particularly in domains with rapidly evolving requirements. Second, while the framework is domain-agnostic, significant customization is required to adapt it to fields such as medical diagnosis, finance, or education. Third, the effectiveness of the framework depends on the quality and timeliness of the external tools that it integrates. Lastly, as the framework builds on existing LLMs, it inherits their limitations, including occasional reasoning errors and susceptibility to biases. In future work, we aim to expand the framework to new domains by automating tool configuration and incorporating advancements in LLM capabilities. Specifically, our future research will focus on automating tool discovery and configuration by leveraging meta-learning techniques or automated API integration, enabling LLMs to dynamically infer tool descriptions and parameter requirements from API documentation or metadata. Conceptualization, H.Z. ; Methodology, X.Y. ; Software, X.Y. ; Validation, J.W. ; Investigation, J.W. ; Resources, H.Z. ; Writing--original draft, X.Y.