###### Abstract  
Capture the Flag (CTF) is an important form of competition in cybersecurity, which tests participants' knowledge and problem-solving abilities. We propose a multi-agent framework based on large language models to simulate human participants and attempt to automate the solutions of common CTF problems, especially in cryptographic and mis-cellaneous challenges. We implement the collaboration of multiple expert agents and access external tools to give the language model a basic level of practical competence in the field of cybersecurity. We primarily test two capabilities of the large model: to analyze, reason, and determine solutions to CTF problems, and to assist with problem-solving by generating code or utilizing unannotated existing external tools. We construct a benchmark based on the puzzles from the book \"Ghost in the Wires\" and the THUCTF competition. The experiment results showed that our agents performed well on the former and were significantly improved with some human hints, compared with related work. We also discuss the challenges that language models face in cybersecurity challenges and the effect of leveraging reasoning models. c 2025 15 7159 11 May 2025 21 June 20  
methods are often employed as an auxiliary to human experts, with their effectiveness highly dependent on the expertise and judgment of users. The emergence of Large Language Models (LLMs) has endowed machines with human-like capabilities in learning and reasoning, paving the way for the development of intelligent agents capable of autonomous decision-making and task execution. However, due to the wide-ranging topics, non-linear thinking, and complex interactions involved in CTF challenges, current research on using LLMs to automatically solve these problems remains limited, exhibiting a low degree of automation. Our research represents an exploration and an attempt in this direction. Specifically, to simulate the general human problem-solving process (involving understanding the problem, devising strategies, and testing potential solutions), we designed a multi-agent framework composed of distinct modules for analysis (acting with human reasoning), decision-making (generating structured commands), and execution (interacting with the operating system environment). These modules work collaboratively in a cyclic \"think-decide-act-reflect\" manner, allowing the system to iteratively explore challenges and gradually approach a solution. In implementation, the large language model agent interacts with the system by executing commands, writing code, browsing and interacting with web pages, and reading and analyzing files. That enables the agent to autonomously carry out the entire problem-solving process in the way that the problems were originally designed. Human intervention is not necessary, but users can still monitor the agent's behavior and provide guidance to steer its reasoning when necessary. To measure the ability of LLM agents in cybersecurity automation, we constructed a CTF problem dataset from the attached challenges of cybersecurity biography _Ghost in the Wires_ (GITW) and from past THUCTF competitions hosted by Tsinghua University, focusing primarily on Crypto and Misc categories. These two sources represent distinct levels of difficulty: GITW for effectively evaluating LLM agent performance, and THUCTF for realistic open-ended scenarios. We evaluated our agent and baseline models on this dataset. The experimental results show that, on the GITW test set, our agent outperformed the ChatGPT-4o (2024-08-06 version) base model by 14.5% when both were provided with human hints, and achieved a performance gain of 2.7% to 5.3% without any hints. On the THUCTF benchmark, our agent was able to generate correct solution plans for 10% of the problems and independently solved 4.3% of them. Our main contributions are as follows:  
* We develop a multi-agent framework that utilizes the LLM to emulate human-like reasoning and decision-making, as well as external tools to solve CTF challenges in an autonomous manner.