Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Authors: name: Terry Yue Zhuo, name: Dingmin Wang, name: Hantian Ding, name: Varun Kumar, name: Zijian Wang

Large language models (LLMs) have demonstrated exceptional capabilities when
trained within executable runtime environments, notably excelling at software
engineering tasks through verified feedback loops. Yet, scalable and
generalizable execution-grounded environments remain scarce, limiting progress
in training more capable ML agents. We introduce CTF-Dojo, the first
large-scale executable runtime tailored for training LLMs with verifiable
feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style
challenges containerized in Docker with guaranteed reproducibility. To enable
rapid scaling without manual intervention, we develop CTF-Forge, an automated
pipeline that transforms publicly available artifacts into ready-to-use
execution environments in minutes, eliminating weeks of expert configuration
traditionally required. We trained LLM-based agents on just 486 high-quality,
execution-verified trajectories from CTF-Dojo, achieving up to 11.6% absolute
gains over strong baselines across three competitive benchmarks: InterCode-CTF,
NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1,
establishing a new open-weight state-of-the-art that rivals frontier models
like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a
benchmark for executable-agent learning, CTF-Dojo demonstrates that
execution-grounded training signals are not only effective but pivotal in
advancing high-performance ML agents without dependence on costly proprietary
systems.