Title: Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI Agents, Cybersecurity, Risk

Abstract: We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment, conducting a head-to-head comparison on a large university network consisting of ~8K hosts across 12 subnets. Ten cybersecurity professionals are evaluated alongside 6 existing open-source AI agents and ARTEMIS, our new agent scaffold. ARTEMIS is a multi-agent framework that incorporates dynamic task prompt generation, the ability to instantiate an arbitrary number of sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 11 valid vulnerabilities with 82\% accuracy and outperforming 8 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed most human participants, ARTEMIS demonstrated comparable technical sophistication and submission quality to strong participants. Key capability gaps were also identified as AI agents exhibit higher false-positive rates (18–43\%). Favorable AI agent attributes include systematic enumeration and parallel exploitation as well cost — given certain ARTEMIS variants have an operating cost as low as 18 USD per hour versus 60 USD per hour for the average professional pentester.