Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
  of Language Models
Authors: name: Andy K. Zhang; name: Neil Perry; name: Riya Dulepet; name: Eliot Jones; name: Justin W. Lin; name: Joey Ji; name: Celeste Menders; name: Gashon Hussein; name: Samantha Liu; name: Donovan Jasper; name: Pura Peetathawatchai; name: Ari Glenn; name: Vikram Sivashankar; name: Daniel Zamoshchin; name: Leo Glikbarg; name: Derek Askaryar; name: Mike Yang; name: Teddy Zhang; name: Rishi Alluri; name: Nathan Tran; name: Rinnara Sangpisit; name: Polycarpos Yiorkadjis; name: Kenny Osele; name: Gautham Raghupathi; name: Dan Boneh; name: Daniel E. Ho; name: Percy Liang
Published: 2024-08-15 17:23:10

Abstract: Language Model (LM) agents for cybersecurity that are capable of autonomously
identifying vulnerabilities and executing exploits have the potential to cause
real-world impact. Policymakers, model providers, and other researchers in the
AI and cybersecurity communities are interested in quantifying the capabilities
of such agents to help mitigate cyberrisk and investigate opportunities for
penetration testing. Toward that end, we introduce Cybench, a framework for
specifying cybersecurity tasks and evaluating agents on those tasks. We include
40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
competitions, chosen to be recent, meaningful, and spanning a wide range of
difficulties. Each task includes its own description, starter files, and is
initialized in an environment where an agent can execute bash commands and
observe outputs. Since many tasks are beyond the capabilities of existing LM
agents, we introduce subtasks, which break down a task into intermediary steps
for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
guidance, we find that agents are able to solve only the easiest complete tasks
that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
GPT-4o having the highest success rates. Finally, subtasks provide more signal
for measuring performance compared to unguided runs, with models achieving a
3.2\% higher success rate on complete tasks with subtask-guidance than without
subtask-guidance. All code and data are publicly available at
https://cybench.github.io