Title: NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
Benchmark: NYU CTF Bench
Summary: The benchmark aims to assess the capability of LLMs in solving CTF challenges autonomously.  The NYU CTF Bench includes  CTF challenges from NYU’s CSAW cybersecurity events.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: cybersecurity 
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: CTF challenges
Task item definition: CTF challenge with challenge description, files, host/port information, and validation requirements, etc
Task source: Human exam questions (e.g. GRE questions), Expert-crafted task examples (e.g. hand-written examples)
Task source detail: tasks are sourced from NYU’s annual CSAW CTF competitions
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 200
Dataset metadata available: Yes
Dataset metadata detail: challenge name, description, category, difficulty level, host/port info, files needed, and valid flag
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: challenge type
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They discuss current tool support limitations and the need for more diverse sources of CFT challenges
Taxonomy: Agents / Web
New taxonomy: General Application / Web Agents