Title: SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Security Benchmarking

LLM Agents

Security Engineering

AI for Security

Automated Evaluation
Abstract: Rigorous security-focused evaluation of large language model (LLM) agents is imperative for establishing trust in their safe deployment throughout the software development lifecycle. However, existing benchmarks largely rely on synthetic challenges or simplified vulnerability datasets that fail to capture the complexity and ambiguity encountered by security engineers in practice. We introduce SEC-bench, the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks. SEC-bench employs a novel multi-agent scaffold that automatically constructs code repositories with harnesses, reproduces vulnerabilities in isolated environments, and generates gold patches for reliable evaluation. Our framework automatically creates high-quality software vulnerability datasets with reproducible artifacts at a cost of only $0.87 per instance. Using SEC-bench, we implement two critical software security tasks to rigorously evaluate LLM agents' capabilities: proof-of-concept (PoC) generation and vulnerability patching. A comprehensive evaluation of state-of-the-art LLM code agents reveals significant performance gaps, achieving at most 18.0% success in PoC generation and 34.0% in vulnerability patching on our complete dataset. These results highlight the crucial steps needed toward developing LLM agents that are more practical, intelligent, and autonomous for security engineering.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes (a) a method to create\na benchmark for vulnerability detection and\nexploit generation and vulnerability repair,\nand (b) a specific benchmark constructed in\nthis way.  It also uses the resulting benchmark\nto (c) measure the effectiveness of existing\nLLMs and agents at these tasks.  We learn\nthat current systems are pretty weak at these\ntasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Techniques to construct training data and\n  evaluation benchmarks.  (Not clear if a specific\n  benchmark will be released to others?)\n- Evaluation of effectiveness of existing LLMs\n  and agents at two important security tasks.\n- The paper is well written, very clear, easy\n  to read.\n- The problem is relevant.  Model providers\n  will likely care.  It also sheds light on\n  the capabilities of existing models and might\n  stimulate new research.\n\nWeaknesses:\n- The tool reduces the cost of creating such\n  a benchmark by about 2x.  Does that matter?\n  Is it worth to spend months of a researcher's\n  time, to save about $200?  I'm not sure.\n- The analysis of ways existing agents fail\n  could be a bit clearer.  It's hard to extract\n  insights from the current writing.\n\nWill you publicly release the benchmark?  Not just\nthe code/framework that could be used to create a\nbenchmark, but your benchmark of 200 issues?  I think\nthat would be valuable to the community.\n\nI think it might be confusing to name your tool/framework\nSEC-bench.  A name like SEC-bench would normally be used\nfor the benchmark, not the tool used to generate the\nbenchmark.  You might pick a different name for the\ntool/framework/technique used to generate the benchmark.\n\nl285: Please clarify the meaning and criteria for each of\nthe four categories.  For instance, what's a failed patch?\nIf a patch doesn't compile doesn't that mean it is incorrect?\nWhat's the difference between incorrect vs failed?\n\nl292: Why would integrating with git avoid incorrect\npatches?  I can see why it might avoid situations where\nit generates something that is not a valid patch and\nthus cannot be merged, but that's different from a\npatch being incorrect.\n\nl341: I don't understand how OSS-Fuzz is relevant here\nor what analogy is being drawn.\n\nQUESTIONS:\nCan you help clarify the categories in Section 3.3?\nAre there any insights we can learn about the biggest\nopportunities for improvement, that might be useful\nto improve these agents?\n\nLIMITATIONS:\nSatisfactory.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and implements an automated benchmark framework SEC-bench for evaluating capabilities of LLM agents in real-world software security engineering tasks. Existing security benchmarks suffer from severe flaws, such as relying on artificially constructed challeges, using synthetic data, or including a large number of real-world vulnerabilities that are difficult to reproduce or have incorrect labels, which prevents accurate measurement of LLM agents' performance in actual security work.\n\nSEC-bench introduces an innovative, multi-agent-based automated verification process (SECVERIFIER). This process decomposes complex vulnerability reproduction and verification tasks into three subtasks executed by specialized LLM agents (Builder, Exploiter, Fixer), systematically handling CVE instances collected from public databases (such as OSV). It can automatically configure isolated compilation and runtime environments (Docker), reproduce vulnerabilities (generate working Proof-of-Concepts), and verify patches. The framework uses the reports of memory safety sanitizers as an \"oracle\" to ensure the objectivity and reliability of the verification process.\n\nThe main contributions of this paper are: 1) proposing the first general framework capable of automatically and scalably constructing high-quality security benchmarks from real-world vulnerability reports; 2) building the SEC-bench dataset and accompanying evaluation tasks; 3) revealing through extensive experiments the current shortcomings of LLM agents in real-world security scenarios, thereby guiding the development of the field.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nStrength: 1. Breaking down complex tasks into three stages—compilation-exploitation-repair—and assigning specialized agents to handle them is a reasonable abstraction for addressing this complex problem. 2. Choosing memory-safe sanitizers as the de facto standard for vulnerability trigger and fix verification is an extremely important advantage. It avoids the subjectivity and inconsistency of manual judgment, making the entire benchmark construction and evaluation process reproducible and trustworthy.\n\nWeaknesses: The success rate of SECVERIFIER (22.3%) is commendable in difficult tasks, but this also means that the 200 instances successfully validated by the framework were \"filtered\" from 898 candidate instances. This may inadvertently introduce selection bias, i.e., SEC-bench may consist mainly of vulnerabilities that are \"easier to process with automated workflows.\" This could lead to future agents optimized for SEC-bench having their capabilities limited to solving such structured or relatively complete vulnerabilities, while lacking generalization ability for more chaotic, complex real-world problems.\n\n---\n\nClarity\n\nStrength: The organization of the paper is very clear, from the motivation of the problem, the limitations of existing work, to framework design, experimental evaluation, and even case analysis of failures, the logical chain is complete.\n\nWeakness: None\n\n---\n\nSignificance\n\nStrengths: 1. This paper accurately identifies the shortcomings of existing work and provides a feasible solution. This work will push the community to shift focus from general coding capabilities to more challenging safety capabilities. 2. the evaluation results (PoC generation < 13%, vulnerability fixes < 32%) contrast sharply with general software engineering benchmarks (SWE-bench > 60%), which is a very important discovery. It sends a clear message to the entire field: the success of LLM in general coding tasks cannot be directly extrapolated to the security domain.\n\nWeakness: None\n\n--- \n\nOriginality\n\nStrengths: Although multi-agent systems and code generation/repair have been studied, the concept of using multi-agent systems for automated construction and verification of security benchmarks is entirely new. \n\nWeakness: None\n\nQUESTIONS:\nQuestion 1: The reliability foundation of the entire SEC-bench framework is the memory safety sanitizer. This is a clever engineering decision, but it also introduces a profound methodological bias. The sanitizer primarily detects specific categories of vulnerabilities (such as buffer overflows, use-after-free), while it is completely powerless against other equally common and dangerous security vulnerabilities (such as logical errors, injection vulnerabilities, improper access control, cryptographic misuse, etc.). Therefore, SEC-bench is essentially a benchmark for memory safety vulnerability fixes, rather than a universal \"software security\" benchmark. Naming and promoting this benchmark as a general \"Software Security Task\" benchmark might mislead the community? Over-optimizing agents that perform well on this benchmark might give rise to a batch of \"one-trick ponies\" that only know how to fix \"bugs reportable by sanitizers\" and are helpless against other types of vulnerabilities? Would this undermine its claimed \"realism\"?\n\nSuggestions: The authors should more explicitly acknowledge this limitation in the paper and provide an in-depth discussion on how this design choice shaped the distribution of vulnerability types in the dataset. A good response should analyze the distribution of vulnerability types (CWE) in SEC-bench compared to all reported vulnerability types in real-world C/C++ projects, and discuss how to expand the coverage of the benchmark in the future by introducing other static/dynamic analysis tools as new \"de facto standards.\" This will demonstrate the authors' deep understanding of the limitations of the methodology.\n\n---\n\nQuestion 2: The Exploiter Agent is the bottleneck in the entire process (success rate of 39.4%), and also the most exciting part from a technical perspective. The authors mention that when there is no existing PoC, the intelligence will \"generate one from scratch by analyzing the root cause, vulnerability patterns, and affected code paths.\"  In the successful verification cases of SECVERIFIER , what proportion of PoCs were completed by simply adjusting the example code in the bug report, and how many were truly \"creating from scratch\"? If the latter really occurred, what kind of reasoning process did the LLM demonstrate? Did it simply provide input that caused a crash, or did it construct complex payloads with specific structures (such as meeting specific offsets and lengths)?\n\nSuggestions: The author should provide more detailed case studies or data breakdowns. For example, categorize the successful cases of Exploiter into \"PoC adaptation\" and \"PoC generation from scratch,\" and provide the respective counts. For the latter, provide a specific and convincing case that demonstrates the complete thought process of the LLM, which will greatly enhance the technical depth and credibility of the paper. If the majority of successful cases belong to \"adaptation,\" the wording should be adjusted to better reflect the actual situation. \n\n---\n\nQuestion 3: The ablation experiments in Table 3 strongly demonstrate that SECVERIFIER (multi-agent) outperforms CODEACT (single-agent). The authors attribute this to the multi-agent method. However, this advantage may stem from two coupled factors: 1) the structural advantages of task decomposition itself; 2) the prompts designed for each specialized agent, which are more context-focused. A prompt designed for a \"Builder\" would contain more specific instructions and context about compilation compared to a generic \"Please verify this CVE\" prompt. Can we decouple these two? A well-designed single agent, if given a highly structured, step-by-step \"Chain-of-Thought\" prompt, requiring it to sequentially play the roles of \"Builder,\" \"Exploiter,\" and \"Fixer,\" and providing specific instructions for each step, can it achieve performance comparable to a multi-agent system?\n\nSuggestions: This is a profound research question. The authors could conduct a supplementary experiment or delve deeper into this issue in the discussion. A strong response would acknowledge this ambiguity and discuss its implications for future agent design: whether we are pursuing multiple independent agents or a single, more powerful agent capable of efficient, structured role-playing within a single conversation?\n\nLIMITATIONS:\nThe mentioned limitations are honest and important! However, there might be several extras limitations: \n- Selection Bias in Benchmark Construction: As I mentioned in the weakness of the \"Quality\" section, the author should explicitly discuss the potential selection bias introduced by the 22.3% success rate, and its impact on the representativeness of the SEC-bench dataset.\n\n- Potential Negative Social Impact: The Exploiter Agent in SECVERIFIER is essentially an automated vulnerability exploitation tool. The authors should discuss the possibility of this technology being misused by malicious actors. For example, a powerful Exploiter Agent could be used to quickly generate PoCs for newly discovered 0-day vulnerabilities, thereby accelerating cyberattacks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SEC-bench, a benchmarking framework designed to evaluate LM agents on real-world software security engineering tasks. The method addresses the limitations of existing security benchmarks which rely on synthetic tasks or simplified vulnerability cases. SEC-bench uses an effective multi-agent scaffold that automatically constructs code repositories, reproduces vulnerabilities in isolated environments, and generates gold patches for evaluation at just $0.87 per instance. The framework is tested with actual CVE instances from open-source projects with two tasks: Proof-of-Concept (PoC) generation (12.5% success rate), Vulnerability patching (31.5% success rate).\n\nOverall, SEC-bench represents a valuable contribution to security engineering benchmarking of LM agents. The low success rates on PoC generation and patching highlight the significant challenges in this domain. Although the limitations in current language support is a major obstacle to larger impact, the framework's cost-effective automation make it a promising foundation for future work.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper tackles a real gap in security benchmarking for LMs by moving from synthetic vulnerabilities to real-world problems.\n- The agent scaffolding of the framework can be a good foundation for future work. The architecture achieves 85.7% improvement over single-agent baseline, and remains cost-effective, costing $0.87 per verified instance.\n- The paper uses the proposed framework to evaluate multiple state-of-the-art LM agents and revealed significant performance gaps.\n\n**Weaknesses**\n- Limited Language Support. The framework only supports C/C++ and is limited to memory safety vulnerabilities detectable by sanitizers, excluding many modern languages and web vulnerabilities.\n- The current low success rate (22.3% verification, 12.5% for PoC generation, 31.5% for patching) raises concerns about the framework's validity for benchmarking models. While the authors mention other possible tasks like fuzzing and vulnerability discovery, only two tasks are actually implemented and evaluated.\n- The benchmark currently only contains 200 verified instances from 898 candidates, despite the cost-effective automation.\n\nQUESTIONS:\n- Can you elaborate on how the framework can be extended to support more languages and more vulnerability types?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper introduces SEC-bench, a novel and fully automated framework for benchmarking the performance of Large Language Model (LLM) agents on real-world software security tasks. The authors argue that existing security benchmarks are inadequate, as they often rely on synthetic challenges, simplified vulnerability datasets, or suffer from poor reproducibility, failing to capture the complexity and ambiguity that security engineers face in practice.

STRENGTHS AND WEAKNESSES:
Strengths:
1. he work addresses a critical and timely problem: the rigorous evaluation of LLM agents' security capabilities. The proposed SEC-bench framework is highly novel, presented as the first end-to-end automated system for constructing benchmarks from authentic, real-world vulnerabilities.
2. he design of SEC-bench is robust. The multi-agent architecture of SECVERIFIER, which decomposes the complex verification process into specialized sub-tasks (building, exploiting, fixing), is a logical and effective approach.

Weaknesses:
1. The framework's current scope is primarily limited to memory safety vulnerabilities within C/C++ projects. This is far from sufficient for generalization to real applications.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07