Title: RedCode: Risky Code Execution and Generation Benchmark for Code Agents
Benchmark: RedCode
Summary: Benchmark for unsafe code recognition and generation for LLM-based agents.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Safety of code agents
Phenomenon defined by authors: Yes
Phenomenon definition: An agent's ability to execute or generate unsafe code.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are two tasks: code execution (RedCode-Exec) and generation (RedCode-Gen). Execution is defined as the ability to recognise and handle unsafe code. Generation is defined as the ability to generate unsafe code given malicious user instructions.
Task item definition: RedCode-Exec contains risky Python code snippets and instructions to implement the code. Each item in the dataset is a code snippet from a different risky content. For RedCode-Gen, each item is a function signature and a docstring instructing the model to implement an unsafe functionality. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: LLM human-in-the-loop generation for RedCode-Gen 
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 4210
Dataset metadata available: Yes
Dataset metadata detail: Task type (execution vs generation), programming language, mallicious scenario
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Structured response (e.g. valid JSON, API call alone)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Execution-based evaluation
Metric definition detail: Each evaluation script is tailored to the problem.
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric subscores detail: Unsafe domain, task (execution vs generation), programming language.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Discussion of how it only looks at Python and Bash and therefore may not generalise outside of this.
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents