Title: Selene: Pioneering Automated Proof in Software Verification
Benchmark: Selene
Summary: A benchmark for automated proof generation in software verification.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Automated proof generation in software verification.
Phenomenon defined by authors: Yes
Phenomenon definition: Write proofs that project-level software has the required properties. These can then be formally checked by an independent prover.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: The subject LLM’s goal is to write proofs for the given specifications from seL4 (a real-world industrial-level operating system microkernel) and pass the verification.
Task item definition: Generation of a target lemma, which is then merged with the other lemmas to verify the proof. You are given the specifications to prove. Write the proofs in the Isabelle language. Prompted with "several" demonstrations of perfect proofs in-context.
Task definition detail: This benchmark is in a complex domain.
Task source: Real task examples (e.g. GitHub issues)
Task source detail: All tasks are taken from a single piece of software (seL4) as this has ground truth proofs.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 360
Dataset metadata available: Yes
Dataset metadata detail: Difficulty level of the lemmas.
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Generated proof verified by an independent prover system.
Metric aggregation: Simple Mean
Metric statistics: Mean, 
Metric subscores: Yes
Metric subscores detail: Difficulty of the target lemma.
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Identify that software verification requires two stages: the prerequisite specification stage and the proof stage. Their benchmark only considers the second stage.
Taxonomy: Code Generation / 
New taxonomy: General Capability / Logical Reasoning
Other notes: This was quite far outside my skillset.