Title: kGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution
Benchmark: KBENCHSYZ
Summary: The paper introduces KGYM, a platform, and KBENCHSYZ, a dataset, to benchmark Large Language Models on the task of resolving real-world Linux kernel crashes. The KGYM platform provides a software engineering environment for large-scale experiments on the Linux kernel, including compiling, running, detecting crashes, inspecting logs, and querying/patching the codebase. KBENCHSYZ is a benchmark dataset derived from real-world Linux kernel bugs, containing crash reports, reproducers, and developer-written fixes. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: code writing, data analysis, tool navigation, grounding, user interaction
Phenomenon defined by authors: Yes
Phenomenon definition: "Given a state of the Linux code‑base, a crash report, and the crash‑inducing input, the target is to repair the code‑base such that the input no longer triggers a crash." (page 2)
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: Only crashes that (i) were found by Syzkaller and (ii) meet reproducibility criteria are included ​
Task definition: The model receives the crash trace plus selected C source files and must output a single git‑diff patch that, once applied and re‑compiled, prevents the reproducer from crashing the kernel.
Task item definition: One row/item = the 7‑tuple (buggy commit, config, reproducer, fix commit, parent commit, crash trace, gold patch) 
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Each item is an actual Linux‑kernel bug report, crash trace and maintainer patch taken from the public Syzbot tracker.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Only the Syzkaller reproducer is run.
Dataset size: 279
Dataset size detail: For experiments, subsets are used depending on the LLM's context window and the retrieval method (Oracle or BM25), ranging from 117 to 275 bugs. 
Dataset metadata available: Yes
Dataset metadata detail: subsystem, crash_type, fix_type, lines_changed, files_changed, year, email_discussions
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple sum/mean - raw percentages
Metric subscores: Yes
Metric subscores detail: Scores are provided by Fix Type (Single Line, Single Function but Multiline, Multi Function but Single File, Multi Files) and by bug localisation efficacy (complete overlap, partial overlap).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Authors discuss validity: No
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents
Procedural notes: The authors discuss the motivation for KBENCHSYZ and its realism, but they do not explicitly evaluate or argue for the construct validity of the benchmark.