Benchmarking and Enhancing LLM Agents in Localizing Linux Kernel Bugs

## Abstract

Abstract The Linux kernel is a critical system, serving as the foundation for numerous systems. Bugs in the Linux kernel can cause serious consequences, affecting billions of users. Fault localization (FL), which aims at identifying the buggy code elements in software, plays an essential role in software quality assurance. While recent LLM agents have achieved promising accuracy in FL on recent benchmarks like SWE-bench, it remains unclear how well these methods perform in the Linux kernel, where FL is much more challenging due to the large-scale code base, limited observability, and diverse impact factors. In this paper, we introduce LinuxFLBench , a FL benchmark constructed from real-world Linux kernel bugs. We conduct an empirical study to assess the performance of state-of-the-art LLM agents on the Linux kernel. Our initial results reveal that existing agents struggle with this task, achieving a best top-1 accuracy of only 41.6% at file level. To address this challenge, we propose LinuxFL + , an enhancement framework designed to improve FL effectiveness of LLM agents for the Linux kernel. LinuxFL + substantially improves the FL accuracy of all studied agents (e.g., 7.2% - 11.2% accuracy increase) with minimal costs. Data and code are available at https://github.com/FudanSELab/LinuxFLBench .

## 1 Introduction

The Linux kernel is a critical system which serves as the foundation for numerous operating systems, servers, and embedded systems. To date, the Linux kernel has undergone continuous evolution over several decades, involving thousands of developers and billions of active users (linux_kernel_history_report,). Given the widespread adoption of the Linux kernel, bugs in the Linux kernel can cause serious consequences, affecting a vast number of users. Therefore, extensive research has been dedicated to developing automated software quality assurance techniques (e.g., testing (bligh2006fully,; DBLP:conf/hpcc/ChenWYZCYM13,; DBLP:conf/asplos/YangZZ25,) and debugging (DBLP:conf/kbse/BissyandeRLM12,; drgn,; DBLP:conf/usenix/SerranoNTJ0LM20,; DBLP:conf/eurosys/JeongJLLSK23,)) specifically for the Linux kernel.

Fault localization (FL), which aims at identifying the buggy code elements (e.g., files or functions) in software, plays a critical role in software quality assurance. Given the codebase of the buggy software and the bug report (e.g., a user-reported bug symptom description), automated FL techniques return a list of buggy code elements ranked by their suspiciousness (i.e., the probability of being buggy). In particular, accurate FL is a prerequisite for bug fixing, as a bug cannot be resolved without correctly identifying the faulty code location.

Traditional FL techniques mainly leverage heuristics (DBLP:conf/prdc/AbreuZG06,; DBLP:journals/tr/WongDGL14,) or information retrieval (IR) (DBLP:conf/icse/ZhouZL12,; DBLP:conf/kbse/SahaLKP13,) to identify buggy code elements. More recently, with the advance in large language models (LLMs), LLM agents DBLP:journals/corr/abs-2409-02977 have demonstrated remarkable accuracy in FL. By equipping LLMs with the tool invocation capability, agents can autonomously navigate through the codebase to identify the buggy location. For example, the state-of-the-art agents such as SWE-Agent (DBLP:journals/corr/abs-2405-15793,), AutoCodeRover (DBLP:conf/issta/0002RFR24,), Agentless (DBLP:journals/corr/abs-2407-01489,), achieve around 70% accuracy in localizing buggy files for Python software in the benchmark SWE-bench (DBLP:conf/iclr/JimenezYWYPPN24,).

Although achieving promising FL effectiveness, existing agents have been mainly evaluated on general software at moderate scales. It remains unclear how existing agents perform in complex, large-scale software systems like the Linux kernel. In particular, FL in Linux kernel is more challenging than general software due to the following factors. (1) Large-scale Codebase: the Linux kernel has a massive codebase significantly larger than general software. For example, the v5.8 release of Linux kernel includes over 69K files and 28M lines of code (linux_kernel_history_report,), which is over 30 times the scale of even the largest project in the most widely-used benchmark SWE-bench. (2) Limited Observability: given the real-time nature of the Linux kernel with the need to minimize overhead, the kernel restricts the use of instrumentation and logging mechanisms during runtime. Additionally, the kernel operates in a privileged mode, isolated from user space. As a result, user-reported bug descriptions often lack detailed runtime information and debugging hints, creating a significant gap between the user description and the actual root causes. (3) Diverse Impact Factors: kernel bugs are influenced by a wide range of factors, including hardware variability (e.g., architectural configurations) and runtime variability (e.g., system load or timing). These factors lead to an exponentially large reasoning space to accurately diagnose the root causes of errors. Therefore, given the unique challenges and the importance of the Linux kernel, this work aims at investigating the FL effectiveness of state-of-the-art LLM agents in the Linux kernel.

Benchmark. We first build a new benchmark LinuxFLBench of 250 real-world FL tasks for the Linux kernel. Each FL task in LinuxFLBench includes a user-submitted bug report, the buggy Linux kernel codebase, and the ground-truth buggy locations based on the associated commit patches. LinuxFLBench involves a wide range of Linux kernel bugs, spanning over 120 Linux kernel versions and 66 different kernel components. The FL tasks are significantly more challenging than those in SWE-bench, as evidenced by the substantially larger codebases (10–30× more files and lines of code) and more complex bug reports (approximately 1.5× more words).

Empirical Study. On LinuxFLBench, we make the first attempt to evaluate state-of-the-art (SOTA) LLM agents in localizing Linux kernel bugs. Our results reveal the limited FL effectiveness (e.g., 36.8% - 41.6% accuracy) of existing agents in the Linux kernel; moreover, such a FL accuracy is much lower than their performance on general software systems (as seen in SWE-bench), representing a 16.7% - 31.9% accuracy drop.
We further perform bad case analysis to find that existing agents mainly miss the buggy files as they fail to capture the related files or to cover complete root causes of kernel bugs. The results indicate that FL in the Linux kernel is indeed a more challenging task, highlighting the need for building more advanced agents to localize bugs in large and complex software systems like the Linux kernel.

Technique. Inspired by our study above, we further propose an enhancing framework LinuxFL^+, which improves the FL effectiveness of existing agents for the Linux kernel. LinuxFL^+ incorporates two expansion strategies to refine the prediction results of existing agents: directory-aware expansion to include buggy files based on the repository structure, and potential cause expansion to identify buggy files based on the additional bug knowledge from Linux kernel mailing list (LKML) (LKML,). Our evaluation results show that LinuxFL^+ can substantially improve the FL accuracy of all studied agents (e.g., 7.2% - 11.2% accuracy increase) with minimal costs. Moreover, the ablation analysis confirms the contribution of each expansion strategies.

## 2 Background and Related Work

FL Task Definition.
Given the buggy codebase and the bug report, FL techniques identify buggy code elements (e.g., files or functions).
Formally, let a codebase be represented as a set of code elements (e.g., files or functions), $\mathcal{C}=\{ce_{1},ce_{2},\dots,ce_{N}\}$, where $N$ denotes the total number of code elements. A bug report $BR$ typically includes a title, a description, and optional metadata (e.g., component and hardware information in the context of Linux kernel), and can be expressed as $BR=(title,desc,meta)$. A FL task can be modeled as:

$$ $\textbf{FL}:BR,\mathcal{C}\to list(\mathcal{C}),$ (1) $$

where $list(\mathcal{C})$ denotes a list of code elements that ranked by their suspiciousness (i.e., the probabilities of being buggy).

Existing FL techniques. FL techniques have been extensively studied in literature:

- •
Coverage-based FL. Besides bug reports, some FL techniques leverage test coverage to identify buggy locations, such as SBFL (DBLP:conf/prdc/AbreuZG06,; DBLP:journals/tr/WongDGL14,), GNN-based FL (DBLP:conf/sigsoft/LouZDLSHZZ21,), AutoFL (DBLP:journals/pacmse/KangAY24,), and AgentFL (DBLP:journals/corr/abs-2403-16362,). However, coverage and executable failure-triggering tests are not always available in practice. Especially for the large systems like Linux kernel, users report bugs by textually describing the error symptoms. Therefore, coverage-based FL cannot be applied to the Linux kernel when only bug reports are available, which thus is not included in this work.
- •
Information Retrieval (IR) Based FL.
FL can be formulated as an information retrieval (IR) problem, where a bug report serves as a query to rank code files by relevance. Existing IR-based FL techniques use various similarity measures, such as Vector Space Model (VSM) (DBLP:conf/icse/ZhouZL12,; DBLP:conf/kbse/SahaLKP13,; DBLP:conf/icsm/SahaLKP14,; DBLP:conf/iwpc/WangL14,; DBLP:conf/icsm/WongXZHZM14,), Dirichlet Language Model (DLM) (DBLP:journals/smr/SismanAK17,), or deep learning approaches (DBLP:journals/tse/HuoTLLS21,; DBLP:conf/icse/CiborowskaD22,; DBLP:journals/access/MohsenHWMM23,). In this work, we empirically evaluate IR-based FL in the Linux kernel.
- •
Agent-based FL. With the advance in LLMs, LLM agents have demonstrated remarkable accuracy in diverse software maintenance tasks, including FL. For instance, SWE-Agent (DBLP:journals/corr/abs-2405-15793,) incorporates a custom-built Agent-Computer Interface to navigate entire repositories and edit code files; AutoCodeRover (DBLP:conf/issta/0002RFR24,) equips LLMs with code search capabilities to retrieve relevant code contexts; Agentless (DBLP:journals/corr/abs-2407-01489,) refines the localization process with developer expertise, restricting the decision-making autonomy of agents.
In this work, we not only make the first attempt to empirically evaluate existing agents in the Linux kernel, but also propose an enhancement framework to improve agents in localizing Linux kernel bugs.

**Table 1: Existing Benchmarks for Software Maintenance**
| Benchmark | Language | # Repo | # Bugs | Data Source | Linux-Related | User-reported |
| --- | --- | --- | --- | --- | --- | --- |
| Defects4J (DBLP:conf/issta/JustJE14,) | Java | 17 | 854 | Bug Tracking Systems | ✗ | ✓ |
| Linux-3.16 (DBLP:conf/icsm/SahaLKP14,) | C | 1 | 1,548 | Bug Tracking Systems | ✓ | ✓ |
| SWE-bench (DBLP:conf/iclr/JimenezYWYPPN24,) | Python | 12 | 2,294 | GitHub Pull Requests | ✗ | ✓ |
| FAUN-Eval-fix (hu2024realworldbenchmarkevaluatingfinegrained,) | Multiple | 17 | 300 | GitHub Pull Requests/Issues | ✗ | ✓ |
| k-Bench-Syz (DBLP:journals/corr/abs-2407-02680,) | C | 113 | 279 | Fuzzing-Detected Crashes | ✓ | ✗ |
| Loc-Bench (DBLP:journals/corr/abs-2503-09089,) | Python | 165 | 560 | GitHub Issues | ✗ | ✓ |
| SWE-lancer (DBLP:journals/corr/abs-2502-12115,) | Python | 1 | 1,488 | Upwork Issues | ✗ | ✓ |
| LinuxFLBench | C | 120 | 250 | Bug Tracking Systems | ✓ | ✓ |

Benchmarks for Software Maintenance.
As FL is an essential sub-task in software maintenance, we revisit existing software maintenance benchmarks and summarize their key characteristics in Table [1](https://arxiv.org/html/2505.19489v1#S2.T1). The majority of existing benchmarks focus on general software systems in Java or Python. Different from these benchmarks, our benchmark LinuxFLBench specifically targets the larger-scale system Linux kernel. In particular, there are only two benchmarks (i.e., Linux-3.16 (DBLP:conf/icsm/SahaLKP14,) and k-bench-Syz (DBLP:journals/corr/abs-2407-02680,)) involving Linux kernel. While Linux-3.16 (DBLP:conf/icsm/SahaLKP14,) only includes bugs from one specific old version (i.e., Linux kernel 3.16), our benchmark spans a broader range of Linux kernel versions, offering a more comprehensive evaluation.
Particularly, the most relevant work to this paper is the recent benchmark kBenchSyz (DBLP:journals/corr/abs-2407-02680,), a Linux kernel crash resolution benchmark comprising Syzkaller (Syzkaller,) (i.e., fuzzing testing tool) detected crashes in the Linux kernel.
LinuxFLBench is different from kBenchSyz by providing more comprehensive and real-world user-reported Linux kernel bugs, including (1) different bug scopes: while kBenchSyz only includes crash bugs, LinuxFLBench includes a broader spectrum of real-world bugs beyond crash bugs (e.g., functionality bugs and performance bugs); (2) different bug sources: while kBenchSyz focuses exclusively on crashes automatically detected by the fuzzing tool Syzkaller, the bugs in LinuxFLBench are all real-world bugs reported by human users during their daily usage. Therefore, we believe that LinuxFLBench complements existing efforts in benchmarking FL techniques for maintaining Linux kernel. Moreover, while kBenchSyz mainly evaluates basic LLMs, our work evaluates LLM agents for Linux kernel for the first time.

## 3 LinuxFLBench : An FL Benchmark for Linux Kernel

LinuxFLBench is a new benchmark of 250 FL tasks derived from real-world Linux kernel bugs.

### 3.1 Construction of LinuxFLBench

The construction process of LinuxFLBench includes three phases, detailed in Appendix [A.1](https://arxiv.org/html/2505.19489v1#A1.SS1).

Step 1: Bug Report Collection.
We collected bug reports for Linux kernel from Kernel.org Bugzilla (Bugzilla,), the official platform for reporting Linux kernel bugs. Each bug report typically contains key information, including a title, a bug description, and additional relevant metadata, such as the kernel version and environment details. To ensure the availability of corresponding codebase, we considered only kernel versions accessible on the official Linux kernel website (Linuxwebsite,). For ground-truth reliability, we focused on bug reports resolved with confirmed code fixes, specifically those marked as “CLOSED” and “CODE_FIX” in the tracking system. Furthermore, we included only bug reports with patches available as attachments on the website, enabling us to identify the buggy locations based on the patch information. In total, we collected 2,138 bug reports during this step.

Step 2: Buggy Location Identification.
For each collected bug report, we identified the location modified in the developer-committed patch as the ground-truth buggy location. Specifically, we traversed source files with the extensions .c or .h, skipping other file types such as README or Makefile.
Following SWE-bench-lite (DBLP:conf/iclr/JimenezYWYPPN24,), we ensured the reliability and non-ambiguity of the ground truth by keeping bug reports involving exactly one buggy file.
After this step, 635 bug reports with identified buggy files were obtained.

Step 3: Manual Inspection.
To further ensure quality, we manually reviewed the collected data. Three human annotators checked each bug as follows: (1) bug reports without actual bugs (e.g., those that primarily submit patches) were excluded; (2) bug reports with sufficient information (e.g., clear natural language descriptions or detailed system logs) were retained; (3) bug reports that explicitly mentioned buggy locations or fix solutions were excluded.
As a result, the final dataset comprises 250 high-quality FL tasks, and each task includes a bug report, the buggy codebase, and the ground-truth buggy file and method(s). A detailed sample is shown in Appendix [A.2](https://arxiv.org/html/2505.19489v1#A1.SS2).

### 3.2 Characteristics of LinuxFLBench

LinuxFLBench offers multidimensional task diversity, spanning various Linux kernel versions and products, and each task involves a complex bug description and large-scale codebase.

Figure: Figure 1: Task Distribution across Products
Refer to caption: extracted/6477688/fig/bug_product_distribution.png

Scale.
Table [2](https://arxiv.org/html/2505.19489v1#S3.T2) compares the scale characteristics of the tasks in LinuxFLBench and SWE-bench. Notably, the Linux kernel codebase is significantly larger, containing tens of thousands of code files and millions of lines of code. Additionally, the bug descriptions in LinuxFLBench tend to be more complex and lengthy. As a result, localizing the buggy location within such a large-scale codebase is more challenging than in the relatively smaller projects of SWE-bench.

Products.
Fig. [1](https://arxiv.org/html/2505.19489v1#S3.F1) shows the distribution of LinuxFLBench across different kernel products (i.e., high-level components officially defined in Bugzilla). In particular, bugs in LinuxFLBench cover a diverse range of products (i.e., 16 different products) with Drivers, ACPI, and File System being the three largest categories.

Versions.
The Linux kernel has undergone continuous evolution over several decades, resulting in the release of numerous versions. LinuxFLBench captures this temporal diversity by including bugs from a broad range of kernel versions, covering a total of 120 distinct versions.

## 4 Evaluation of LLM agents on LinuxFLBench

We empirically evaluate SOTA LLM agents on LinuxFLBench to investigate their FL effectiveness in the Linux kernel.

### 4.1 Study Setup

Studied Baselines.
(1) LLM agents. We study three SOTA LLM agents, i.e., SWE-Agent (DBLP:journals/corr/abs-2405-15793,), AutoCodeRover (DBLP:conf/issta/0002RFR24,), and Agentless (DBLP:journals/corr/abs-2407-01489,), as they are fully open-sourced and achieve high effectiveness in recent software maintenance leaderboard (sweleaderboard,). All agents are equipped with GPT-4o (gpt-4o-2024-08-06) as backbone LLMs (gpt4o,). The detailed implementation of these agents is in Appendix [B](https://arxiv.org/html/2505.19489v1#A2).
(2) IR-based baselines.
To further investigate the effectiveness of agent-based methods, we also include traditional IR-based FL baselines for comparison. Specifically, we include the classic IR-based methods BugLocator (DBLP:conf/icse/ZhouZL12,) and BLUiR (DBLP:conf/kbse/SahaLKP13,), along with widely used IR techniques such as BM25 (DBLP:journals/ipm/RobertsonWH95,) and Sentence-BERT (DBLP:conf/emnlp/ReimersG19,).

Evaluation Metrics.
In line with previous FL work (10114452,; DBLP:conf/icse/ZhouZL12,; DBLP:conf/icsm/SahaLKP14,), we include the widely-used metrics like recall at top-k (k = 1, 5, 10) and the Mean Reciprocal Rank (MRR) to evaluate the FL effectiveness.

### 4.2 Quantitative Analysis

Table [3](https://arxiv.org/html/2505.19489v1#S4.T3) shows the overall file-level FL effectiveness of studied techniques on LinuxFLBench.

**Table 3: FL effectiveness on LinuxFLBench.**
| Methods | Recall@1 | Recall@5 | Recall@10 | MRR |
| --- | --- | --- | --- | --- |
| BM25 | 0.168 | 0.328 | 0.396 | 0.231 |
| BugLocator | 0.127 | 0.209 | 0.272 | 0.215 |
| BLUiR | 0.228 | 0.317 | 0.404 | 0.321 |
| Sentence-BERT | 0.056 | 0.136 | 0.180 | 0.090 |
| SWE-Agent | 0.416 | 0.552 | 0.584 | 0.476 |
| AutoCodeRover | 0.388 | 0.496 | 0.496 | 0.435 |
| Agentless | 0.368 | 0.492 | 0.504 | 0.419 |

Comparison with IR-based methods. Overall, existing agents outperform all traditional IR methods, indicating the benefits from agentic solutions in identifying buggy locations for large scale systems. For instance, SWE-Agent achieves the best effectiveness with an MRR of 0.476, significantly surpassing other methods. Among IR methods, BLUiR performs the best, but only with an MRR of 0.321.

Comparison with general software system.
Although outperforming traditional IR methods, existing agents still exhibit limited overall effectiveness on Linux kernel. For instance, even the best-performing SWE-Agent only achieves a top-1 recall of only 0.416 on LinuxFLBench, which is much lower than when it is applied to general software systems (i.e., SWE-bench). In particular, Fig.[3](https://arxiv.org/html/2505.19489v1#S4.F3) compares the FL effectiveness of agents in Linux systems (i.e., on LinuxFLBench) and in general software systems (i.e., on SWE-bench). The reported SWE-bench results are from previous work (DBLP:journals/corr/abs-2407-01489,).
We can observe a marked performance decline for all the LLM agents on LinuxFLBench compared to SWE-bench, with recall values decreasing by more than 0.15. Such an effectiveness drop underscores the heightened challenges associated with FL in the larger and more intricate Linux kernel codebase than general software systems.

Figure: Figure 2: Performance of LLM agents on SWE-bench and LinuxFLBench.
Refer to caption: extracted/6477688/fig/acc_compare.png

Uniqueness and Union.
Fig. [3](https://arxiv.org/html/2505.19489v1#S4.F3) presents the overlapped/unique bugs that are correctly localized at top-1 by studied agents. We could observe complementary strengths of the different approaches, as each agent can uniquely resolve 12 - 20 bugs. Nevertheless, even when combining the correctly-localized bugs of all agents, only 146 bugs out of 250 total bugs can be successfully localized (i.e., 58.4% top-1 recall). It further highlights the considerable challenges that agents still face in performing FL within the complex Linux kernel.

### 4.3 Qualitative Analysis

To further understand why agents perform poorly in Linux kernel, we manually examine bad cases where all studied agents fail to correctly localize the buggy files. Overall, we find two main reasons for the limited effectiveness as follows.

Confusion Among Related Files.
As a large-scale software system, bugs in Linux kernel often propagate along a long chain, where many related files are associated with each other via function calls or data dependencies.
While agents might be capable of coarse-grained FL (e.g., correctly identifying the buggy directories or high-level modules), they struggle to further precisely pinpoint the exact faulty file/method among all the related files. This challenge is indirectly evidenced by the fact that each Linux directory in LinuxFLBench contains, on average, approximately twice as many files (16 vs. 8) as those in SWE-bench, making fine-grained localization within directories more difficult.
For example, Appendix [C.1](https://arxiv.org/html/2505.19489v1#A3.SS1) shows a bad case where all agents wrongly localize the files that are in the same directory as the buggy file.

Limited Exploration of Potential Causes. Given the complexity of the Linux kernel, a bug can arise from diverse and non-obvious root causes. Current agents narrowly focus on a small set of highly probable causes, failing to explore a broader range of potential causes. Consequently, this limited exploration leads to missed opportunities for correctly identifying the buggy file.
Appendix [C.2](https://arxiv.org/html/2505.19489v1#A3.SS2) shows a bad case that all agents miss the real cause.

## 5 LinuxFL + : An Enhancing Framework

To address the limitations of existing agent-based methods, we propose a novel enhancing framework LinuxFL^+, which improves the FL effectiveness of agents in the Linux kernel.

### 5.1 Approach

As discussed in Section [4.3](https://arxiv.org/html/2505.19489v1#S4.SS3), given the huge space of Linux kernel, existing agents fail to capture the relationship between files or to cover a complete pool of potential causes. Therefore, the main insight of LinuxFL^+ is to expand the prediction results of existing agents with both the repository structure and the root causes.

Fig. [4](https://arxiv.org/html/2505.19489v1#S5.F4) shows the overall workflow of LinuxFL^+.
Given the buggy files predicted by any agent (e.g., AutoCodeRover), LinuxFL^+ refines the prediction via the following three phases.
(1) Directory-Aware Expansion: LinuxFL^+ expands the search scope within directories of the initial predictions generated by LLM agents. LinuxFL^+ then re-selects bug-related files within these directories, enabling a more thorough exploration of related files;
(2) Potential Cause Expansion: LinuxFL^+ explores as many potential causes as possible to scale the related files. LinuxFL^+ includes two hypothesizing strategies to expand the potential causes for the given bug report, leveraging both the original capabilities of LLMs (i.e., direct hypothesis) and the additional knowledge from Linux kernel mailing list (i.e., mail-augmented hypothesis);
(3) Candidate Integration: all relevant files are merged as candidates, followed by a re-ranking process to further refine the results.

Figure: Figure 4: Overview of LinuxFL^+.
Refer to caption: extracted/6477688/fig/method.png

#### 5.1.1 Directory-Aware Expansion

While existing agents can generally identify the correct modules related to a bug, they often struggle to distinguish relevant files within those modules. To address this limitation, LinuxFL^+ first expands the search scope to include all files in the directories of the initially predicted files. Using this expanded candidate set, the LLM re-selects files likely related to the bug. We retain the top-k (k=10) most relevant files as the expanded results. This approach provides the LLM with an additional opportunity to identify buggy files, enabling a more comprehensive exploration of related files. Detailed prompts are in Appendix [D.1](https://arxiv.org/html/2505.19489v1#A4.SS1).

#### 5.1.2 Potential Cause Expansion

Current agents tend to focus narrowly on a small set of highly probable causes within a limited number of steps. However, diagnosing complex bugs often requires an iterative “guess-and-check” process (DBLP:conf/uist/AlaboudiL23,; DBLP:conf/esem/LaymanDNSDV13,; DBLP:conf/eurosys/Liu0025,), where developers form experience-based hypotheses and progressively refine their understanding to isolate the root cause. Inspired by this process, we expand bug-related files by exploring a broader range of potential causes.

In this phase, we design two types of hypothesizing strategies to expand probable causes, namely Direct Hypothesis and Mail-Augmented Hypothesis. The former leverages the models’ inherent knowledge on Linux kernel bugs obtained from pre-training, while the latter incorporates historical bug knowledge from the developer mailing list discussions.

Direct Hypothesis. As LLMs possess a foundational understanding of the Linux kernel from extensive pre-training, a straightforward expansion approach is to fully leverage the intrinsic knowledge of models. To this end, we design prompts that instruct the model to generate as many plausible potential causes as possible, and rank these causes based on their estimated likelihood of being responsible for the bug. To ensure the practicality and relevance of each hypothesized cause, the LLM is also required to propose a corresponding fix and identify the specific files that would need modification. We then extract the predicted target files, preserving their original ranking from the associated causes.
Detailed prompts are in Appendix [D.2](https://arxiv.org/html/2505.19489v1#A4.SS2).

Mail-Augmented Hypothesis.
Relying solely on the intrinsic capabilities of LLMs is insufficient, as general-purpose models still lack in-depth and domain-specific knowledge of Linux kernel. To address this limitation, we incorporate historical bug knowledge from the Linux kernel mailing list (LKML) (LKML,). The LKML is the communication channel among Linux kernel developers, including massive emails discussing bugs, patches, and diverse topics on maintaining Linux kernel. Specifically, we adopt a Retrieval-Augmented Generation (RAG) approach, using mailing list data as an external knowledge base to provide more comprehensive and diverse bug causes in Linux kernel.

Mail Collection.
To construct the knowledge base, we first collect emails from the LKML channel.
To ensure the quality of the data, we retain only emails that include patches, as these are more likely to involve discussions of bug fixes or feature implementations, providing useful context for FL. However, we exclude patch emails that modify more than 10 files, as such patches are likely non-atomic, often representing merged changes. Additionally, to avoid potential data leakage, we exclude any emails containing external URLs or the keyword “bugzilla”.

Mail Retrieval.
To improve the efficiency of retrieval, we design a hierarchical retrieval strategy. First, we categorize all emails based on the files modified in the attached patches. Using the files predicted by agents, we restrict the search space to only emails associated with the predicted files. Next, since raw bug reports may contain noisy content (e.g., hexadecimal logs), we reformulate queries by summarizing the bug reports along four key dimensions: (1) bug behavior, (2) potential causes, (3) expected behavior, and (4) possible solutions.
We then apply the BM25 algorithm (bm25s,) to retrieve the top-k (k=10) most relevant mails. To ensure temporal consistency, we restrict the retrieval to emails sent before the bug report was reported.

With the retrieved mails, we further instruct LLMs to hypothesize more diverse and informed causes for the given bug report. The expanded causes can further guide LLMs to identify buggy files associated with the causes.
Detailed prompts are in Appendix [D.2](https://arxiv.org/html/2505.19489v1#A4.SS2).

#### 5.1.3 Candidate Integration

In this final phase, we consolidate the files predicted by previous two expansion strategies and rank the aggregated candidate files to produce the final FL results.

We adopt a simple yet effective merging strategy. Specifically, for each candidate file $f$, we collect its ranks from the three sources: $R_{dir}(f)$ (Directory-Aware Expansion), $R_{direct}(f)$ (Direct Hypothesis), and $R_{mail}(f)$ (Mail-Augmented Hypothesis). If a file does not appear in the results of a particular method, its rank is set to $\infty$. We then compute an aggregated score for $f$ as follows:

$$ $\text{score}(f)=1/{R_{dir}(f)}+1/{R_{direct}(f)}+1/{R_{mail}(f)}$ (2) $$

Files that achieve better ranks in any individual method receive higher scores, while those consistently ranked highly across methods are further prioritized. All candidate files are sorted by their aggregated scores to produce the initial merged ranking. To further refine this list, the LLM is prompted to re-rank the files based on the semantic correspondence between their path and bug report.

### 5.2 Experimental Setup

Baselines.
To evaluate the effectiveness of LinuxFL^+ in improving existing agents, we apply LinuxFL^+ to refine the prediction outputs of recent agents (i.e., SWE-Agent, AutoCodeRover, and Agentless) on LinuxFLBench.

Implementation Details.
We leverage GPT-4o (gpt4o,) (gpt-4o-2024-08-06) as the backbone model for implementing LinuxFL^+. We configure the model temperature as 0 to ensure relatively deterministic outputs with other parameters as default settings.

**Table 4: Evaluation results of LinuxFL^+ on LinuxFLBench.**
| Methods | Recall@1 | Recall@5 | Recall@10 | MRR | # Tokens | $ Cost |
| --- | --- | --- | --- | --- | --- | --- |
| SWE-Agent | 0.416 | 0.552 | 0.584 | 0.476 | 72.4 K | 0.194 |
| - w/ LinuxFL^+ | 0.524 (+0.108) | 0.720 (+0.168) | 0.768 (+0.184) | 0.610 (+0.134) | 14.0 K | 0.041 |
| AutoCodeRover | 0.388 | 0.496 | 0.496 | 0.435 | 206.6 K | 0.560 |
| - w/ LinuxFL^+ | 0.500 (+0.112) | 0.712 (+0.216) | 0.744 (+0.248) | 0.589 (+0.154) | 11.8 K | 0.035 |
| Agentless | 0.368 | 0.492 | 0.504 | 0.419 | 150.2 K | 0.396 |
| - w/ LinuxFL^+ | 0.440 (+0.072) | 0.684 (+0.192) | 0.724 (+0.220) | 0.549 (+0.130) | 15.3 K | 0.044 |

### 5.3 Results and Analysis

#### 5.3.1 Overall Performance

Table [4](https://arxiv.org/html/2505.19489v1#S5.T4) presents the improvements of LinuxFL^+ on all studied agents and the costs of LinuxFL^+.

Effectiveness.
LinuxFL^+ exhibits strong performance in enhancing the FL capabilities of agents, as evidenced by substantial improvement across all evaluation metrics. For example, when applied to SWE-Agent, Recall@10 increases from 0.584 to 0.768, an absolute gain of 18.4 percentage points. Moreover, Recall@1 improves by 10.8 percentage points (from 0.416 to 0.524). The improvement indicates the effectiveness of the candidate expansion strategies of LinuxFL^+, which successfully recover the buggy files missed by existing agents.

Generalizability.
Beyond improving individual methods, LinuxFL^+ consistently boosts performance across all SOTA agents. Notably, AutoCodeRover and Agentless, both of which initially exhibited lower performance in Recall@5 and Recall@10, achieve results comparable to SWE-Agent after incorporating LinuxFL^+. For instance, despite the large initial gaps in Recall@10 across methods, the integration of LinuxFL^+ consistently elevates their scores to surpass 0.7. These improvements underscore the generalizability of LinuxFL^+ and its applicability across agent-based approaches with varying baseline capabilities.

Ablation study.
We perform an ablation study to investigate the contribution of each component in LinuxFL^+. In particular, we find all the expansion strategies, i.e., directory-aware expansion and potential causes expansion (with either direct or mail-augmented hypothesis) can improve the FL effectiveness of agents. Detailed results can be found in Appendix [E](https://arxiv.org/html/2505.19489v1#A5).

Cost-efficiency.
While LinuxFL^+ delivers strong performance, it incurs only a modest additional cost. On average, the total number of tokens used per task by LinuxFL^+ ranges from 11.8k to 15.3k, resulting in an estimated cost of approximately $0.04. This is roughly one-tenth of the cost incurred by agent-based baselines. The primary cost of LinuxFL^+ stems from its use of email content. These results suggest that LinuxFL^+ can substantially enhance FL for the large-scale system Linux kernel at a affordable cost.

In summary, by enhancing the capabilities of existing agents, LinuxFL^+ facilitates more accurate FL with minimal costs. Our findings underscore the potential of LinuxFL^+ to significantly support software maintenance tasks in Linux kernel.

#### 5.3.2 Method-level FL

To further assess the effectiveness of LinuxFL^+ on fine-grained FL, we extend our evaluation to method-level FL.
In particular, we proceed to predict buggy methods among the buggy files predicted by LinuxFL^+.
Specifically, following prior work (DBLP:journals/corr/abs-2407-01489,), we provide LLMs with a skeleton representation of each file. This format retains only the signatures of functions and structures, along with their associated comments, thereby reducing input length while preserving essential contextual information. The LLM is then prompted to identify the top-k (k=10) most relevant code elements.

Given the characteristics of the C language, we define method-level elements to include functions, structures, and other code blocks. We consider the methods that are modified in the developer-committed patches as the ground truth for buggy methods.

**Table 5: Method-level FL results.**
| Methods | Recall@1 | Recall@5 | Recall@10 | MRR |
| --- | --- | --- | --- | --- |
| SWE-Agent | 0.089 | 0.178 | 0.214 | 0.170 |
| - w/ LinuxFL^+ | 0.138 | 0.271 | 0.326 | 0.253 |
| AutoCodeRover | 0.042 | 0.088 | 0.094 | 0.077 |
| - w/ LinuxFL^+ | 0.137 | 0.292 | 0.349 | 0.259 |
| Agentless | 0.098 | 0.147 | 0.179 | 0.162 |
| - w/ LinuxFL^+ | 0.111 | 0.229 | 0.269 | 0.217 |

Table [5](https://arxiv.org/html/2505.19489v1#S5.T5) presents the method-level FL results of existing agents and those enhanced with LinuxFL^+. Overall, LinuxFL^+ can consistently improve agents in method-level FL for Linux kernel. All three agent baselines exhibit low Recall@1 (below 0.1), while LinuxFL^+ consistently improves this metric beyond 0.1. The improvements are more pronounced in other metrics, e.g., for Recall@10, LinuxFL^+ enhances all baselines by more than 0.09. While localizing finer-grained elements is inherently much more challenging specifically for large scale systems like Linux kernel, the overall accuracy at method level remains relatively lower than at the file level, highlighting the need for further research in this direction.

## 6 Limitations

Limited Evaluation on Different LLMs.
To maintain consistency with prior work (DBLP:journals/corr/abs-2405-15793,; DBLP:conf/issta/0002RFR24,; DBLP:journals/corr/abs-2407-01489,) and enable fair comparison of agent performance on SWE-bench and our proposed LinuxFLBench, all experiments in this study employed the widely used GPT-4o as the backbone LLM. While LinuxFL^+ demonstrates notable improvements in enhancing agent performance with GPT-4o, we conducted limited investigation into its effectiveness when integrated with other LLMs.

Rough Usage of Mail Data.
LinuxFL^+ leverages external knowledge from Linux kernel mailing list to enhance FL. While we apply several heuristics to filter out noisy entries, the mail content may still contain irrelevant or outdated discussions. Although we mitigate this by discarding invalid predicted files, there remains room for improvement. Future work could explore more sophisticated approaches to effectively utilize mailing list knowledge for improved software maintenance tasks on the Linux kernel.

## 7 Conclusion

In this work, we introduce LinuxFLBench, a new and challenging software engineering benchmark designed for fault localization in the Linux kernel. To assess the effectiveness of existing LLM agents in complex software systems, we conduct an empirical study using LinuxFLBench. Initial results reveal that these agents struggle to accurately identify buggy files. To address this challenge, we propose LinuxFL^+, a fault localization enhancement framework that leverages diverse expansion strategies to enrich candidate selection. Our approach demonstrates substantial improvements in localization performance.

## References

- [1]
Kernel.org bugzilla.
https://bugzilla.kernel.org.
- [2]
Kernel.org rsync.
http://rsync.kernel.org/pub/linux/kernel.
- [3]
Linux kernel mailing list.
https://lore.kernel.org.
- [4]
Syzkaller.
https://syzkaller.appspot.com.
- [5]
Swe-bench leaderboards, 2025.
https://www.swebench.com/.
- [6]
R. Abreu, P. Zoeteweij, and A. J. C. van Gemund.
An evaluation of similarity coefficients for software fault
localization.
In 12th IEEE Pacific Rim International Symposium on Dependable
Computing (PRDC 2006), 18-20 December, 2006, University of California,
Riverside, USA, pages 39–46. IEEE Computer Society, 2006.
- [7]
A. Alaboudi and T. D. LaToza.
Hypothesizer: A hypothesis-based debugger to find and test
debugging hypotheses.
In S. Follmer, J. Han, J. Steimle, and N. H. Riche, editors, Proceedings of the 36th Annual ACM Symposium on User Interface Software and
Technology, UIST 2023, San Francisco, CA, USA, 29 October 2023- 1 November
2023, pages 73:1–73:14. ACM, 2023.
- [8]
T. F. Bissyandé, L. Réveillère, J. Lawall, and G. Muller.
Diagnosys: automatic generation of a debugging interface to the linux
kernel.
In M. Goedicke, T. Menzies, and M. Saeki, editors, IEEE/ACM
International Conference on Automated Software Engineering, ASE’12, Essen,
Germany, September 3-7, 2012, pages 60–69. ACM, 2012.
- [9]
M. Bligh and A. P. Whitcroft.
Fully automated testing of the linux kernel.
In Proceedings of the Linux Symposium, volume 1, pages
113–125. Citeseer, 2006.
- [10]
Y. Chen, F. Wu, K. Yu, L. Zhang, Y. Chen, Y. Yang, and J. Mao.
Instant bug testing service for linux kernel.
In 10th IEEE International Conference on High Performance
Computing and Communications & 2013 IEEE International Conference on
Embedded and Ubiquitous Computing, HPCC/EUC 2013, Zhangjiajie, China,
November 13-15, 2013, pages 1860–1865. IEEE, 2013.
- [11]
Z. Chen, X. Tang, G. Deng, F. Wu, J. Wu, Z. Jiang, V. K. Prasanna, A. Cohan,
and X. Wang.
Locagent: Graph-guided LLM agents for code localization.
CoRR, abs/2503.09089, 2025.
- [12]
A. Ciborowska and K. Damevski.
Fast changeset-based bug localization with BERT.
In 44th IEEE/ACM 44th International Conference on Software
Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pages
946–957. ACM, 2022.
- [13]
J. Edge.
A kernel debugger in python: drgn, 2019.
https://lwn.net/Articles/789641/.
- [14]
R. Hu, C. Peng, J. Ren, B. Jiang, X. Meng, Q. Wu, P. Gao, X. Wang, and C. Gao.
A real-world benchmark for evaluating fine-grained issue solving
capabilities of large language models, 2024.
- [15]
X. Huo, F. Thung, M. Li, D. Lo, and S. Shi.
Deep transfer bug localization.
IEEE Trans. Software Eng., 47(7):1368–1380, 2021.
- [16]
D. R. Jeong, M. Jung, Y. Lee, B. Lee, I. Shin, and Y. Kwon.
Diagnosing kernel concurrency failures with AITIA.
In G. A. D. Luna, L. Querzoni, A. Fedorova, and D. Narayanan,
editors, Proceedings of the Eighteenth European Conference on Computer
Systems, EuroSys 2023, Rome, Italy, May 8-12, 2023, pages 94–110. ACM,
2023.
- [17]
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R.
Narasimhan.
Swe-bench: Can language models resolve real-world github issues?
In The Twelfth International Conference on Learning
Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
OpenReview.net, 2024.
- [18]
R. Just, D. Jalali, and M. D. Ernst.
Defects4j: a database of existing faults to enable controlled testing
studies for java programs.
In C. S. Pasareanu and D. Marinov, editors, International
Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA
- July 21 - 26, 2014, pages 437–440. ACM, 2014.
- [19]
S. Kang, G. An, and S. Yoo.
A quantitative and qualitative evaluation of llm-based explainable
fault localization.
Proc. ACM Softw. Eng., 1(FSE):1424–1446, 2024.
- [20]
L. Layman, M. Diep, M. Nagappan, J. Singer, R. DeLine, and G. Venolia.
Debugging revisited: Toward understanding the debugging needs of
contemporary software developers.
In 2013 ACM / IEEE International Symposium on Empirical
Software Engineering and Measurement, Baltimore, Maryland, USA, October
10-11, 2013, pages 383–392. IEEE Computer Society, 2013.
- [21]
H. Liu, Y. Jiang, and C. Xu.
Understanding the linux kernel, visually.
In Proceedings of the Twentieth European Conference on Computer
Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April
2025, pages 1044–1060. ACM, 2025.
- [22]
J. Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou.
Large language model-based agents for software engineering: A
survey.
CoRR, abs/2409.02977, 2024.
- [23]
Y. Lou, Q. Zhu, J. Dong, X. Li, Z. Sun, D. Hao, L. Zhang, and L. Zhang.
Boosting coverage-based fault localization via graph-based
representation learning.
In D. Spinellis, G. Gousios, M. Chechik, and M. D. Penta, editors,
ESEC/FSE ’21: 29th ACM Joint European Software Engineering
Conference and Symposium on the Foundations of Software Engineering, Athens,
Greece, August 23-28, 2021, pages 664–676. ACM, 2021.
- [24]
X. H. Lù.
Bm25s: Orders of magnitude faster lexical search via eager sparse
scoring, 2024.
- [25]
A. Mathai, C. Huang, P. Maniatis, A. Nogikh, F. Ivancic, J. Yang, and B. Ray.
Kgym: A platform and dataset to benchmark large language models on
linux kernel crash resolution.
CoRR, abs/2407.02680, 2024.
- [26]
S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke.
Swe-lancer: Can frontier llms earn $1 million from
real-world freelance software engineering?
CoRR, abs/2502.12115, 2025.
- [27]
A. M. Mohsen, H. A. Hassan, K. Wassif, R. Moawad, and S. Makady.
Enhancing bug localization using phase-based approach.
IEEE Access, 11:35901–35913, 2023.
- [28]
OpenAI.
Hello gpt-4o, 2024.
- [29]
Y. Qin, S. Wang, Y. Lou, J. Dong, K. Wang, X. Li, and X. Mao.
Agentfl: Scaling llm-based fault localization to project-level
context.
CoRR, abs/2403.16362, 2024.
- [30]
N. Reimers and I. Gurevych.
Sentence-bert: Sentence embeddings using siamese bert-networks.
In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of
the 2019 Conference on Empirical Methods in Natural Language Processing and
the 9th International Joint Conference on Natural Language Processing,
EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980–3990.
Association for Computational Linguistics, 2019.
- [31]
S. E. Robertson, S. Walker, and M. Hancock-Beaulieu.
Large test collection experiments on an operational, interactive
system: Okapi at TREC.
Inf. Process. Manag., 31(3):345–360, 1995.
- [32]
R. K. Saha, J. Lawall, S. Khurshid, and D. E. Perry.
On the effectiveness of information retrieval based bug localization
for C programs.
In 30th IEEE International Conference on Software Maintenance
and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014, pages
161–170. IEEE Computer Society, 2014.
- [33]
R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry.
Improving bug localization using structured information retrieval.
In E. Denney, T. Bultan, and A. Zeller, editors, 2013 28th
IEEE/ACM International Conference on Automated Software Engineering, ASE
2013, Silicon Valley, CA, USA, November 11-15, 2013, pages 345–355. IEEE,
2013.
- [34]
L. Serrano, V. Nguyen, F. Thung, L. Jiang, D. Lo, J. Lawall, and G. Muller.
SPINFER: inferring semantic patches for the linux kernel.
In A. Gavrilovska and E. Zadok, editors, Proceedings of the 2020
USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020,
pages 235–248. USENIX Association, 2020.
- [35]
B. Sisman, S. A. Akbar, and A. C. Kak.
Exploiting spatial code proximity and order for improved source code
retrieval for bug localization.
J. Softw. Evol. Process., 29(1), 2017.
- [36]
The Linux Foundation.
Linux kernel history report 2020, 2020.
- [37]
S. Wang and D. Lo.
Version history, similar report, and structure: putting them together
for improved bug localization.
In C. K. Roy, A. Begel, and L. Moonen, editors, 22nd
International Conference on Program Comprehension, ICPC 2014, Hyderabad,
India, June 2-3, 2014, pages 53–63. ACM, 2014.
- [38]
C. Wong, Y. Xiong, H. Zhang, D. Hao, L. Zhang, and H. Mei.
Boosting bug-report-oriented fault localization with segmentation and
stack-trace analysis.
In 30th IEEE International Conference on Software Maintenance
and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014, pages
181–190. IEEE Computer Society, 2014.
- [39]
W. E. Wong, V. Debroy, R. Gao, and Y. Li.
The dstar method for effective software fault localization.
IEEE Trans. Reliab., 63(1):290–308, 2014.
- [40]
C. S. Xia, Y. Deng, S. Dunn, and L. Zhang.
Agentless: Demystifying llm-based software engineering agents.
CoRR, abs/2407.01489, 2024.
- [41]
X. Xia and D. Lo.
Information Retrieval-Based Techniques for Software Fault
Localization, pages 365–391.
2023.
- [42]
C. Yang, Z. Zhao, and L. Zhang.
Kernelgpt: Enhanced kernel fuzzing via large language models.
In L. Eeckhout, G. Smaragdakis, K. Liang, A. Sampson, M. A. Kim, and
C. J. Rossbach, editors, Proceedings of the 30th ACM International
Conference on Architectural Support for Programming Languages and Operating
Systems, Volume 2, ASPLOS 2025, Rotterdam, Netherlands, 30 March 2025 - 3
April 2025, pages 560–573. ACM, 2025.
- [43]
J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and
O. Press.
Swe-agent: Agent-computer interfaces enable automated software
engineering.
CoRR, abs/2405.15793, 2024.
- [44]
Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury.
Autocoderover: Autonomous program improvement.
In M. Christakis and M. Pradel, editors, Proceedings of the 33rd
ACM SIGSOFT International Symposium on Software Testing and Analysis,
ISSTA 2024, Vienna, Austria, September 16-20, 2024, pages 1592–1604.
ACM, 2024.
- [45]
J. Zhou, H. Zhang, and D. Lo.
Where should the bugs be fixed? more accurate information
retrieval-based bug localization based on bug reports.
In M. Glinz, G. C. Murphy, and M. Pezzè, editors, 34th
International Conference on Software Engineering, ICSE 2012, June 2-9,
2012, Zurich, Switzerland, pages 14–24. IEEE Computer Society, 2012.

## Appendix A Additional Details of LinuxFLBench

### A.1 Construction Pipeline of LinuxFLBench

Figure: Figure 5: Construction pipeline of LinuxFLBench.
Refer to caption: extracted/6477688/fig/dataset_construction.png

Fig. [5](https://arxiv.org/html/2505.19489v1#A1.F5) illustrates the pipeline for constructing LinuxFLBench, which consists of three main phases: Bug Report Collection, Buggy Location Identification, and Manual Inspection. Through this process, we curate a total of 250 high-quality fault localization tasks.

### A.2 A Sample Kernel Bug from LinuxFLBench

We collect Linux kernel bugs (as shown in Fig. [6](https://arxiv.org/html/2505.19489v1#A1.F6)) from the reported and fixed bugs on Kernel.org Bugzilla. For each bug, the key information includes:

- 1.
Title: The summary title of the bug report.
- 2.
Description: A human-written description of the bug, which may include various types of information such as observed buggy behavior, reproduction steps in natural language, system logs, or call traces.
- 3.
Product: The product category to which the bug is assigned.
- 4.
Component: The specific component within the product affected by the bug.
- 5.
Hardware: The hardware configuration on which the bug was observed.
- 6.
Kernel Version: The version of the Linux kernel in which the bug occurred (e.g., 5.6.7).
- 7.
Paths: The paths of the buggy files, extracted from the golden patch that fixes the bug.
- 8.
Modified Functions: The method-level code elements modified by the patch.

Figure: Figure 6: A sample kernel bug from LinuxFLBench.
Refer to caption: extracted/6477688/fig/bug_sample.png

## Appendix B Details of Baselines Used in This Paper

### B.1 Studied LLM agents.

This paper evaluates three SOTA LLM agents: SWE-Agent [43], AutoCodeRover [44], and Agentless [40].

- •
SWE-Agent. SWE-Agent navigates the entire repository to identify the bug’s location.
To adapt this system to our benchmark, we modified the task description in the system prompt, specifying the objective as identifying suspicious files, while keeping the rest of the framework unchanged.
- •
AutoCodeRover. AutoCodeRover locates suspicious Python files based on the give GitHub issues through advanced code search techniques. We extended its functionality to support C/C++ projects, enabling it to perform code search within Linux kernel codebases.
- •
Agentless. Agentless identifies the suspicious files based on a concise representation of the repository structure. Given the vast number of files in the Linux kernel, we partition the repository structure into manageable portions by folder and feed them to the LLM in multiple iterations.

### B.2 IR-based Baselines.

To further investigate the effectiveness of agent-based methods, we also selected traditional IR-based baselines for comparison. Specifically, we included the classic methods BugLocator [45] and BLUiR [33], along with widely used IR techniques such as BM25 [31] and Sentence-BERT [30].

- •
BM25. BM25 is one of the most widely used IR methods, and we include it as one of our baselines. BM25 is a bag-of-words retrieval function that ranks a set of documents based on term frequency and inverse document frequency of each document.
- •
BugLocator. BugLocator retrieves buggy files from a codebase by treating a bug report as a query and ranking files based on similarity using a revised vector space model (rVSM). The rVSM method prioritizes longer documents, assuming these files are more likely to contain bugs. Additionally, BugLocator incorporates historical bug fixes to further assess the likelihood of defects in a given file. In this work, we do not leverage this historical bug fix feature of BugLocator due to the unavailability of the necessary data.
- •
BLUiR. BLUiR enhances bug localization by extracting code entities, such as classes, methods, and variable names, from source code files. It calculates the relevance of these entities to both the title and description of a bug report respectively, aiding in the identification of buggy files.
- •
SentenceBERT. SentenceBERT enhances the traditional BERT model by incorporating siamese and triplet network architectures, enabling more efficient semantic search with reduced computational overhead. For our implementation, we utilize the sentence-transformer model all-MiniLM-L6-v2.

## Appendix C Failure Cases of LLM agents on LinuxFLBench

The suboptimal performance of agent-based methods can be attributed to several limitations, including confusion among related files and insufficient exploration of potential root causes. This section presents representative failure cases to illustrate these limitations.

### C.1 Confusion Among Related Files

An illustrative case is shown in Fig. [7](https://arxiv.org/html/2505.19489v1#A3.F7). In this example, the update of the computer’s battery and AC status involves interactions among the ACPI battery, AC adapter, and the embedded controller (EC). The corresponding drivers for these components all reside in the drivers/acpi directory. While different agent baselines identify the files related to the ACPI battery and adapter, they confuse and overlook the deeper component in the bug propagation chain—the EC driver—resulting in incorrect FL.

Figure: Figure 7: An illustrative case for "Confusion Among Related Files".
Refer to caption: extracted/6477688/fig/bug_sample_for_dir.png

### C.2 Limited Exploration of Potential Causes

A representative case is provided in Fig. [8](https://arxiv.org/html/2505.19489v1#A3.F8). the bug behavior “hangs on shutdown” could stem from various causes, since system shutdown involves a sequence of operations across multiple components. Agents with limited exploration may employ a "depth-first search"-like strategy, focusing on superficially obvious reasons—such as failures in general power-off routines—while overlooking less apparent causes rooted in hardware state handling.

Figure: Figure 8: An illustrative case for "Limited Exploration of Potential Causes".
Refer to caption: extracted/6477688/fig/bug_sample_for_reason.png

## Appendix D Prompt Design of LinuxFL +

### D.1 Prompt Templates in Directory-Aware Expansion

LinuxFL^+ re-selects related files within the same directories as the originally predicted files. Given the bug report(”bug information”) and the list of files(”candidate files”) in these directories, the LLM is instructed to select the relevant files using the following prompt.

[linecolor=myblue!50,linewidth=2pt,roundcorner=10pt,backgroundcolor=gray!20]
Prompt for Directory-Aware Expansion:
Please look through the following Linux kernel bug report and candidate files, and select a list of files that one would need to edit to fix the bug.

Here is the information about the bug:

### Linux kernel bug report ###

{bug information}

###

Based on the bug provided above, I will present a list of candidate files that may be relevant to the bug.

### Candidate files ###

{candidate files}

###

Please select files that are most likely to need modification to fix this bug.

Your response should be in the format of a list of file paths, and should be ordered by relevance in descending order.
Please return at most 10 files.

### output example ###

[’net/ipv6/proc.c’, ’net/ipv6/netfilter/ip6_tables.c’]

###

Please format your response strictly according to the format provided above without commentary.

### D.2 Prompt Templates in Potential Cause Expansion

In the phase of Potential Cause Expansion, LinuxFL^+ instructs the LLM to enumerate as many potential causes as possible using two approaches: direct hypothesis and mail-augmented hypothesis. The prompt for Mail-Augmented Hypothesis is presented below. Given the bug report (”bug information”) and the retrieved emails (”mail content), the LLM is prompted to generate potential causes along with corresponding fix suggestions and the affected code files in a specified JSON format. The prompt for Direct Hypothesis is similar, but without including the retrieved email content.

[linecolor=myblue!50,linewidth=2pt,roundcorner=10pt,backgroundcolor=gray!20]
Prompt for Mail-Augmented Hypothesis:
Please review the following Linux kernel bug report, and then deduce the possible causes of the bug and provide corresponding code files and a potential fix. The bug is known to be related to the kernel code, and the fix should involve modifications to kernel code files.

Here is the information about the bug:

### Linux kernel bug report ###

{bug information}

###

To assist in your analysis, here are some emails retrieved using BM25 that may be relevant to the bug. Use them to inspire and identify additional possible causes:

### Mails ###

{mail content}

###

Based on the bug provided above, please output the possible causes, relevant code files, and solutions. Your response should follow the format below.

### Output example ###

[
{
’cause’: ’A description of the potential cause of the bug.’,
’code_file’: ’Path of the code file that is most likely related to the bug.’,
’fix_solution’: ’A short description of the fix solution to apply in the code file.’
},
…
]

###

Please ensure the following:

- List as many causes as possible, ordered by relevance in descending order, with the most likely cause first.

- For each cause, list all relevant code files and their corresponding fixes, but only provide one code file and one fix per entry.

- The relevant code file is not necessarily the one causing the bug but should be a file where the bug can be fixed.

- The code file should be in the format of "net/ipv6/proc.c".

- Format your response strictly according to the format provided above without commentary.

## Appendix E Ablation study

Figure: Figure 9: Venn Diagram for Correctly Localized Bugs by Agentless with Different Strategies.
Refer to caption: extracted/6477688/fig/agentless_scaling_venn_diagram.png

Table [6](https://arxiv.org/html/2505.19489v1#A5.T6) presents the results of integrating individual components of LinuxFL^+ into the baselines, examining how each localization phase contributes to the final performance.

Complementarity of Scaling Strategies.
As shown in Table [6](https://arxiv.org/html/2505.19489v1#A5.T6), agent baselines augmented with the three scaling strategies—Directory-Aware Expansion, Direct Hypothesis, and Mail-Augmented Hypothesis—exhibit varying FL performance on LinuxFLBench. Merging the results from these strategies leads to improved performance, suggesting their complementary nature. To further investigate this characteristic, we present a Venn diagram in Fig. [9](https://arxiv.org/html/2505.19489v1#A5.F9), illustrating the top-1 successfully localized bugs achieved by each strategy beyond Agentless. Each strategy independently identifies a substantial number of bugs that the others fail to locate. This highlights the rationale behind our merging approach. The three strategies emphasize different aspects: directory-level structural information from the codebase, intrinsic knowledge from the LLM, and external expertise from historical mailing lists. Integrating these perspectives allows for more effective and robust fault localization.

Impact of Re-Ranking.
LinuxFL^+ performs a re-ranking step on the candidates obtained from previous phase. The results in Table [6](https://arxiv.org/html/2505.19489v1#A5.T6) demonstrate the effectiveness of this re-ranking process. By comparing the results after merging with the final performance of LinuxFL^+, further improvement in localization accuracy could be observed. It may stem from the high-quality candidate files provided by the different expansion strategies. With these enriched candidates, even a simple re-ranking allows the model to more easily identify the correct buggy files.

Benefit of Mail Knowledge.
To investigate the benefits of incorporating knowledge from the LKML, we compare the results of baseline methods integrated with different Potential Cause Expansion strategies: Direct Hypothesis and Mail-Augmented Hypothesis. As shown in the Table [6](https://arxiv.org/html/2505.19489v1#A5.T6), Mail-Augmented Hypothesis consistently outperforms Direct Hypothesis. The latter relies solely on the intrinsic knowledge of LLMs, without utilizing predictions from agent methods, and achieves a recall@1 of only 0.316. In contrast, with the assistance of mail knowledge, Mail-Augmented Hypothesis achieves a recall@1 as high as 0.488, with even more significant improvements observed in recall@10.

These results suggest that the additional knowledge from LKML helps LLMs explore potential causes more comprehensively, leading to the identification of more relevant code files. This demonstrates that mailing list data can effectively bridge the knowledge gap LLMs face in localizing bugs within the Linux kernel. It is worth noting that the effectiveness of Mail-Augmented Hypothesis varies across different agent methods. For instance, in the case of SWE-Agent, the predicted files facilitate the retrieval of more relevant emails, which provide stronger guidance during cause exploration.

## Appendix F Human Participation

In this work, human involvement is limited to the Manual Inspection step during the construction of our benchmark, LinuxFLBench. This task was approved by the Institutional Review Board (IRB) at our institution. All participants were compensated at a rate of $15 per hour.

During Manual Inspection, each annotator was provided with the following instruction:
“Given the title and description of the bug report, please label the report as ‘yes,’ ‘no,’ or ‘unsure’ for each of the following three questions: (1) Does the report describe an actual bug (e.g., not merely submitting a patch)? (2) Does the report contain sufficient information, such as clear natural language descriptions of the buggy behavior, reproduction steps, or detailed system logs? (3) Does the report avoid including solutions, such as identifying the buggy location or attaching patches? If unsure, please select the label ‘unsure.’”
A report was assigned a final label of “yes” only if all three questions received a “yes” from an annotator. Each bug report was independently labeled by three participants. Reports that received at least two “yes” labels across annotators were retained in the final dataset.

## Appendix G Experiment Statistical Significance

**Table 7: Experiment Statistical Significance of LinuxFL^+**
| Method | Enhanced (Mean ± Std) | Original (Mean ± Std) | Mean Diff | t-stat | p-value | CI (Enhanced) | CI (Original) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Agentless | 0.549 ± 0.431 | 0.419 ± 0.463 | 0.129 | 6.126 | 0.000 | [0.493, 0.600] | [0.361, 0.471] |
| AutoCodeRover | 0.589 ± 0.437 | 0.435 ± 0.469 | 0.154 | 5.825 | 0.000 | [0.537, 0.643] | [0.374, 0.493] |
| SWE-Agent | 0.610 ± 0.433 | 0.476 ± 0.463 | 0.134 | 5.679 | 0.000 | [0.561, 0.663] | [0.416, 0.533] |

To evaluate the effectiveness of the proposed LinuxFL^+, we performed statistical significance tests comparing the MRR scores of LLM agents enhanced with LinuxFL^+ to those of their original counterparts. As presented in Table [7](https://arxiv.org/html/2505.19489v1#A7.T7), all enhancements introduced by LinuxFL^+ yield statistically significant improvements, with paired t-tests producing p-values below 0.0001. estimated using 1,000 resamples. Importantly, the confidence intervals for the enhanced models do not overlap with those of the original models, providing additional evidence for the significance of the observed improvements. These consistent and statistically significant gains across multiple LLM agents underscore the robustness and effectiveness of our FL-enhancing framework.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Q1: Could you clarify what key conceptual or methodological innovation distinguishes LinuxFL+ from standard RAG or search expansion frameworks?

Q2: LinuxFLBench includes only 250 single-file bugs, which may limit its realism and generalizability. How do you plan to address the limited dataset scale and the exclusion of multi-file or cross-module bugs?

Q3: Can you provide a more systematic analysis of the types of failures or reasoning gaps observed in LLM agents, and evidence or procedures confirming that LinuxFLBench and the LKML data are not contaminated with model pretraining data?
soundness: 1
strengths: - The paper tackles an important and underexplored domain, namely fault localization in large-scale, low-level software like the Linux kernel.

- Provides a well-constructed benchmark (LinuxFLBench) with real-world bug reports and ground-truth patches.

- The enhancement framework (LinuxFL+) offers practical, empirically validated gains with low cost and good reproducibility.
summary: This paper introduces LinuxFLBench, a new benchmark for evaluating fault localization (FL) in the Linux kernel, and proposes LinuxFL+, an enhancement framework designed to improve existing LLM-based agents’ performance on this challenging domain.
The authors benchmark several leading LLM agents (SWE-Agent, AutoCodeRover, Agentless) and find that they perform significantly worse on Linux kernel bugs than on existing datasets like SWE-bench.
weaknesses: 1. **Limited methodological novelty in LinuxFL+**
Although LinuxFL+ improves empirical results, the method primarily combines straightforward retrieval expansion and RAG-style augmentation.
The two strategies( i.e., directory-aware expansion and potential-cause expansion) are intuitive extensions rather than novel algorithmic contributions.
No clear theoretical insight or principled reasoning is provided for why these expansions work or how they might generalize beyond this specific setup.

2. **Benchmark coverage and representativeness remain narrow**
LinuxFLBench contains only 250 bugs despite the vast number of real-world Linux kernel issues.
Moreover, the dataset filters only “single-file fix” bugs, which simplifies the problem but undermines realism.
As a result, the benchmark might not capture the complexity of multi-file interactions and cross-component dependencies typical of real kernel debugging.

3. **Insufficient Evaluation analysis**
The evaluation remains superficial, reporting results without diagnostic analysis of why LLM agents fail or how the solution addresses specific reasoning errors. The failure analysis in Section 4.3 is anecdotal, lacking a systematic categorization of error types. Furthermore, the reasons behind the limited method-level improvements

4. **Missing discussion on contamination and data leakage**
Since LinuxFLBench is built from public Bugzilla and mailing list data, and the models (especially GPT-4o) are trained on internet-scale corpora, there is a non-trivial risk of data contamination.
The paper does not describe any checks for overlapping content between pretraining data and evaluation tasks.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How did the authors modify the prior agents to perform fault localization tasks instead of solving software issues
2. Please comment on if LinuxFL+ can be utilized for effective fault localization in other software systems. For example, have the authors applied similar approach for fault localization on the SWE-bench problems?
3. Using the Linux mailing list could lead to some data leakage (even excluding Bugzilla), as the developer may discuss patches that are very relevant for the bug you want to localize, how did the authors address this issue?
4. Can the authors also comment on if the constructed benchmark can be utilized for repair evaluation?
soundness: 3
strengths: - deals with a very important problem of fault localization in large software systems such as the Linux kernel
- construct a benchmark for this specific task that can be used by future research projects
- benchmark includes manual annotation which ensures high quality
summary: The paper tackles the problem of fault localization for large scale software systems like the linux kernel. The authors first propose a well-defined FL benchmark created from bug reports/patches in the Linux Kernel. The authors compare baseline agent-based/and other prior software issue solving techniques and demonstrate significant drop in fault localization on the benchmark. Furthermore the authors propose LinuxFL to improve localization performance specific on the linux kernel through a combination of reranking and retrievel using linux mailing list.
weaknesses: - benchmark ignores more complex and difficult bugs/fault localization tasks:
	- The authors claims that "we kept only unambiguous cases where exactly one file was modified to ensured the reliability of the ground truth". This would ignore a lot of multi-file bugs which are extremely important to evaluate 
- baseline setups:
	- The paper compares against some prior baselines used for solving software development issues like the ones in SWE-bench. 
	- however, the task in this work is different (fault localization), its unclear how the authors modify an approach (e.g., SWE-agent) for a fault localization task?
- proposed approach are aimed specifically for Linux kernels and may not generalize to other large systems:
	- building on my prior point regarding the baseline setup, the prior tools work well as they can be utilized for other repos as well as different software engineering tasks (e.g., repair, feature implementation) 
	- On the other hand, the proposed LinuxFL+ is extremely tailored for Linux kernels with the authors even using Linux mailing lists

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. **Leakage auditing**: Can you provide a formal audit for mail-augmentation (time windows, near-duplicate detection, blinded assessment that emails don’t directly reveal locations)?
2. **Single-file constraint**: What fraction of reports were excluded due to multi-file patches? How do agents and LinuxFL+ perform on a multi-file track?
3. **Sensitivity analyses**: Report sensitivity to directory expansion k, retrieval top-k, and re-ranking. Are gains robust across settings?
4. **Non-LLM re-ranking baselines**: How does LLM re-ranking compare to strong lexical/sparse/dense rankers over file path + report?
5. **Backbone diversity**: Beyond GPT-4o and Qwen3-32B, how do other open-source models (7B–72B) fare? Any scaling trend?
6. **Method-level bottlenecks**: Can you decompose failure into (i) file recall vs. (ii) method ranking? Would richer skeletons help?
7. **Generalization**: Any early results on other kernels/large C++ codebases? Will you release scripts for reproducing analogous benchmarks?
8. **Cost/latency**: Per-task wall-clock times with/without LinuxFL+ and guidance on batching/parallelization?
soundness: 3
strengths: - **Originality**: New, domain-specific FL benchmark for Linux with real user-reported bugs at larger scale than SWE-bench(-lite). Novel and important observation of degraded agent performance on the kernel.
- **Quality**: Careful curation; strong baseline coverage (IR and agents); robust gains from LinuxFL+ across agents/backbones; statistical significance analysis; cost accounting shows practical overhead (~$0.04/task).
- **Clarity**: Clear motivation and high-level method; results/ablations are informative; limitations stated.
- **Significance**: Highlights challenges of applying LLM agents to large, low-observability, multi-factor systems. Likely practical impact for practitioners evaluating Linux bugs.
summary: This paper introduces LinuxFLBench, a benchmark of 250 real-world Linux kernel fault-localization (FL) tasks constructed from user-reported bugs (Kernel.org Bugzilla), spanning 120 kernel versions and 66 components. Each task provides a bug report, the buggy codebase, and ground-truth buggy file/method(s) derived from patches. The paper evaluates state-of-the-art LLM agents (SWE-Agent, AutoCodeRover, Agentless) and classical IR baselines (BM25, BugLocator, BLUiR, Sentence-BERT). Results show that LLM agents outperform classical IR methods but still struggle on the Linux kernel compared to general software benchmarks like SWE-bench (e.g., best top-1 file-level recall is 41.6% vs. ~70% on SWE-bench). To address these gaps, the authors propose LinuxFL+, a post-hoc enhancement framework with (i) directory-aware expansion, (ii) potential-cause expansion via LLM hypothesis (direct and mail-augmented using LKML), and (iii) candidate integration with re-ranking. LinuxFL+ improves all agents with modest token cost (e.g., SWE-Agent Recall@10: 0.584 → 0.768; MRR: 0.476 → 0.610), and provides consistent gains even for a smaller open-source backbone (Qwen3-32B). Method-level FL remains challenging (Recall@1 < 0.14 even with LinuxFL+).
weaknesses: - **Benchmark bias via single-file constraint**: Excluding multi-file fixes may skew composition and agent failure modes. Please quantify exclusions, characterize them, and consider a multi-file track.
- **Leakage/provenance in mail-augmentation**: Despite pre-report restriction and filters, subtle leakage is possible. A formal audit (time-window sensitivity, near-duplicate detection, blinded checks) and a “clean” subset would strengthen claims.
- **Heuristic under-specification**: Directory-aware expansion (choice of k), merging and LLM re-ranking are lightly formalized. Compare against strong non-LLM re-rankers (BM25/SPLADE/ColBERT on path+report) and provide sensitivity studies.
- **Limited LLM/agent diversity**: Most results rely on GPT-4o with one open-source model. Explore more open-source backbones and alternative agent frameworks/policies to bolster generality.
- **Method-level FL still weak**: Although improved, recalls remain low. Decompose errors into file recall vs. intra-file ranking; explore alternative skeletonization (call-graph stubs, macro expansion hints).
- **Scope/portability**: C/Linux focus. Discuss portability to other large C/C++ systems and consider releasing scripts to replicate the construction elsewhere.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 4
questions: - When evaluating these agents, did the authors provide them with an execution environment?  
- Were the agents’ tool invocation or command execution abilities (e.g., writing and running search scripts during iteration) considered in the experiments?
soundness: 3
strengths: - The paper is well-written and clearly presented.
- The proposed benchmark introduces a higher level of difficulty compared to SWE-Bench.
- This benchmark focuses on user-reported bugs rather than fuzzing-based ones, which makes the dataset more realistic.
- The experimental evaluation is fairly comprehensive, covering most mainstream fault localization approaches.
summary: This paper introduces LinuxFLBench, a fault localization benchmark for Linux that includes 250 manually verified cases. Compared with SWE-Bench, LinuxFLBench presents more challenging Linux bug reports, primarily due to the large scale of the Linux codebase and the diversity of bug causes. In the evaluation section, the authors assess both traditional information retrieval–based methods and agent-based fault localization methods, finding that compared with SWE-Bench, these agents experience more than a 15% drop in recall on LinuxFLBench. Finally, the authors propose a new framework to mitigate these challenges.
weaknesses: - When constructing the benchmark, the authors still define the ground-truth root cause based on the patch location. While simple, this approach can be imprecise since the patch location does not always correspond to the actual root cause.  
- The benchmark focuses solely on Linux bugs. Although Linux is one of the largest codebases, the motivation provided by the authors is not fully convincing:  
  - **Limited Observability**: the main challenge lies in the lack of runtime information. However, couldn’t one instead use bug reports from other large projects such as Redis or QEMU, filtering out those that include runtime information? Moreover, this challenge does not seem to be reflected in the benchmark’s evaluation results. The difficulties summarized in the evaluation are also unrelated to this point.  
  - **Diverse Impact**: while the diversity of root causes is indeed a major challenge in FL, is this a sufficient reason to choose only Linux? Couldn’t other large-scale projects (e.g., Redis, QEMU) also achieve this? Furthermore, does maximizing root-cause diversity conflict with the decision to focus solely on Linux cases?  
- In the evaluation, the authors only used GPT-4o, which is not a state-of-the-art (SOTA) LLM and lacks strong reasoning capabilities. Since the performance of agents is highly correlated with model capability, although the authors discuss this limitation, I would still encourage them to include comparisons with current SOTA models such as GPT-5 or Claude 4.5.