RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing
Authors: Jinyao Guo
Chengpeng Wang
Xiangzhe Xu
Zian Su
Xiangyu Zhang
Venue: ICML 2025 poster (2025)
Keywords: agent
code reasoning
code auditing
bug detection
Primary area: applications
TL;DR: LLM-powered code auditing technique for detecting security vulnerabilities at the repository level

Abstract: Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability.

This work introduces an LLM-based agent, RepoAudit, designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project. Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit.

REPOAUDIT: An Autonomous LLM-Agent for Repository-Level Code Auditing
Jinyao Guo Chengpeng Wang"!
Xiangzhe Xu'
Zian Su
Xiangyu Zhang
Abstract
LLM-powered IDE plugins, such as Copilot, enable agile code generation by facilitating flexible interactions with LLMs (Barke et al., 2023). However, in the era of LLMs, auditing the rapidly expanding codebase presents a more formidable challenge than writing code. Traditional pro- gram analysis techniques, such as dynamic and static analy- sis, focus primarily on observing runtime behaviors or sym- bolically reasoning about intermediate code generated dur- ing compilation (King, 1976; Cadar al., 2008; Calcagno et al., 2009; Sui and Xue, 2016; Shi et al., 2018a), Un- fortunately, software systems are often non-executable and even uncompilable during the human-I.LMM collaborative development phase. Moreover, existing analysis techniques demand substantial expertise, such as a deep understanding of compiler internals (Zhang et al., 2024; Zhou et al., 2024). Therefore, current program analysis approaches often fall short in meeting the practical requirements of code auditing in real-world contexts (Johnson et al., 2013).
Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising ca- pabilities for this task without requiring compi- lation, while also supporting user-friendly cus- tomization. However, auditing a code repository with LLMS poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large- scale repositories incurs substantial time and to- ken costs, hindering efficiency and scalability.
This work introduces an LLM-based agent, RE- POAUDIT, designed to perform autonomous repository-level code auditing. Equipped with agent memory, REPOAUDIT explores the code- base on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator mod- ule to mitigate hallucinations by verifying data- flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. REPOAUDIT detects 40 true bugs across 15 real-world benchmark projects with precision of 78.43% requiring on aver- age only 0.44 hours and $2.54 per project.Also. it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced REPOAUDIT at https: //github wwwpe///u////ePPPeeee
In recent years, extensive research has focused on leverag- ing LLMs for code auditing via prompt engineering (Fang et al., 2024; Hao al., 2023; Ding et al., 2024). Unlike traditional analysis techniques, LLM-driven code auditing directly analyzes source code without program compilation or execution. By describing analysis requirements through natural language and few-shot examples in prompts, the cus- tomization of analysis is also significantly simplified. How- ever, many existing LLM-driven code auditing techniques are largely restricted small-scale codebases, such as smart contracts (Sun et al., 2024; Zhang and Zhang, 2024), and lack the capability to support repository-level code auditing in complex, real-world scenarios.
Direct Prompting Hardly Works. A straightforward solu- tion is to break down a repository into smaller pieces and prompt the model with individual pieces. This approach often falls short for non-local bugs, which may require rea- soning across a large number of interconnected code snip- pets spanning multiple functions, classes, and files. Even with significant future advancements in model reasoning capacity, the fundamental differences between programs and natural language texts for which transformer models were designed make LLMs insufficient for comprehensive repository analysis. In particular, code repository can be conceptualized as an enormous graph, with nodes repre- senting individual statements and edges capturing intricate
1. Introduction
The rapid innovation of large language models (LLMs) has remarkably enhanced the productivity of software develop- ers (Wang et al., 2021; ;; Rozière et al., 2023; Guo et al., 2024).
'Equal contribution 'Department of Computer Science, Pur- due University, West Lafayette, IN, USA. Correspondence to: Jinyao Guo <guo846@ <wang6590@@ purdue..uuuuu..
@guo846@@urrue.cduu>> pur, Chengpeng Wang
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Levd Code Auditing
relationships such as control flow, data flow, and other in- terdependencies between statements. These relationships are precisely defined and immensely complex at the reposi- tory level. For instance, the number of fata-flow edges in the project shadowsocks 11 i bev exceeds one million, significantly surpassing the complexity of the implicit graph- ical structure present in any LLM's pre-training data sample. Such distribution differences render direct prompting inef- fective as shown by our experiments in Section 2.2.
when scanning for null pointer dereference bugs. Second, the explorer traverses the relevant functions on demand. Similar to how human auditors analyze code function by function, the explorer starts by querying the LLMs with the functions containing these starting points. Instead of ex- plicitly and programmatically enumerating individual paths inside a function, as in traditional compiler- automated scanners, the explorer leverages LLMs' inherent ability to implicitly distinguish relevant paths from irrelevant ones and only reasons about the former. If invocations to other functions and returns within function are considered rele- vant after analyzing paths within the function for instance, when null pointer values propagate through these function boundaries- system synthesizes follow-up prompts to extend the scanning into callee or caller functions as needed, Third, REPOAUDIT checks the output of the explorer before storing it in the agent memory and also examines the bug report candidates by examining the path conditions of the buggy program paths. This validation design can signifi- cantly improve the precision of REPOAUDIT.
In addition, detecting many types of bugs requires reasoning about properties along specific program paths. For instance, a memory leak occurs when allocated memory is not freed along some program path. Detecting such so-called path- sensitive bugs (Shi et al., 8a) necessitates unfolding the program's graph structure into individual paths and ana- lyzing these paths one by one. However, this leads to the well-known path explosion problem, as the number of paths grows exponentially with the number of statements. Hence, direct prompting is akin to presenting large project on an enormous screen and expecting a human auditor to identify bugs along complex and lengthy paths solely by reading and interpreting the code-a a task highly unlikely to succeed.
We implement REPOAUDIT powered with Claude 3.5 Son- net and test it on three typical bug types of memory con- curruption. We first evaluate REPOAUDIT upon fifteen real- world projects used in existing studies, with an average size of 251 KLoC. It is shown that REPOAUDIT effectively reproduces 21 previously reported bugs and uncovers 19 newly discovered bugs, 14 of which have already been fixed in the latest commit, achieving a precision of 78.43% In contrast, the industrial static bug detector Amazon CODE- GURU reports 18 false positives (FPs) with true positives (TPs), while Meta INFER also only reports seven TPs along with two FPs. Besides, REPOAUDIT exhibits high efficiency and incurs low token costs, averaging 0.44 hours and $2.54 per project. Powered by Deepseek R1, Claude 3.7 Sonnet, and OpenAl o3-mini, it achieves the precision of 88.46%, 86. 16,7%,, and 82.35%, respectively. To demonstrate its real- world impact, we further scan ten actively maintained repos- itories and detect 185 new bugs in two months, 95 and 79 of which have been confirmed and fixed by developers, respectively. Notably, REPOAUDIT facilitates development- time code auditing, which cannot be supported by Meta INFER and other compilation -eppedddnnnneeeeentt bug detectors. To the best of our knowledge, REPOAUDIT the first purely LLM-driven code auditor for real-world code repositories.
Human Auditing. In practice, human auditors do not op- erate in such a manner. As revealed by existing cognitive science literature (Anicic et al., 2012), humans are highly effective at reasoning about events that occur in order, e.g- in temporal or spatial order. Human auditors hence tend to explore complex graphical structures in code by following paths that denote the execution order. To avoid excessive exploration, they traverse only subset of paths most rele- vant to the targeted property, leveraging implicit abstraction to preclude irrelevant paths. An example of detecting null pointer dereference can be found in Section 2.1.
Our Solution. It is widely believed that LLMs operate sim- ilarly to humans but with significantly greater "endurance" and access to a broader spectrum of knowledge (Li al., 2023; Long et al., 2024; Qian et al., 2024). Building this perspective, we propose a novel LLM--asedd repository auditing agent, named REPOAUDIT, inspired by human au- diting practices. REPOAUDIT addresses the fundamental misalignment between LLMs' tendency to reason sequen- tially and the inherently complex graphical structures of soft- ware repositories through path-sensitive and demand-driven graph traversal. By leveraging LLMs' abstraction capabil- ities, REPOAUDIT mitigates path explosion by excluding irrelevant code regions and sub-paths. Also, it minimizes inherent hallucinations by sanitizing final outputs through the validation of several well-formed properties.
2. Preliminaries
In this section, we first discuss the essence of repository- level code auditing. Next, we illustrate the limitations of the LLMs in this task. Finally, we highlight several intrinsic strengths of LLMs in tackling primitive tasks that can be leveraged to build our repository-level auditing tool.
More specifically, the agent REPOAUDIT consists of three components, including the initiator, the explorer, and the validator. First, the initiator identifies starting points based on the properties under investigation, such as NULL values
2

---

REPOAUDIT: An Autonomous L.M--geet for Repository-Leve Code Auditing
poogle/pootauu//deerrrror.
sec/ssffa/pirge//pssss..c
bool is e...ptiionaluss (u32 rember_ oofeer) (-)
bool 1s_ repeatad() const (-)
1. Value* ficld2jsonniiceldsssppppo *field, Value* default)
arcc/safa//brppc//pfsso.cc
bool repegted field->is repeaate)))
Value" Json 1
Value" parse parse_nsgt nig(const Nessago tusg, Value: Allocctooooa allocator)
1f (ropeated) { json new (x 5.
2.
3.
For (size_t Bj I- count; (+))(
... (field-sis poppidddis_pptionna10 optioonl (1)
8.
switch (field-xcpp. Ex)
5.
9.
case CPPTYPE_OOIRE
6.
else
10.
7.
Value* field ison (onss,)sonnns,, field, allocator)
11.
default: breaks
8.
root-sAdHenbeecCarre FFeeld Ison, allocator);
12.
9.
13.
(Irepeated II field-rvalue. GetType() t= kirrayType)
18.
}
Null Pointer Dereference
14.
retunn 1son;
15.
return default;
return root;
16.
13.
arc/rapiidjan/docceent.
1. AddMesber (Cn* name, Gener1ccaluee value, Allocaton& allocator) (+)
1. Type GetType(){ return static_ canttcccaattypoofffags kTTypeaask);
}
Figure 1. A simplified code snippet from the project sofa -porpc contains a real NPD hug found by REPOAUDIT. The hlue dashed arrows indicate the edges in the call graph. The red solid arrows show the data-flow facts indicating the null value propagation. The call graph of the project contains 1,508 nodes and 6, 6,196 edges, while its data dependence graph contains 160,875 nodes and 360,096 edges.
2.1. Auditing Entails Path-Sensitive Reasoning on Complex Graphs
ing the data-flow fact NULL + json@i8,, which labeled with ① in Figure 1. When repeated is false, the value of json at line 4 propagates to the return statement at line 14. This data-flow fact, denoted by json @s4 + j sonn@114 is labeled with ② Subsequently, in the func- tion parse_msg, the return value of field2 json is as- signed to the pointer field_ jsson at line 7 and further dereferenced line 8 of the function pa aaressssseess which eventually causes an NPD bug.
While several bug types, such as API misuse (Li et al., 2021), require only localized reasoning upon abstract syntax trees (ASTs) and are relatively easy to detect with highly effec- tive scanners, many critical bug types demand modeling the entire project as massive graph and reasoning about prop- erties along and across individual paths in that graph. For example, detecting Null Pointer Dereference (NPD) bugs relies on constructing and analyzing a specialized graph structure called the data dependence graph (DDG) (Fer- rante et al., 1984), where the nodes represent statements, and the edges indicate data-flow facís between different pro- gram values at specific statements. Specifically, a data-flow fact exists from the variable u at the statement sta to the variable v at the statement sla, denoted as w@sta > v@slp, if the value of the variable at the statement st, may affect the value of the variable u at the statement st, following some program path. A program path is sequence of state- ments that follow the execution order. Also, it is feusible if there exists an input that satisfies all the conditional checks along the path. Hence, a Nata-flow fact can occur between two statements that are far apart, such as when a global variable read and written across different directories.
Furthermore, we study the 2024 CWE Top 25 Most Dan- gerous Software Weaknesses, curated list of the most critical and prevalent vulnerabilities underlying the 31,770 Common Vulnerabilities and Exposures (CVE) records re- ported in 2024. Our investigation reveals that 19 out of the 25 weakness (76%) categories necessitate global, path- sensitive reasoning of source-sink reachability upon call graphs, DDGs, and control -low graphs (CFGs), whereas only 6 categories can be effectively identified through local- ized analysis based on ASTs.
2.2. LLMS' Inadequacy
According to (Rozière et al., 2023; OpenAI, 2023), many foundation models are initially pre-trained on relatively short text or code snippets. For long contexts, models like DeepSeek- V3 (Liu et al., 2024), Llama3 series (Dubey et al., 2024), and QWen2.5- adopt NTK-aware length interpolation such as YaRN (Peng et al., 2024) progressively extend context window from the initial 4K/8K to 128K tokens. Although these models perform well in "Needle in Haystack" (Kamradt, 2023) evaluation, the task is intended for RAG-and does not align well with the path-sensitive program understanding ability needed in our application.
Determining whether a data-flow fact may hold in the pro- gram requires collecting feasible program paths and analyz- ing data-flow facts along them. Consider the detection of Null Pointer Dereference (NPD) as an example. Given the DDG, the code auditor ought to find a chain of data-flow facts leading from a null value (as a source value) to a deref- erenced pointer (as a sink value) along feasible program path. In Figure 1, for instance, the function fiel c2 jsn initializes a null value at the line 4 of the function, induc-
(Hui et al., 2024) typically
3

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Leve Code Auditing
Please analyze the function field2json in Figure 1. Me only about the statonents that affect the propagation of the NULL value at line 3. Please simplify the function field2json for progran abstraction.
What are the points-ta facts of the variable jsan at 1ine 8 in the function field2json shown in Figure 17
The pointer variable json is initialized to NULL value at line and then points to the newly allocated nenory
line if repeated is true, Hence, we have: The pointer variable json points to the object allocated at line 6;
Here 1s the progran after abstraction.
repeated
1. Value* fieldkjson
*field,
(2) repeated false: the pointer variable jsan NULL
Value* default)
2. bool repeated fheld->is_repeatelle
(b) An example of pointer handling
Value* json NULL;
4. if (repeated) {
Collect the feasible progran paths from the function field2json in Figure
json Velue((rppdijoon::Arrryyyy
6.
(Irepeated fieldyywelee.GeTTyye(
kArrayType)
The branch conditions at lines 5 and contredict.
8.
return ]son;
Feasible path 1: Lines 1-5, 8-12, 13-14 Feasible path 2: Lines 1-5, B-12, 15
Feasible path 3: Lines 1-7, 8-12, 13-14
(n) An example of program abstraction
(c) An example of feasible program path exploration
Figure 2. Three examples showing intrinsic strengths LLMs
To validate our speculation, we conducted a controlled ex- periment where we prompt Claude 3.5 Sonnet with all the five functions shown in Figure to identify the NPD bug, following& a methodology similar to a recent study (Fang et al.. 2024). This is a controlled experiment because, in practice, we cannot guarantee knowledge of the compre- hensive set of functions associated with bug. The model exhibited substantial hallucinations, reporting that almost all the dereferenced pointers have null values. Even if we improve the prompts by offering several few-shot examples and explanations on how a null pointer dereference bug oc- curs, the model still hallucinates, producing false positives and incorrect explanations. More numerical results can be found in Appendix A.
in the program and focuses on key statements such as null pointer assignments, value propagations, conditional checks guarding the propagations, and cross-function propagations (e.g., passing the pointer to a callee function via a parameter, returning it to a caller function, or writing it to a global vari- able). In a preliminary experiment shown in Figure 2(a), for example, we feed the function eield2
in Figure 1 to Claude 3.5 Sonnet and ask it to abstract the program. Ob- serve that the program returned by the LLM only contains critical statements relating to the null value propagation, while irrelevant ones are removed, such as the switch state- ment from line 8 to line 12 in the function field2 json in Figure 1. Notably, human auditors often perform such abstraction implicitly, enabling them to analyze complex code without the excessive path exploration that hampers classic tools like symbolic execution (Cadar et al., 2008) and software model checking (Clarke, 1997).
2.3. Intrinsic Strengths of LLMs
On the bright side, we observe that LLMs can effectively perform basic analyses when the scope is limited. This ca- pability enables to surpass traditional program analysis methods, which require compilation and struggle to scale ef- ficiently. Specifically, we identify several primitive abilities critical for auditing: program abstraction, pointer handling, and feasible program path exploration. While traditional au- diting tools rely on heavy-weight yet rigorous techniques to achieve these capabilities, skilled human auditors often rely on intuition to handle such challenges within constrained analysis scope. We observe similar traits in LLMs.
2.3.2. POINTER HANDLING
Pointer variables in C and reference variables in Java may point to different memory objects depending on their run- time values. Consequently, reads and writes via pointer dereferences create data-flow facts that dynamically vary based on pointer values. Determining the set of memory objects pointer variable may point to is known as points-to analysis (Smaragdakis et al., 2015), one of the most chal- lenging problems for downstream static analysis, such as DDG construction. Traditional points-to analysis techniques often rely on conservative and relational methods, which tend to significantly ovpr-appro the set of possible memory objects, complicating downstream analysis tasks. In contrast, human auditors can often intuitively and ac- curately determine the points-to facts of a specific pointer variable through their understanding of program semantics. Advanced L.MMs exhibit similar capabilities, particularly within the scope of individual functions. As shown in Fig- ure 2(b), for example, we prompt Claude 3.5 Sonnet with the function fiel d2 son in Figure and query the points-to
1. 1. PROGRAM ABSTRACTION
Abstraction is essential for scalability in program analy- sis. Given a property, a set of initial program points, and defined scope (e.g.. a function), abstraction identifies the subset of statements relevant to the property within the scope. These statements form a seff-contained, smaller pro- gram, significantly reducing the number of paths to analyze. For instance, in the NPD detection (as illustrated in Sec- tion 2.,,, abstraction targets how null values are propagated
4

---

REPOAUDIT: An Autonomous LLM-Agent for Repository- Level Code Auditing
Parser
LLM
i tv
Initistor
Program Definiton
Bug
Validator
Bug Reports
Explorer
Memory
Figure 3. The architecture of REPOAUDIT
facts of the return value of the function. The model accu- rately identifies the two possible points-to facts within the function. It even identifies the path constraints that make the corresponding points-to facts hold. In contrast, the sym- bolic static analyzer SVF (Sui and Xue, 2016) computes the points-to facts without path conditions by default due to its inherent limitations in semantic analysis.
sponses for each function, and dedicated agent memory ensures that analysis results across functions are seamlessly shared. To achieve cost-effective and path-sensitive analysis at the function level, REPOAUDIT leverages LLMs' intrin- sic capabilities by providing explicit prompts for program abstraction, pointer handling, and feasible program path exploration, fully capitalizing on the model's strengths.
Figure 3 depicts the architecture of agent. The initia- tor tool takes specified bug definition and the target repository as inputs, identifying the source values for anal- ysis, such as the null values for the NPD detection. Each source value triggers a scanning procedure. The explorer tool conducts iterative, demand-driven exploration of the repository by prompting the model to analyze one func- tion at time. The results of each analysis are stored in the agent memory, guiding further exploration. The anal- ysis prompts are dynamically generated for each function, providing detailed instructions for abstraction tailored to the specific function and adjusted based on the results of prior analyses. While the agent may still hallucinate the analysis of single functions and produce false positive bug reports, set of validator tools verify the result of the explorer from multiple perspectives, including the validity of the control flow order and the satisfiability of the path conditions across functions.
2.3.3. FEASIBLE PROGRAM PATH EXPLORATION
A bug report is often considered valid only if it provides feasible program path (from the root cause to the symp- tom) as evidence. To determine feasibility, traditional tools rely on modeling conditional checks along a path as sym- bolic constraints (e.g., in the form of first-order logic for- mulas) and querying a theorem prover (e.g.. SMT solver Z3 (de Moura and Björner, 2008)) to check if an input can satisfy the constraints. This process is computationally pensive and prone to failure, as converting program paths to logic formulas requires exploring an explosively large number of program paths and modeling a wide range of program behaviors that lack direct representation in logi- cal term, such as loops, array indexing, aliasing, pointer arithmetic, and unbounded string operations. In contrast, hu- mans, as well as the LLMss, rely on abstraction and intuitive logical reasoning to assess feasibility, which is highly effec- tive within limited scope. In Figure 2(c), we demonstrate an example of feasible program path exploration upon the function f field2 json using Claude 3.5 Sonnet. Observe that the model can skip the irrelevant statements, such as the switch statement, avoiding exploring a huge number of program paths. It also discovers the contradiction between the branch conditions at lines 5 and 13, which refutes pro- gram paths covering both lines 6 and 14, thereby reporting the three feasible program paths in Figure 2(c).
3.1. Initiator
The initiator identifies the starting points of scanning (i.e., the sources). In our implementation, we employ the t tree-sitted parsing library to create a suite of tailored pattern matchers for the sources of the bug types supported by REPOAUDIT. These matchers are concise, often con- sisting of just few lines of code, and require a one-time implementation effort. Alternatively, recent advancements offer a promising avenue of synthesizing such matchers using LLMs (Wang et 2024a).
3. REPOAUDIT
Building on the findings of the previous section, the core design rationale of REPOAUDIT is follows: Since LLMs struggle to reason about path-sensitive properties in large- scale program graphs, REPOAUDIT employs an agent- centric approach to navigate these graphs externally, prompt- ing the model with one unit (c.g., a function) at a time. This demand-drin -riven navigation adapts based on the model's re-
3.2. Explorer
For each source value identified by the initiator, the explorer conducts a round of scanning over the repository. During each round, the explorer traverses a subset of functions on demand, beginning with the identified sources. It queries the I.IM to analyze one function at a time, storing the results
5

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Levd Code Auditing
Task: Determine data-flow facts starting from a given value along
Agent Memory. After analyzing a function, the explorer obtains a set of data-flow facts for each feasible program path and stores them into the memory of the agent. Specif- ically. the agent memory is a function M relating to the function I and a program value rãs. Each element in Mif, v@s) is a pair of a program path and a set of data- flow facts. After analyzing the function f field2json and the null value at the line 4, for instance, the memory maps 12 NULL@sa) as follows:
different program paths in a single function...
Hints: You can collect the feasible program paths into steps:
(1) Identify the aliased pointers in the program.
(2) Extract the critical statements relating the value propagation,
like function call statements, retum statements, and assignments.
(3) Collect the feasible program paths covering critical statements. For each feasible program path, simulate the program execution and determine the data- flow facts starting from the given value...
Examples: Here are several examples.
Example I: User: [Program] [Question]
System: <<ath information: Lines 4-7, 11> Explanntatnn
M(Field2jjon, NUI.I.Qs.))))
<Data-flow facts: ptr@t, qtr@t,, ptr@e, 4 1tr
[Other exemples]
(NULLGS + json@si, Jaon@ss + json@suu))
Question: Given FUNCTION] what are the data-flow facts from
VVL. NAMF at line VVL LINE along different program paths?
(pz, (NULLIES + jsonuss))),
Figure 4. The prompt template for analyzing individual functions
(ps, (NULL + json@si))
in the agent memory. The explorer performs several actions, each guided by corresponding prompts. These actions in- clude: analyzing individual functions, selecting functions for exploration, and generating bug report candidates. We will introduce the details of the three actions as follows.
3.2.2. SELECTING FUNCTIONS FOR EXPLORATION
After analyzing a function, if the targeted program value propagates across function boundaries, the explorer queries the underlying call graph to identify the relevant functions for further exploration. This follow-up exploration is guided by the program values escaping the current function bound- aries. For example, in Figure I, the null value at the line 4 of Field2 json is propagated to the return value. Hence, in the next step, the explorer analyzes the caller function of fiel lello d2 json, i.e., parse and examines how the return value is propagated.
3.2.1. ANALYZING INDIVIDUAL FUNCTIONS
As discussed in Section 2.1, scanning for most bug types can be reduced to traversing a limited set of graphs, such as the DDG and the CFG. This enables the use of general analysis prompts for these graph types, eliminating the need for bug-specific prompts. Moreover, instead of explicitly enumerating and analyzing individual paths, we leverage the intrinsic capabilities of LLMs to distinguish paths and per- form path- south-sesitive reasoning. The key of this approach lies in pointer handling and program abstraction demonstrated in Section 2.3.2 and Section 2. 1, respectively, which im- significantly smaller code
Note that if the program value does not escape, the analysis does not lead to the explorations of other functions. For example, if we consider the second and third program paths, i.e., P2 and ps the example shown in Section 3.2.1, the value of json at the line 4 of the funetion field2) json does not propagate. Hence, the explorer does not enter any other functions. Furthermore, we leverage the existing data- flow facts stored in the memory as caches to avoid redundant analysis. Specifically, before analyzing the propagation of a specific program value in a function, the explorer first checks the agent memory and determines whether this has been done before. In our evaluation, we will quantify how the caching strategy reduces computational costs.
plicitly reduces the function snippet with far fewer paths. This can eventually facilitate the efficient exploration of feasible program paths demon- strated in Section 2.3.3. We design the prompt template used for analyzing individual functions in Figure 4. After describing the task at the beginning, we offer the three-step hints to the LLM, enforcing the LLM to unleash its power step-by-step in pointer handling, abstraction, and feasible program path exploration. By offering several few-shot ex- amples, we pose the question at the end and ask the LLM to identify the data-flow facts starting from the initial value(s) of interest along different feasible program paths.
3.2.3. GENERATING BUG REPORT CANDIDATES
After analyzing a function, the explorer evaluates whether any new bug candidates can be identified. Consider the NPD detection as an example. The explorer examines whether it identifies any new data-flow facts reaching a sink value, i.e., a dereferenced pointer. If so, a bug report is generated by assembling the complete trail of lata-flow facts across functions along with the corresponding inter- -roceduulll program path. For example, the explorer enters the func- tion parse mms and examines how the return value of field2 json propagates. Based on the two discovered data-flow facts labeled as 3 and 4 in Figure 1, the explorer
Consider the function filed2 (addjsson jao and the initial null value at the line as an example. Utilizing the instrinsic strengths of LLMs, REPOAUDIT collects three feasible pro- gram paths shown in Figure 2(c). By simulating the program execution along the first program path, denoted by Pl, the LLM identifies the data-flow facts NULL@s; + json@s1 and json@s, > json@s 14. For the second and third pro- gram paths, denoted by p2 and ps. the LLM only identifies the dlow-flow fact NJLLAs > @ss.
6

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Lewe Code Auditing
Task: Determine whether a given path is feasible or not.
Table
The statistics evaluation subjects
Hint: branch conditions conflict or variable values contradict the required conditions, the path is infeasible.
Bug Type
ID
Repository Name
Size (LoC)
Stars
N1
sofä-pbrpe
40,723
Examples: Here are several examples.
N2
BaaggeMagicc/MMaickcCoo
242.555
12.6K
Example User: [Program] [Question]
NPD
N3
coturn/sto/erver
8,976
11.7K
System: <Explanation> <Annwwrr Yes or No>
N4
libfreenect
37,582
3.6K
[Other examples]
N5
openidap
442,955
486
Question: Given the functions [FUCCTION] is the following
MI
libsass
40,934
4.3K
program path feasihle? [ATH
M2
memeached
14.654
13.7K
MLK
M3
linux/driver/nct
914,025
186K
Pigure 5. The prompt template for feasibility validation
M4
linux/sound
1,378,262
186K
M5
linux/mm
171,721
186K
reaches a sink value, i.e., the dereferenced pointer at line 8. Hence, the explorer identifies a potential bug and report by concatenation the data-flow facts O 2, @ and a For several bug types, such as MLK, the explorer reports potential bug if it fails to reach any sink point along feasible paths, e.g., the argument of free function.
U1
Redis
179,723
67.7K
U2
linux/drivers/pecc
2,130
186K
UAF
U3 shadowsocks-libee
71.080
186K
U4
wabl-tool
3,214
7K
US
icufeu4c/souree//i 88n
220,359
2.9K
common practice in evaluating reasoning tasks (Ye et al., 2023), we set the temperature 0.0 to reduce the random- ness. Similar to existing code auditing works (Heo et al., 2017), we introduce an upper bound K on the calling con- text and set it to 4, i.c., REPOAUDIT investigates flta-flow facts across a maximum of four functions.
3.3. Validator
To improve the quality of bug reports, we introduce two kinds of validation mechanism for the explorer.
Alignment Validation of Data-flow Facts and Control Flow. When dealing with complex code, the LLM may hallucinate and incorrectly infer a data-flow fact u@s1 + 20s2 along some program path p that violates the control- flow order. Specifically, this implies that statement $2 must appear after another statement 81 in the program path p, yet the model concludes that a variable defined at 82 can be used by $1. To detect such misalignments, parsing-based analyzer is employed to verify the control-flow order. Only the data-flow facts that conform to the control-flov order will be stored in the agent memory.
4.1. Bug Types and Dataset
We focus on three bug types, namely NPD, MLK, and UAF, which are among the CWE Top 25 most dangerous weak- nesses. Our evaluation first aims to reproduce bugs reported in previous works. Specifically, we investigate recent works in the venues of computer security and software engineering and collect the bug reports published by the authors (Huang et al., 2024; Shi et al., 2021; 2018b). Also, attempt to detect new bugs within the targeted code repositories. As shown by Table 1, we choose five well-maintained projects for each bug type from the bug reports of previous works, which mostly have thousands of stars on GitHub with 251 KLoC and thousands of functions on average.
Path Feasibility Validation. While path feasibility within individual functions is inherently checked by the explorer, contradictions can arise in conditional checks across func- tions, resulting in infeasible inter-procedural program paths. Recall that each bug report is the concatenation of multiple data-flow facts, such as ①. @ I. and 4 in Figure 1. The data-flow facts in different functions may hold under the specific conditions along the corresponding program paths. A bug report candidate is valid only if the path conditions in different functions do not contradict, implying their logical conjunction satisfiable. To check the validity, we prompt the I.IM with the inter-procedural path and task it with identifying any contradictions in the path conditions. If a contradiction is found, the bug report is discarded. The prompt template shown in Figure 5.
4.2. Evaluation Results
Main Result. As shown in Table 2, REPOAUDIT success- fully detects all the previously reported bugs, and mean- while, reports 19 new bugs in historic versions, 14 of which have already been fixed in the latest commit. In total, it re- ports 40 TPs and 11 FPs, resulting in precision of 78.43%. Notably, 21 TPs are inter-procedural bugs. REPOAUDIT also demonstrates both efficiency and cost-effeectvvensss. On average, it takes only 0.44 hours (i.e., 1,577.22 secs) to analyze a project, completing the code auditing within 100.67 prompting rounds. The average cost per project audit is $2.54, with each true bug detection costing $0.95. Hence, REPOAUDIT can effectively detect the bugs upon real-world programs with low time cost and financial cost.
4. Evaluation
We utilize the tree-sitter parsing library to provide a set of primitive tools for REPOAUDIT, e.g., call graph constructor and control flow order validator. We select LLM Claude 3.5 Sonnet to power REPOAUDIT, Following the
Comparison with LLM--rrven Bug Detectors. We com- pare REPOAUDIT with two kinds of LLM-driven tech- niques, namely end-to-end few-shot CoT compting-based approaches (Chen et al., 2023; Ding et al., 2024) and agent-
7

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Levee Code Auditing
Table The statistics of REPOAUDIT powered by Claude 3.5 Sonnet. The column "Old" denotes the hugs reported by existing works.
pair (m, n) in the column New indicates m new bugs detected in the versions used by existing works, of which still exist in the latest versions. The columns "Intra" and "Inter" show the numbers of intra procedural and inter procedural bugs, respectively.
TP
Feature
# Tokens
Bug Type
FP
Prompts
Financial ($)
Time
Old
New
# Intra
# Inter
Input
Output
NI
(3.3)
0
4
145
709,919
55,863
2.97
2026.13
(1,0)
4
97,717
8,518
0.42
283.84
NPD
(1.0)
599,674
52,936
2.59
1747.90
(0,0)
126,852
13,654
0.59
435.09
(5.4)
O
6
420,710
31,375
1.73
1059.57
MI
(2.1)
2
I
205
1,132,763
85,279
4,68
2,917.91
M2
(6.6)
4
3
146
845,148
71,243
3,60
2282.31
MI.K
M3
(0.0)
0
10,481
1,019
0.05
34.34
M4
(0.0)
5691
619
0,03
17.94
(0.0)
181,348
20,779
0.86
599,92
(0.0)
179,939
17,547
0.80
582.23
U2
(0.0)
0
8900
869
0.04
31.95
UAF
U3
(0.0)
0
48
317,713
23,067
1.30
791988
U4
(0,0)
10
48,087
5,883
0.23
185.22
US
(1.0)
662
4,534.444
303,645
18.15
10,661.98
Average
100.67
614,625.73
46,153.07
2.54
1,577.22
Table statistics Meta INFER Amazon CODEGURU
true bugs and two false positives. Amazon CODEGURU sup- ports the detection of NPD and UAF bugs, while it reports 18 false positives with no true positives. Compared with Meta INFER, REPOAUDIT obtains comparable precision while detecting more true bugs. More detailed illustrations on the comparison results are provided in Appendix B.
INFER
CODEGURU
Bug Type
ID
Build
TP
FP
TP
FP
NI
NA
N/A
0
N2
1
0
NPD
N3
NA
N/A
0
N4
5
N5
M1
Ablation Study. We introduce three ablation variants of REPOAUDIT, without abstraction, without validators, and without caching. We observe that without abstraction, the number of TPs is decreased by 47.50% and the number of FPs is increased by 181.82% Without validators, the number of FPs increases by 245.45% Without caching, the costs become 3-4 times higher on average, and the worst case, 30 times higher. Details can be found in Appendix C.
MLK
N/A
N/A
N/A
M4
0
M5
N/
N/A
U1
N/A
N/A
0
U2
N/A
N/A
0
UAF
U3
X
N/A
N/A
0
U4
O
O
U5
O
0
13
Total
7
2
18
Different Model Choices and Temperature Settings. We also evaluate REPOAUDIT with Deepseek R1, Claude 3.7 Sonnet, and OpenAl o3-mini, achieving precisions of 88. 88.46%, 86.79%, and 82.35%, respectively. In addition, we assess REPOAUDIT powered by Claude 3.5 Sonnet under the temperatures 0.25, 0.5, 0.75, and 1.0, achieving con- sistently high precision (2 72.92%) and recall (2 85. (5).1%). More details are provided in Appendix D and Appendix E.
centric approaches (Wang et al., 2024a; Li et al., 2024a;b). It is shown that CoT prompting can only detect one true bug in the single function-level bug detection and 10 true bugs in the multiple-level bug detection when analyzing the functions relating to the true bugs detected by REPOAUDIT. For the agent-centric solution LLMDFA, the numbers of its prompting rounds and token costs 165.23 and 123.18 18 times with the ones of REPOAUDIT on average when it ana- lyzes relevant functions relating to the buggy program paths, showing its high computation costs analyzing real-world programs. More details can be found in Appendix A.
Real-Worle Impact. We further scan nine open-source GitHub projects spanning diverse domains and sizes rang- ing from 14K to 1.7M LoC. On average, they have 420K LoC and 8.8K GitHub stars, reflecting their complexity and popularity. Table 4 shows the detailed statisties. Specifically, the columns TP and FP show the numbers of TPs and FPs reported by REPOAUDIT, respectively, while the columns Con and Fix denotes the numbers of confirmed bugs and fixed bugs, respectively. Overall, REPOAUDIT detects 185 true bugs with the precision 85.71 %. Notably, 95 and 79 bugs confirmed and fixed by developers, respectively.
Comparison with Industrial Tools. We compare REPOAU- DIT with two representative industrial tools, namely Meta INFER (Meta, 2025) and Amazon CODEGURU (Amazon, 2025). As shown in Table 3, two projects (labeled by X) can- not be successfully built in our environment. Another five projects (labeled with NA) canmot handled by INFER due to incompatibilities, a prominent limitation of build/compi- lation dependent tools. total, Meta INFER reports seven
8

---

REPOAUDIT: An Autonomous LLM-Ageet for Repository- Levell Code Auditing
Table 4. The statistics of REPOAUDIT in nine additional real-world projects
NPD
MI.K
UAF
Project
Size (LoC)
Stars
TP
FP
Con
Fix
TP
FP
Con
Fix
TP
FP
Con
Fix
chickhouse-olbc
209,197
258
0
0
0
0
2
0
0
2
0
0
hlop
35,910
6.9K
1
3
0
1
0
0
0
0
0
0
TrinatrEmulator
1,767.100 100
292
2
2
0
2
4
4
0
0
rtl_433
66,253
6.5K
0
0
0
0
0
0
frr 1,050,020
3.6K
4
5
4
0
0
0
0
0
libuy
80,360
25K
86
84
0
openldap
442,955
486
3
nginx
505,513
26.4K
memcached
14,654
13.7K
67
Total
107
19
89
13
76
10
66
3 0
4.3. Limitations and Future Works
context for the buggy functions, thereby degrading the va- lidity of such function-level code auditing techniques (Risse and Böhme, 2024). As for repository existing techniques can generally be categorized into two groups. The first group harnesses LLMs to provide special- ized pre-knowledge to symbolic code analyzers or examine initial bug reports, while the main body of the code base is scanned by conventional symbolic analyzers (Wang et al., 2024b; Li et al., 2024b;a). Typically, IRIS employs LLMs to pinpoint sensitive values within programs (Li al., 2024b), aiding traditional analyzers like CODEQL (Avgustinov et al., 2016) in taint-style bug detection. However, due to the com- pilation reliance on symbolic analysis, these techniques lack the capacity for IDE-time analysis. The second category utilizes LLMs as code interpreters, extracting semantic prop- erties via prompt engineering (Fang et al., 2024: Wang et al., 2024c;a; Sun et al., 2024). Typically, LLMDFA (Wang et al., 2024a) and LLMSAN (Wang et al., 2024c) incorpo- rate few-shot CoT prompting to discover data-flow paths for bug detection. REPOAUDIT belongs to the latter category. Unlike LLMDFA, REPOAUDIT adopts a demand-driven strategy for codebase exploration, which avoids exhaustive data-flow summary generation for all functions, thereby enhancing analysis scalability. Additionally, REPOAUDIT outperforms LLMSAN in terms of recall by employing individual function analysis instead of repository-level end- to-end prompting, effectively mitigating hallucinations in the overall analysis.
Apart from the false positives and negatives caused by LLM hallucinations, REPOAUDIT faces several limitations. First, the analysis overhead of REPOAUDIT is highly sensitive to the number of source elements within a repository. In cases where a large number of sources exist, the tool may incur substantial time and token costs. Second, REPOAUDIT is not sound in detecting inter-procedural data-flow facts, as it currently limits flow analysis to a maximum of four functions. This constraint may cause miss complex bugs involving longer call chains. To address these limitations, we outline several directions for future improvement:
code auditing,
Enhancing Model Reasoning Capabilities: We can im- prove the LLM's ability to reason by either adopting more advanced models or fine-tuning the current models for specific tasks. Fine-tuning can be tailored to particular sub-tasks like exploring feasible program paths in single functions. Additionally, incorporating specialized training datasets focused on programming languages, debugging, and code analysis could further enhance the model's accu- racy in these contexts.
Expanding the Tool Suite for Better Retrieval: Develop- ing a more comprehensive tool suite that facilitates the re- trieval across the entire codebase is crucial. A significant en- hancement would involve integrating existing compilation- free analysis tools to identify all potential branches and loops within a program. This integration would offer clearer representation of the program's 's control flow struc- ture to the model. Such an improved RAG design has the potential to significantly reduce the false positives and false negatives produced by RepoAudit.
6. Conclusion
This paper introduces REPOAUDIT, an autonomous LLM- agent that facilitates precise and efficient repository-level code auditing. By mimicking manual code auditing, RE- POAUDIT leverages the intrinsic strength of the LLM, such as program abstraction, and conducts the path-sensitive rea- soning. Powered by Claude-3.5-Sonnel it detects 40 true bugs 15 real- real-world benchmark projects, achieving the pre- cision of 78.43% and reproducing all the bugs discovered by existing techniques. It also detects 185 previously unknown bugs in nine high-profile open- projects, 174 of which have been confirmed or fixed by developers.
5. Related Work
A considerable volume of literature has focused on utilizing LLMs for code auditing (Zheng et al., 2025; Zheng et al.; Li et al., 2024b;a; Wang et al., 2024a). In recent years, various benchmarks like Big Vul (Fan et al., 2020), Vul (Ding al., 2024), and Diverse (Chen et al., 2023) have been established. Nevertheless, these benchmarks lack calling

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Leve Code Auditing
Acknowledgement
Third, classical code auditing requires reasoning about the semantics of IRs and reimplementing the same algorithm for different languages. In contrast, LLMs serve as general code interpreters and have exceptional performance in un- derstanding short code snippets, no matter which program- ming languages are used. By following similar prompting strategies, we can easily extend REPOAUDIT to analyze programs in other languages. including but not limited to C/C++ Python, and JavaSeript.
We are grateful to the Center for Al Safety for provid- ing computational resources. This work was funded in part by the National Science Foundation (NSF) Awards SHF-1901242, SHF-1900300, Proto-OKN 2333736, IIS- 2416835, DARPA VSPELLS HRO01 12000058, ONR N00011423-1-2081, and Amazon. Any opinions, findings and conclusions or recommendations expressed in this ma- terial are those of the authors and do not necessarily reflect the views of the sponsors.
References
Impact Statement
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and gen- eration. In Maric-Francine Moens, Xuanjing Huang, Lu- cia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages
This paper presents work whose goal is to advance the field of Machine Learning, targeting a complicated code- reasoning task, namely repo-level code auditing. We demon- strate the limitations of our work above. We do not expect our work have a negative broader impact, though lever- aging LLMs for repo-level code auditing may come with certain risks, c.g., the leakage of source code in private or- ganizations and potential high token costs. Meanwhile, it is worth more discussions to highlight that our work has the potential to dramatically change the field of software engineering with the power of LLMs. Specifically, LLM- powered code auditing not only enables the analysis of incomplete programs with little customization but addresses other challenges in the code auditing.
Association for Computational Lin- guisties, 2021. doi: 10. 10.18653/V1/2021..22 Ww1.2222.EMNL--MMA. 685.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexan- dre Défossez, Jade Copet, Faisal Azhar, Hugo Tou- vron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 10.48555/ARRXIV23300 12950.
First, classical code auditors primarily rely on specific ver- sions of intermediate representations (IRs) generated by compilation infrastructures. As compilation infrastructures evolve, IR formats often change across compiler versions, necessitating continuous adaptation of the analysis imple- mentation. For instance, the Clang compiler has experienced ten major version updates over the past decade, each intro- ducing variations in the generated IR. These differences require substantial manual effort to migrate and maintain compatibility with newer IRs. In contrast, LLM-powered code auditing operates directly on source code, inherently supporting multiple language standards and eliminating the dependency on compiler-specific IRs.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek coderr When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401. 2024.
Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code- generating models. Proceedings of the ACM on Program- ming Languages, 7(00PSLA1):85-111 2023.
Second, classical code auditing heavily relies on various ab- straction designs, particularly in pointer analysis, which serves as a foundational pre-analysis step. Developers must carefully select and implement specific analysis strate- gics-such as Andersen-style or Steensgaard's 's pointer anal- ysis-each involving distinct trade-offs in precision and scalability. This process requires substantial implementa- tion effort and domain expertise. In contrast, LLMs, which are inherently aligned with program semantics, can interpret code behavior directly. As a result, they obviate the need for manually crafting abstractions or implementing analysis algorithms tailored to particular precision levels.
James C. King. Symbolic execution and program testing. Commun, ACM, 19(7):385-33) -3944,, 1976. doi: 10.1145/ 360248.360252.
Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. KLEE: unassisted and automatic generation of high- coverage tests for complex systems programs. In Richard Draves and Robbert van Renesse, editors, 8th USENIX Symposium on Operating Systems Design and Implemen- tation, OSDI 2008 December 8-10, 2008 San Diego
10

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Leve Code Auditing
California, USA, Proceedings, pages 209-224. USENIX
Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun, et al. Large language models for code analysis: Do {LLMs} really do their job? In 33rd USENIX Security Symposium (USENIX Security 24), pages 829-846, 2024.
Association, 2008.
Cristiano Calcagno, Dino Distefano, Peter W. O'Hearn, and Hongseok Yang. Compositional shape analysis by means of bi-abduction. In Zhong Shao and Ben- jamin C. Pierce, editors, Proceedings of the 36th ACM SIGPLAN-SIGACT Symposium on Principles of Program- ming Languages, POPL 2009, Savannah, GA, USA, Jan- uary 21-23, 2009, pages 289-300. ACM, 2009. doi:
Yu Hao, Weiteng Chen, Ziqiao Zhou, and Weidong Cui. E&v: Prompting large language models to perform is by pseudo-code execution and verification. arXiv preprint arXiv:2312.0847) 2023.
40.1145/448088111 480881.. 1480818441199177.
Yulei Sui and Jingling Xue. SVF: interprocedural static fllue-flow analysis in LLVM. In Ayal Zaks and Manuel V. Hermenegildo, editors, Proceedings of the 25th Interna- tional Conference on Compiler Construction, CC 2016, Barcelona, Spain, March 12-18, 2016, pages 265-266. ACM, 2016. doi:
Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vuinerability detec- tion with code language models: How far are we? arXiv preprint arXiv:2403. 18624, 2024.
Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang. Zhengzi Xu, Xiaofei Xie, and Yang Liu. Gptscan: Detecting logic vulnerabilities in smart contracts by com- bining gpt with program analysis. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024.
Qingkai Shi, Xiao Xiao, Rongxin Wu, Jinguo Zhou, Gang Fan, and Charles Zhang. Pinpoint: Fast and precise sparse value flow analysis for million lines of code. In Jeffrey S. Foster and Dan Grossman, editors, Proceedings of the 39th ACM SIGPLAN Conference on Programming Lan- guage Design and Implementation, PLDI 2018, Philadel- phia, PA, USA, June 18-22, 2018, pages 693-706. ACM, 2018a. doi: 10. (0.1144/3192366.319224
Brian Zhang and Zhuo Zhang. Detecting bugs with sub- stantial monetary consequences by Ilm and rule- ule-based reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Bowen Zhang, Wei Chen, Peisen Yao, Chengpeng Wang, Wensheng Tang, and Charles Zhang. SIRO: empower- ing version compatibility in intermediate representations via program synthesis. In Rajiv Gupta, Nael B. Abu- Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2024, La Jolla, CA. USA, 27 April 2024- May 2024, pages 882- 899. ACM, 2024. doi: 10. 1111144//6660666665555555 36516366
Darko Anicic, Sebastian Rudolph, Paul Fodor, and Nenad Stojanovic. Real-time complex event recognition and reasoning-a logic programming approach. Applied Artifi- cial Intelligence, 26(1-2);6 (1,)):6-77,,012 2012.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii
Khizbullin, and Bernard Ghanem, Camel: Commu- nicative agents for" mind" exploration of large language model society. Advances in Neural Information Process- ing Systems, 36:51991-5200 12208,,,,,,,,,,,111111111919,, 2023.
Anshunkang Zhou, Chengfeng Ye, Heqing Huang, Yuandao Cai, and Charles Zhang. Plankton: Reconeiling binary code and debug information. In Rajiv Gupta, Nael B. Abu- Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- May 2024, pages 912- 928. ACM, 2024. doi: 10. 4445///666666533443333
Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou Genera- tive modti-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pages 18733-18741, 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174-15188 15186, 2024.
Brittany Johnson, Yoonki Song. Emerson R. Murphy- and Robert W. Bowdidge. Why don't software devel- opers use static analysis tools to find bugs? In David Notkin, Betty H. C. Cheng, and Klaus Pohl, editors, 35th International Conference on Software Engineering, ICSE '13, San Francisco, CA. USA, May 18-26, 2013, pages 672-681. IEEE Computer Society, 2013. doi: 10. 10.100//CSE.200
Ziyang Li, Aravind Machiry, Binghong Chen, Mayur Naik, Ke Wang. and Le Song. ARBITRAR: user-guided API misuse detection. In 42nd IEFE Symposium on Security

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Levd Code Auditing
and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2,21, pages 4400-1155. IEEE, 2021. doi: 10.1109/
Chengpeng Wang. Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: Analyzing dataflow in code with large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a
Jeanne Ferrante, Karl Ottenstein, and Joe D Warren. The program dependence graph and its use in optimization. In International Symposium on Programming, pages 125-
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. SatLM: Satisfiabilty-aiieed language models using declarative prompting. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Process- ing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
132. Springer, 1984.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08874, 2023. doi: 10448550/ARXX1223300.00777
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
Kihong Heo, Hakjoo Oh, and Kwangkeun Yi. Machine- soerinng-gidder selectively unsound static analysis. In 2017 IEEE/ACM 39th International Conference on Soft- ware Engineering (ICSE), pages 519-529. IEEE, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Jianjun Huang, Jianglei Nie, Yuanjun Gong, Wei You, Bin Liang, and Pan Bian. Raisin: Identifying rare sensi- tive functions for bug detection. In Proceedings of the IEEF/ACM 46th International Conference on Software Engineering, pages 1-12, 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Qingkai Shi, Peisen Yao, Rongxin Wu, and Charles Zhang. Path-sensitive sparse analysis without path conditions. In Proceedings of the 42nd ACM SIGPLAN International Conference Programminng Language Design and Im- plementation, pages 930-943, 2021.
Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Qingkai Shi, Xiao Xiao, Rongxin Wu, Jinguo Zhou, Gang Fan, and Charles Zhang. Pinpoint: Fast and precise sparse value flow analysis for million lines of code. In Pro- ceedings of the 39th ACM SIGPIAN Conference on Pro- gramming Language Design and Implementation, pages 693-706, 2018b.
Greg Kamradt. Needle in haystack pressure test- ing LLMs, 2023. URL ftttps://githh: com/ gkamradt WtweL//eeeeeeeeeeIeiaaaeae
Edmund M Clarke. Model checking. In Foundations of Software Technology and Theoretical Computer Science: 17th Conference Kharagpur. India, December 18-20, 1997 Proceedings 17, pages 54-56. Springer, 1997.
Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. Diversevul: A new vulner- able source code dataset for deep learning based vulner- ability detection. RAID '23, page 654-668, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400707650, doi: 10. 10.1145/360719993600
Yannis Smaragdakis, George Balatsouras, et al. Pointer analysis. Foundations and Trends@ in Programming Languages, 2(1):1169, 2015.
Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. En hancing static analysis for practical bug detection: An llm-integrated approach. Proc. ACM Program. Lang., 8 (00PSLA1):474 499, 2024a. doi: 10. 10.1145/3649828.
Leonardo Mendonça de Moura and Nikolaj Björner. Z3: an efficient SMT solver. In C. R. Ramakrishnan and Jakob Rehof, editors, Tools and Algorithmms for the Con- struction and Analysis of Systems, 14th International Conference, TACAS 2008, Held Part of the Joint Eu- ropean Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings, volume 4963 of Lecture Notes in Computer Science, pages 337-340. Springer, 2008. doi: 10. 1007/9978-3-540-8880 24.
Ziyang Li, Saikat Dutta, and Mayur Naik. LLM-assisted static analysis for detecting security vulnerabilities. CoRR, abs/2405. 2024b. doi: 10.48550/ARXIVV. 2405.17238.
Meta. Infer Static Analyzer. ht https:: /tttps:///oiffe..oo com/, 2025. [Online; accessed 29-Jan-2022 2025].
12

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Leve Code Auditing
Amazon. Code Review Tool: Amazon CodeGuru Se-
NPD
MLK
UAF
curity. https:/ /awss ama zon. com/ codeguru/,
25
2025. [Online; accessed 29-J5.-2025).
20
Mingwei Zheng, Danning Xie, and Xiangyu Zhang. Large language models for validating network proto- col parsers, 2025. URL wwtptttss///xiiii org// abs/ 13515.
Bug Number
15
13
10
6
6
5
Mingwei Zheng, Danning Xie, Qingkai Shi, Chengpeng Wang, and Xiangyu Zhang. Validating network protocol parsers with traceable rfc document interpretation. In Proceedings of the 34th ACM SIGSOFT International Symposium Software Testing and Analysis, ISSTA 2025.
Single-tunction
Muttiple-tunction
RepoAudit
Figure 6. The comparison results with single- fingle-function level and funetio- level bug detectors
bug detection is widely adopted and evaluated by many re- cent studies (Chen et al., 2023; Ding et al., 2024). These techniques are applicable for models with a limited context length. To compare with single-function level bug detection, we collect all the functions that contain the sink values yield- ing TPs to the LLM along with few-shot examples, asking the LLM to determine whether the function can introduce specific types of bugs. Second, multiple-functinn function level bug detectors attempt to feed the whole program to the LLM so that the calling contexts of buggy functions can be in- cluded (Wang et al., 2024c). Unfortunately, the huge size of a real-world software system often makes the prompts exceed the context limit of LLM. Hence, we only feed the relevant functions covered by the buggy program paths to the LLM in our evaluation.
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. A CIC++ code vulnerability dataset with code changes and eve summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 508- 512. 2020.
Niklas Risse and Marcel Böhme. Top score on the wrong exam: On benchmarking in machine learning for vul- nerability detection. arXiv preprint 20t:24008.12886 2024.
Chengpeng Wang, Jipeng Zhang, Rongxin Wu, and Charles Zhang. DAInfer: Inferring API aliasing specifications from library documentation via neurosymbolic optimiza- tion. Proc. ACM Softw. Eng., 1(FSE)) 2024b.
Among agent-centric approaches (Wang et al., 2024a; Li et al., 2024a;b), we select LLMDFA (Wang et al., 2024a) as a baseline for comparison, as it supports compilation -reee and customizable analysis. Since LLMDFA cannot support the MLK detection, we focus our comparison between RE- POAUDIT and LLMDFA specifically on NPD and UAF bugs. LLMDFA works by summarizing all data-flow facts for each function and then correlating these facts. Its overall compu- tational costs time and token costs can be substantial. To avoid the excessive computation costs, we conduct group of controlled experiments under two settings. In the first setting, LLMDFA only applied to functions covered by buggy program paths, generating data-flow summaries for these functions. In the second setting. LLMDFA is tasked with generating data-flow summaries for functions reach- able from each source value. We refer to these settings as respec- LLLDFA- SADDFA-PATHSAN and LLMDDA--SRCSAA
doi: 10.1145/3660816.
Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. QL: object-oriented queries on relational data. In Shriram Krishnamurthi and Benjamin S. Lerner, editors, 30th European Conference on Object- Orieented Programming, ECOOP 2016, July 18-22, 2016, Rome, Italy, volume 56 of LIPlcs, pages 2:1-2:25. Schloss Dagstuhl- - Leibniz-Zentrum für Informatik, 2016. doi: 1142200..
Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, and Xiangyu Zhang. Sanitizing large language models in bug detection with data-flow. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pages 3790-3805, 2024c.
tively. Notably, the number of prompting rounds and com- putational costs of LLMMFA--ATHHCAA PATHŠCAN and LLMDFA- SRCSCAN are lower than those of LLMDFA, as the latter two only reason subset of the functions in the repository.
A. Comparison with LLM-driven Detectors
Setup and Metrics. To the best of our knowledge, there are two kinds of LLM--riven code auditing techniques, namely end-to-end few-shot chain-of-thoughe (CoT) prompting- based approaches and agent-centric approaches. Specif- ically, the former can be divided into two categories, in- cluding single-functien level bug detection and multiple- function level bug detection. First, the fingle-function level
Result. Figure 6 shows the comparison results between the single-function level bug detector, multiple-function level bug detector, and REPOAUDIT. The single-function level bug detector can detect only one intra procedural NPD bug. The key reasons are twofold. First, it only accesses the
13

---

REPOAUDIT: An Autonomous LLM-Agent for Eepository-Leve Code Auditing
Listing 1. An example of control flow facts ignored by single-function level hug detectors
1. static int inft samon_reclatee reclaannnnnee (v
ctx damon_new_etxx1y
3. if (!etx)
return -ENOMER;
5. if ccamon_sellett_opstle
SAMOK_OFS_PADDDS
6. return -EINVAL;
7.
Baackk éamon_ samon_rosllinnnffee secarrssscheete
8.
otx->oal http:laaaac...ttaaggreeeeo
damon_reo gomn_roosiin_attrr.oggggggg
9. Larget demps__ew__Laggel()>
10. LE (Iterget)
11.
samro__dsttray_ctxxcttt
12.
return -FNOMRM;
13.
14.
hamon_add__a bargett (atx, target);
15.
Senedule_delayeee looolllyeddddd work
0);
damon_ inteelaaannnitsaiiiiio initislized trues
17.
return 0j
18.
rounds for SLLDDA--PATISCA is 165 times that of RE- POAUDIT, and the input token count is 123 times higher. Similarly, for LLMDFA-SRCSCAM the number of prompt- ing rounds is 1,737 times that of REPOAUDIT, with an input token count 1,193 times higher. is important to note that the actual cost of LLMDFA would be even greater in practical scenarios. This significant performance gap stems from the fact that LLMDFA follows a desgin similar to compiler-base based scanners that first collect all primitive data-flow facts and then correlate them to find bugs. For example, to detect the NPD bugs in the example program shown in Figure 1, LLMDFA begins with the NULL value at line 4 in the function field22 json, analyzing the de- pendencies of it with all the arguments, return value, and dereference pointers (e.g.. field at line 8) within this func- tion. Hence, LLMDFA fails to scale large-size projects in the woaldworld senarios.
Table The ratios of the prompting rounds (Prompt) and input token costs (In_token) of LLMDFA under two settings
LLMDFA- PATHSSAN
LLMDFA
-SCCAAN
Bug Type
ID
Prompt
Tokenen
Prompt
In. Token
N1
115.39
81.23
871.61
522.40
N2
217.00
120.23
2,839.00
2,087.27
NPD
N3
6.83
3.13
255.57
224.82
N4
115.76
74,91
778.17
715.23
N5
199.43
125.53
7.121.86
5,350.62
U1
41.89
23.08
4,536.75
2,530,74
U2
24.00
12.29
50,00
26.26
CAF
U3
759.48
693.73
17.63
6.17
U4
167.50
95.58
46.60
25.42
U5
4.98
2,09
853.20
447.36
Average
165.23
123.18
1737.04
1193.63
last function in the buggy path and lacks the calling con- text of the function. In NPD detection, for example, the single-function level detector cannot determine whether the parameter is null or not, which makes it fail to detect inter- procedural bugs. Second, the model may overlook essential control flows within function, which results in its inabil- ity to accurately identify the data-flow facts of specific value, thereby achieving low recall in intra-proceed procedural bug detection. For instance, as shown by Listing 1, the I.I.M ignores the error handling branch at line 6, which causes the function to return without releasing the memory object allocated by the function gesmon_new_tt__ (), leading to its failure detect this memory leak bug. With access the level bug detector is able to detect 10 bugs. This demonstrates that providing additional calling context can improve the bug detection capabilities of the LLM. However, the improvement is still limited. Due to hallucinations, the LLM may still wrongly analyze certain inoraurroc and inter-procedural data- flow facts, resulting in significant number of FNs. In contrast, REPOAUDIT performs the path-sensitive reason- ing by employing the program abstraction, which facilitates precisely discovering data-flow facts for code auditing.
B. Comparison with Industrial Bug Detectors
Setup and Metrics. We choose two typical static bug de- tectors as the representatives of industrial tools, namely Meta INFER (Meta, 2025) and Amazon CODEGURU (Ama- zon, 2025). Specifically, Meta INFER is static analysis tool from Meta. Benefiting from its sophisticated mem- ory model (Calcagno et al., 2009), Meta INFER features its outstanding ability in memory bug detection. All three bug types in our evaluation are supported by Meta INFER. Notably, Meta INFER is only applicable to projects that can be successfully compiled. Amazon CODEGURU, as an AWS service, combines machine learning and automated reasoning to identify underlying bugs. Unlike Meta INFER, Amazon CODEGURU can directly analyze source code with- out compilation. As Amazon CODEGURU does not support MLK detection, we only evaluate it for the NPD and UAF detection. After running the two industrial bug detectors, we manually check the bug reports, label the TPs and FPs, and compare with the results by REPOAUDIT shown by Table 2 in Section 4.2.
full calling context, the
Table shows the comparison results between LLMDFA and REPOAUDIT. On average, the number of prompting
14

---

REPOAUDIT: An Autonomous LLM-Agent for Repository- Leeeel Code Auditing
Result. Table 3 presents the results of Meta INFER and Amazon CODEGURU. As shown by the column Build in Table 3, 13 projects can be successfully compiled However, five of them still cause crashes in Meta INFER, namely the projects with the IDs N3, M3, M5, U1, and U2, even after we have tried multiple versions of Infer, including v1.2.0 (latest), v1.0.0, and v0.9.0. We have reported these issues to Meta INFER, but haven't 't received any response yet. The failures of compiling and analyzing successfully compiled projects demonstrate the restricted applicability and insta- bility of compilation-dependen bug detectors. Eventually, we successfully analyze eight projects with Meta INFER, obtaining a total of seven TPs and two FPs. Notably, for project bibf rrenect. with the project ID N4, the five TPs generated by Infer are based on the assumption that external APIS realloc and mall loo can return null point- upon allocation failure. However, this assumption does not always hold. In our implementation, REPOAUDIT does not rely on such assumptions and instead begins its analysis with null literal values or other user-defined APIs. Lastly, although Meta INFER is known for its powerful memory model, it still fails to detect any MLK or UAF bugs discov- ered by REPOAUDIT. In comparison, REPOAUDIT not only supports non-compilation analysis with greater applicability but also shows stronger detection capabilities, identifying 40 TPs in total.
Table 6. The statisties of HEPOAUDIT-NOAHE and REPOAUDIT- NOVAL
REPOAUDIT-NOABS
REPOAUDIT-NOVAL
Bug Type
ID
TP
TP
FP
FP
Old
New
Old
New
N1
(3,3)
(3,3)
2
N2
(0,0)
5
(1,0)
0
NPD
N3
0
(0.0)
6
(1.0)
10
N4
0
(0,0)
0
(0,0)
5
N5
(2.2)
2
(5.4)
4
M1
(1.1)
2
(2.1)
3
M2
(1.1)
5
(6.6)
5
MLK
M3
(0,0)
0
(0,0)
0
M4
(0.0)
0
(0.0)
1
M5
0
(0,0)
0
(0.0)
2
U1
(0,0)
0
(0.0)
0
U2
(0.0)
0
(0,0)
0
CAF
U3
(0.0)
1
(0.0)
0
U4
(0.0)
0
(0.0)
0
U5
0
(1.0)
6
(1,0)
6
Total
13
8
31
21
19
38
Table 7. The computational costs of REPOAUDIT-NOCACHEE OOT: the out-of-time, indicating that REPOAUDIT-NOCACHE does not finish the code auditing in 72 hours.
Prompls
Financial($)
Time(s)
Bug Type
ID
Ratio
Num
Ratio
Num
Ratio
436
8.65
6,316.37
3.12
N2
18
1.06
0.43
1.03
312.01
1.10
NPD
N3
175
1.61
4.12
1.59
3,098.59
1.77
N4
181
6.24
3.06
5.19
2,342.23
5.38
As shown by the last two columns in Table 3, Amazon CODEGURU does not detect any TPs upon the targeted 10 projects, while generating 18 FPs. Due to the limitations of its inherent formal reasoning techniques and machine learn- ing models, Amazon CODEGURU can only detect certain patterns of bugs and is not robust to the various ways of writing similar buggy code.
N5
137
2.17
4.09
2.37
2,626.21
2.48
M1
459
2.24
10.49
2.24
7,070.68
2.42
M2
S84
4.00
15.83
4.40
10,852.80
4.76
MLK
M3
1.00
0.05
0.95
34.34
1.00
M4
1.00
0.03
0.87
20.13
1.12
M5
200
5.71
4.88
5.67
3,635.66
6.06
657
18.25
14.34
17.93
11,528.96
19,80
U2
1.00
0.04
1.00
32.78
1.03
UAF
U3
1.40
2.10
1.62
1,153.46
1.46
U4
10
1,00
0.23
0.99
174.43
0.94
US
N/A
N/A
N/A
N/A
OOT
N/A
C. Ablation Study
Average
209.21
3.55
4.88
3.48
3,514.19
3.75
Setup and Metrics. To evaluate the effectiveness of each technical design, we introduce three ablation variants of REPOAUDIT, namely REPOAUDIT- NoABS, REPOAUDIT- NOVAL, and REPOAUDIT-NO CEPOAAUDI-NNCACHE. NOCACHE. Specifically, RE- POAUDIT-NOABS skips the program abstraction, i.e., re- moving the second step in the prompt template shown in Figure 4. REPOAUDIT-NOVAL removes the validation of the data-flow facts discovered by the explorer and also skips dis-
tion paths created by numerous conditional branches, loop structures, and their nested combinations within functions. When analyzing such cases, the model becomes more sus- ceptible to hallucinations, resulting in missed buggy propa- gation paths incorrect identification of non-existent ones.
When the validators are disabled, the number of FPs in- creases to 31, causing a 245.45% increase. This rise is mainly caused by the presence of various conditional branches and jump statements, such as if-else and switch, as well as early exits in certain branches (e.g.. error handling). Without the validator, the LLM may hal- lucinate and fail to account for these critical branches, re- sulting in a large number of spurious data-flow facts that do not meet feasibility requirements. Besides, the conflicts between branch conditions discovered by the LLM can also contribute to the high precision.
examining the bug reports. REPOAUDIT-NOCAAAE ables the caching strategy upon the agent memory when the explorer analyzes individual functions.
Result. Table 6 presents the results of REPOAUDIT-NOVAL and REPOAUDIT-NOABE Without the program abstrac- tion, REPOAUDIT NOABS decreases the number of TPs by 47.50%, while increasing the number of FPs by 181.82%, leading precision degradation 40.38%. This decline is attributed to the complex control flows and multiple execu-
15

---

REPOAUDIT: An Autonomous LLM-Agent for Repository- Code Auditing
Table The statistics of REPOAUDIT powered DeepScek R1
TP
# Tokens
Bug Type
ID
FP
Prompts
Financial (S)
Time (s)
Old
New
Input
Output
NI
(3,3)
2
136
675,711
46,839
1.20
12,378.10
N2
7
(2.0)
0
14
74,918
6,057
0.14
1,588.65
NPD
N3
(1.0)
524,915
38,644
0.95
9,044.85
N4
(0.0)
29,898
2,288
247.99
N5
(7.6)
0
43
270,880
15,689
0.47
2,936.52
M1
(2.2)
2
165
910,153
55,623
1.58
14,633.28 28
M2
(9,9)
93
562,470
35,498
0.98
5,583.13
MIK
M3
(0.0)
10,481
832
74.01
M4
(0.0)
214
93.23
M5
(0.0)
0
18
92,573
8,289
0.18
1,572.31
UI
(0,0)
0
50
255,048
19,494
0.46
2,444,72
U2
(0.0)
0
8900
704
0.02
88.76
UAF
U3
(0,0)
29
227,336
10,269
0.37
2,768.59
U4
(0.0)
53,647
3,617
0.09
1,997.20
US
(1.0)
1,191,391
78,226
13,124.44
Average
59.60
326,267.47
21,485.53
0.57
4,571.72
Table 9, The statistics of REPOAUDIT powered by Claude 3.7 Sonnet
TP
Tokens
Bug Type
ID
FP
Prompts
Financial (S)
Time (s)
New
Input
Output
N1
(3,3)
2
371,008
58,822
2.00
1,650.53
N2
7
(1,0)
15
98,473
9,015
0.43
264.68
NPD
N3
1
(1,0)
0
74
480,786
78.725
2.62
2,076.32
N4
(0,0)
15
62,634
8,822
0.32
233.96
NS
(7,6)
213,483
21,107
0.96
5y2.74
(2.2)
215
1195594
118735
5.37
3,510.25
(10,10)
537290
61472
2.53
1,842.90
MLK
M3
(0.0)
10548
1097
0.05
46.25
M4
(0.0)
5741
1024
0.03
32.86
M5
(0.0)
16
81,970
9,442
0.39
402.17
UI
(0.0)
18
93466
171.9
0.54
462.51
U2
(0.0)
8870
1145
0.04
117.50
UAF
U3
(0.0)
605597
70,666
1,936.22
(0.0)
10
0.29
314.60
US
(1.0)
207
1069189
147422
5.42
4,387.80
Average
57.07
325,523.67
40,965.06
1.59
1,19142
The results of REPOAUDIT-NOCACHO are presented in Ta- ble 7. For the project i cu, the analysis time exceeds hours and the number of prompting rounds exceeds 20,000, more than 30 times that of REPOAUDIT. For the remaining projects, average, the number of prompting rounds in- creases by 3.55 times, the financial cost by 3.48 times, and the analysis time by 3.75 times compared to REPOAUDIT. Moreover, the cache hit rate is closely related to the project's size and the density of underlying graphs. In projects with intricate call graphs and dense DDGs, single function can appear in multiple propagation paths of faulty values. For example, in the icu project, parsing-related functions are hit in the cache 623 times. In such cases, the caching strat- egy effectively reduces analysis costs and keeps the analysis time within an acceptable range.
Claude 3.7 Sonnet, and OpenAl o3-mini separately. In gen- cral, when integrated with reasoning models, REPOAUDIT demonstrates stronger bug detection capabilities than the one powered by Claude 3.5 Sonnet. It is able to identify all known bugs reported by existing works and further discover additional previously unreported bugs with higher precision. This better performance is largely attributed to the reasoning model's ability to autonomously construct logical reasoning pathways, thus facilitating more precise reasoning control-flow and data-flow facts of a program.
As shown in Table 8. REPOAUDIT powered by DeepSeek R1 successfully identifies 46 true positives, achieving precision of 88.46%, the highest among all evaluated models. This indicates superior capability of DeepSeek RI analyzing program control-flow and data- flow. In terms cost efficiency. DeepSeek RI incurs an average cost of $0.57 per project. However, it is also the slowest model, with an average analysis time of 4,571 sec- onds per project, which may be attributed to factors such as network latency and limited throughput.
D. Evaluation with More Reasoning Models
Table 8. Table 9, and Table 10 show the results of REPOAUDIT with reasoning models DeepSeek R1,
16

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Leve Code Auditing
Table 10. The statistics of REPOAUDIT powered by OpenAl 03-mini
TP
Tokens
Bug Type
ID
FP
Prompts
Financial (S)
Time (s)
Old
New
Input
Output
NI
(3,3)
2
49
237,382
26,360
0.38
782.75
N2
7
(1,0)
0
17
104,462
10,286
0.16
321.19
NPD
N3
(0.0)
0
16
86,930
8,724
0.13
261.61
N4
(0,0)
2
17
80,877
8,169
0.12
299.13
NS
(6.5)
24
170,504
13,115
0.25
598.2588062
MI
(2,2)
196898
18937
0.30
609.81
(7.7)
303005
24773
1,005.86
(0.0)
1134
(0.0)
5691
36.87
(0.0)
66601
7672
0.11
330.96
UI
(0.0)
0
24
128434
12641
0.20
266.63
U2
(0,0)
0
2
9422
958
0.01
24.08
UAF
U3
(1,0)
83
634100
48,528
0.91
1,176.46
U4
(0.0)
10
50966
4044
0.07
190.54
U5
(1,0)
178
976191
82933
1.44
3,427.38
Average
34.73
204.129.60
17,939.00
0.30
625.05
As shown in Table 9, REPOAUDIT powered by Claude 3.7 Sonnet also detects 46 true positives, yielding the precision of 86.79% which marginally below that of DeepSeek RI. Given the inherent randomness, we consider Claude 3.7 Sonnet Somnet's program analysis capabilities to be comparable those of DeepSeek R1. In terms of prompt- ing rounds and token usage, Claude 3.7 Sonnet induces sim- ilar prompting rounds and input token numbers per project as DeepSeek RI. However, its output token count is 1.9 times higher. Based on the pricing policy,Claude 3.7 Sonnet the most expensive model, resulting in the highest average financial cost of $1.59 per project.
Table 11. The statistics of REPOAUDIT with different temperature
settings. The column Reproduce indicates the number of
reproduced bugs that previously reported by existing works.
Temperature
TP
FP
Reproduce
Precision (%)
Recall (%)
0
40
11
21
78.43
100
0,25
38
12
21
76.00
100
0.5
38
12
20
76.00
95.24
0.75
33
11
18
75.00
85.71
1.0
35
13
19
72.92
90.48
temperatures, we evaluate REPOAUDIT under four addi- tional temperature settings: 0.25, 0.5, 0.75, and 1.0. For each setting, record the number of true positives and false positives detected by REPOAUDIT, and compute the corresponding precision. Also, we measure the recall by investigating the proportion of reproduced bugs.
As shown in Table 10, REPOAUDIT powered by OpenAl o3-mini demonstrates slightly lower code analy- sis performance, detecting 42 true positives with a precision of 82.35%. This may be due to its relatively weaker ability to capture complex code semantics, particularly in modeling data-flow, potentially resulting in both false positives and missed bugs. In terms of efficiency, OpenAl o3-mini issues significantly fewer queries per project compared to the other two models, which may also contribute to its narrower func- tion coverage and oversight of potential data-flows. Besides, OpenAl o3-mini incurs the lowest cost per query and of- fers the fastest analysis speed. On average, REPOAUDIT powered by OpenAl o3-mini results in the financial cost of S0.30 and the time cost of 624.05 seconds per project.
Results. As shown in Table 11, REPOAUDIT demonstrates stable performance across varying temperature levels, with precision remaining above 72% and recall consistently high. However, as the temperature increases to 1.0, both precision and recall decline, suggesting that greater randomness in the model's output can lead incorrect reasoning steps, ultimately resulting in increased number of false positives and false negatives.
F. Examples of False Positive/Negative
The above statistics demonstrate that REPOAUDIT lessly benefit from advances in LLM capabilities, leading to improved precision and recall in code auditing. Also, en- hancements in LLM inference efficiency can further boost the overall performance of the auditing process.
We present representative a false positive and false neg- ative REPOAUDIT as follows. These cases can reflect several limitations of our approach, including the model's tendency to hallucinate during complex control-flow reason- ing and its insufficient understanding of implicit semantic constraints.
E. Evaluation with Different Temperatures
In Listing 2, the function vrf_get can return NULL when name=NULL and vrf _ddvRF UNKNOWN. In the func- tion 1ib 1ib_vvrri vrf wrf_create, the return value of f_e_t
Setup and Metrics. In evaluation, set the perature to 0.0 by default. To assess the impact of the
17

---

REPOAUDIT: An Autonomous LLM-Agent for Repository-Levd Code Auditing
Listing 2.
An example false positive reported REPOAUDIT due unawareness the YANG schema constraints. struct vz evif_get (vif_id_t vrš_id, const char +name)
struct vrf. Avrf NUT. Nothing to see, move along here (!name &6 wrf_id VRF UNKNONN)
return NULL;
8
statio int iib_vet_creste on_ch_oreate_agg wargs)
svriname;
struct vel Avrīp;
vefneme yang stt__dddde__see_tteeeeeee (arge-Pddded "name");
6 if (args->evant I= NB NB_EV_AFPL2)
return NR_OK;
vEIp vri_get (VRE UNKNONN,, vrfname); (eTFFFPAtrrff-ptatttta VRF CFP_CONFIIG30 IGURED);
11
Listing An example a false negative reported REPOAUDIT results from Claude 3.5 Sonnet overlooking error-handling path.
strt Sass_Compl lerv AnDCALT, htt____aa_aa__ooppppppppe ler Istruet hata_Context: (data_ctx -- 0) zetuze 0;
data_ctx]
Context+ app_ctx
Deta_Context (+data_ctx)
return rars_prepare_ccneext (dzta_ctx, cpp_etx)
statie Sass Lass_Compilest prep_rre context @aas__Contexe c_etx, Context+ app_stx) throw(1 volds ctxmem (1, sizeof (struct mes__ooppler)) if (ctxmem 0)
std:icerr << "Error allocating memory for context" std:rendl; return 0;
Sasa_ Compiler compiler = (struct Compller)) ctxmem; complle ler-lolr-oooo otx c_etx;
return compiler;
10
assigned to the pointer vrfp, which is subsequently deref- erenced without null check, seemingly leading to po- (NPD) bug. REPOAUDIT reports an NPD bug at line where vrfp->status is accessed without checking whether vrfp is NULL. How- ever, this issue is false positive. Due to YANG schema validation, the vrfname variable is guaranteed to be non- NULL. Given that vrf_get only returns NULL when both name=NULL and vrf_ htt_:d===VE_UUUUU,,,,,, cannot re- turn NULL when vrfname -NUULL. Hence, the derefer- ence is safe in practice. The root cause of the false positive that the LLMs are not aware of the fact that the return value of yang_dnodee stt___iddd_____________eeeeeeeee
sue as it did not accurately track all relevant execution paths, particularly the error-handling path in this case. In contrast, reasoning-oriented models like Deepseek demonstrated superior capability recognizing execution paths precisely, allowing REPOAUDIT detect such memory management issues more effectively.
tential
is never NULL.
In Listing 3, memory object is allocated by the function sass gass_make_ddee latp__daa_aee cantt__comilees ler and passed as the sec- ond argument to the function sass wwwppppeeeppaaaee context. Within sass paepparpeprree__e pzepare_connext,, cooteet, if calloc fails at line 3, ctxmem is set O, and the function returns O without freeing the allocated memory object opp_ctx memory leak. This bug was detected by Deepseek R1 but missed by Claude 3.5 Sonnet. The latter model failed to identify the is-
assigning to any other pointer, leading
18


Reviews:
[{"review_id": "PnnqmMe67a", "reviewer": "Reviewer_EkmU", "summary": "The paper presents RepoAudit, an autonomous LLM-agent designed for repository-level code auditing. RepoAudit leverages large language models to find critical bugs such as null pointer dereference, memory leak, and use-after-free. The agent efficiently scans code repositories by utilizing an agent memory system that enables on-demand exploration of relevant functions. It mimics human code auditing by focusing on path-sensitive reasoning, making it scalable and adaptable to complex repositories. By addressing the challenges of context limits and hallucinations, the system provides significant improvements over previous methods, detecting 38 true bugs in 15 real-world systems with an average cost of $2.54 per project.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Effectiveness in mitigating LLM's intrinsic limitations**\n\nApart from the case studies in Sections 2.2 and 2.3, we evaluated RepoAudit-NoAbs that excludes program abstraction and pointer handling. The column **RepoAudit-NoAbs** in Table 5 demonstrates that this ablation decreases the number of TPs by 44.74% and increases FPs by 105%, causing precision to drop to 33.87%. Further details are available in Appendix B.3.\n\nAfter the paper submission, we scanned 10 more GitHub projects, which averagely have 251K LoC and 8.8K stars, indicating their high complexity and popularity. In total, RepoAudit detected 186 true bugs with the precision of 85.71%. 80 and 95 bugs have been fixed and confirmed, respectively. We will add such results to the revision. \n\nThe above results strongly indicate that RepoAudit effectively mitigates the intrinsic limitations of LLMs in our context. We will change our claim from \"addressing\" to “mitigating” as we agree that there is still substantial room to improve.\n\n\n**2.Performance of analyzing more complex codebases**\n\nIn our additional evaluation, the projects are 1.67 times the size of those in our original evaluation. Despite the increased size, RepoAudit maintained high precision (85.71%) and completed the analysis of each repository in 0.82 hours on average. This is 1.86 times the time cost of analyzing the original benchmarks. Hence, time costs are nearly linear with project sizes, showing the graceful scalability of RepoAudit.\n\n**3.Comparison with Meta Infer**\n\nOur work aims to enhance the applicability and usability of code auditing. Thus, our evaluation against Meta Infer was carefully designed to fairly reflect real-world contexts:\n\n- Given our motivation, the applicability of code auditing tools is a critical focus. In practical scenarios, build errors and compatibility issues directly affect deployment feasibility. Therefore, beyond evaluating precision, recall, and efficiency, assessing a tool's practical applicability is crucial.  \n\n- The continuous evolution and variety of compilers and build systems render build-dependent tools like Meta Infer vulnerable to fundamental deployment issues—not merely integration or implementation challenges. Surveys by industry leaders highlight these fundamental obstacles [R1, R2], significantly hindering the broader deployment of the tools.\n\nIn our revision, we will justify the comparison setting and explain the real-world impact of build failures and incompatibilities.\n\n> [R1] Why don't software developers use static analysis tools to find bugs? ICSE 2013\n>\n> [R2] Christakis M, Bird C. What developers want and need from program analysis: an empirical study. ASE 2016\n\n**4.Performance with alternative LLMs**\n\nPlease refer to the response to the first concern **Model Choice** of [Reviewer Le35](https://openreview.net/forum?id=TXcifVbFpG&noteId=5JaOhTUECl).\n\n**5.Performance using different temperatures**\n\nExisting studies [R3, R4] on reasoning tasks suggest setting the temperature parameter to 0, thus reducing randomness in outcomes. Our work followed this practice. We will justify the setting in our revision.\n\nDuring the author response stage, we further evaluated RepoAudit using Claude-3.5-Sonet under different temperatures. The results are as follows.\n\n| Temp | #TP | #FP | #Reproduced | Precision (%) | Recall (%) |\n|-------------|-----|-----|--------------|----------------|-------------|\n| 0           | 38  | 20  | 21           | 65.52          | 100.00      |\n| 0.25        | 33  | 20  | 20           | 62.26          | 95.24       |\n| 0.5         | 36  | 23  | 21           | 61.02          | 100.00      |\n| 0.75        | 35  | 20  | 18           | 63.64          | 85.71       |\n| 1.0         | 33  | 24  | 18           | 57.89          | 85.71       |\n\nRepoAudit remains robust across a range of temperature settings, with precision fluctuating slightly but remaining above 57%, and recall consistently high. Here, the precision is computed by #TP/(#TP + #FP), while the recall shows the proportion of reproduced bugs. We will incorporate these detailed findings and analyses into the revision.\n\n> [R3] Satlm: Satisfiability-aided language models using declarative prompting, NeurIPS 2023\n>\n> [R4] SWE-bench: Can Language Models Resolve Real-world Github Issues? ICLR 2024\n\n**6.Manual validation of false positives**\n\nAs shown by a large body of existing literature [R5], static code auditing inevitably produces false positives when detecting vulnerabilities in real-world scenarios. Manual post-validation thus remains a common practice. Notably, our evaluation shows RepoAudit surpasses the precision of SOTA industrial tools, indicating a reduction in required manual verification efforts compared to existing solutions. We will include the discussion regarding manual validation and its implications for using RepoAudit in our revision.\n\n> [R5] Mitigating false positive static analysis warnings: Progress, challenges, and opportunities, TSE 2023", "claims_and_evidence": "1. The paper states that REPOAUDIT overcomes intrinsic LLM limitations. Controlled experiments in Sections 2.2 and 2.3 provide qualitative evidence, however, the examples are narrowly scoped. It is not entirely convincing that these mitigations will hold up consistently across more complex codebases.\n\n2. Fairness of performance claim is unclear; specifically, the issues with INFER (such as build errors or incompatibilities) might reflect implementation or integration challenges rather than fundamental performance differences in code auditing.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are largely well-suited to the problem of repository-level code auditing.", "theoretical_claims": "The paper relies on controlled experiments and qualitative demonstrations, such as the examples of pointer handling and feasible program path exploration in Figure 2, to substantiate their assertions about the limitations of LLMs and the benefits of their approach.", "experimental_designs_or_analyses": "1. The reported failures of tools like Meta INFER may, in part, stem from integration or configuration issues rather than intrinsic limitations. This could bias the comparative results.\n\n2. The experiments rely on a specific configuration of Claude 3.5 Sonnet (with a fixed temperature of 0.0). As LLM behavior can be sensitive to prompting parameters, it remains to be seen how robust the approach is under different settings or with other models.", "relation_to_broader_scientific_literature": "REPOAUDIT advances code auditing by combining established static/pointer analysis and LLM-based code understanding. This hybrid approach improves scalability and precision for repository-level analysis, addressing long-standing limitations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\n1. RepoAudit detects bugs with high precision (65.52%) and uncovers new bugs in addition to those reported by existing systems.\n2. The agent’s ability to reason across different program paths, avoiding irrelevant ones, makes it effective in finding bugs like NPD, MLK, and UAF.\n\n### Weaknesses\n\n1. While the system mitigates hallucinations through validation, it still produces some false positives that need manual validation.\n2. Though scalable, the system’s approach to managing large repositories could face performance issues with significantly larger systems or those with complex interdependencies across functions.\n3. Ablation study on different prompt parameters and different LLMs.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q0G4tEGIdq", "reviewer": "Reviewer_yKxy", "summary": "This paper attempts to address a key concern in LLM-based code auditing systems where repositories are too complex and big to be effectively audited by LLMs. To address this, RepoAudit explores the repository on demand by analyzing data flow relations between different sections of the repository to build a more focused context for finding bugs. It also includes a validator to check satisfiability and reduce hallucination. Overall, it shows better performance than industry-standard auditing software.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Customization and Expert Knowledge**\n\nWhat we meant in the introduction section was that in order to detect a new type of bug, a new tool often needs to be developed inside some compiler, implementing bug-specific code checking rules. This often requires substantial compiler and program analysis expertise. With our design, extending RepoAudit to a new bug type may only entail providing a textual definition of the bug and few-shot examples (please also see our response to question 4 below). That said, the current implementation of RepoAudit focuses on bugs that are caused by data-flow, which covers a wide spectrum of bug types, such as SQL injection, Cross-site Scripting, and Absolute Path Traversal. Supporting bugs that do not entail data-flow, such as concurrency bugs, requires further development. We will clarify this in the revision.\n\n**2.More Case Studies and Insights**\n\nPlease refer to the response to the second concern **Case Studies of FPs/FNs of RepoAudit** of [Reviewer HHzn](https://openreview.net/forum?id=TXcifVbFpG&noteId=enl0Gy3OnD).\n\n**3.Comparison with CodeQL**\n\nInitially, we considered using CodeQL as a baseline in our evaluation. However, upon investigation, we found that CodeQL lacks built-in detectors for the three bug types targeted by RepoAudit (i.e., NPD, MLK, and UAF). Therefore, we selected Meta Infer and Amazon CodeGuru as baselines, as both provide relevant built-in analyses for the targeted memory-related bugs. We will include more justification for our baseline selection criteria in the revision.\n\n**4.Migration to Other Languages and Bug Types**\n\nOur evaluation currently focuses on C/C++ programs, but RepoAudit actually supports three additional languages: Java, Python, and Go. Supporting a new language typically requires writing a few hundred lines of Python code to implement a few primitives needed by the auditing engine, such as the caller/callee retrievers and specific program value extractors using the *tree-sitter* parsing library. Although multilingual support is not our paper’s primary contribution, we will include additional discussion on this aspect in the revised evaluation section.\n\nAs we stated in our response to the first concern, RepoAudit supports various types of bugs—including those not associated with well-known CWE categories, such as domain-specific bug types. For example, in financial systems, there are often custom security policies such as *\"user-sensitive data must not be logged in system outputs.\"* RepoAudit can handle such cases as long as the users define the form of sensitive data and logging operations as source and sink values, respectively, via several few-shot examples along with natural language descriptions. Moreover, this customization process is lightweight and incurs little manual effort. In our experiments, each bug type can be configured using no more than five few-shot examples, and the total number of words needed in the natural language prompt does not exceed 50 words.\n\nWe sincerely thank the reviewer for offering valuable suggestions. We will include a more detailed discussion on the migration to other languages and bug types in the revision.", "claims_and_evidence": "I believe the claims are clear and the evidence supports most of them. However in the introduction, one main limitation of existing systems is that using them to find new bugs requires a lot of expertise. The authors do not, to my knowledge, address this limitation in this paper. It would be nice to have a discussion about the effort required to use it in a custom repository to find bugs that may not be well known CWEs.", "methods_and_evaluation_criteria": "The paper presents a good evaluation, but I would have liked to see some more insights in general, especially with the error analysis of any false positives or negatives. For instance, were there any FPs/FNs that RepoAudit missed but were captured by the other baselines? Are there any common characteristics of the bugs missed by the tool? Despite the attempts to validate the results were there any cases which slipped through?", "theoretical_claims": "Didn't verify.", "experimental_designs_or_analyses": "I didn't verify the experiment results, but I believe the design is sound and the analysis makes sense. I would have appreciated more insights (mentioned earlier).\n\nIs there a reason you did not include CodeQL as one of the baselines? It is not necessary, and suffers from much of the same issues as Infer, but it is quite popular, hence the question.", "relation_to_broader_scientific_literature": "The field of bug-finding and code auditing is very actively researched, so I believe this paper is relevant to the broader scientific literature. Summarizing large data to fit the context of LLMs is a significant part of this, and using dataflow analyses to do so within repository-level code makes sense.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "I like the writing of the paper as well as the ideas presented. I personally don't see any other major weaknesses, apart from a lack of discussion on the effort required to reuse the same system for repositories written in different languages or for finding other kinds of bugs. Just a paragraph on generalizing across bugs, or coding languages (e.g. what would I need to do to use RepoAudit to find null pointer dereferences on a repository written in a different language). Actually I don't believe I saw any description of the programming language of the repositories being evaluated, so adding that is important.", "other_comments_or_suggestions": "1. Please add the number of FPs for RepoAudit in the intro, right now you only mention those for CodeGuru and Infer.\n2. In section 2.2, I am unclear on what the hallucinations exhibited by the model are, and it is a bit tough to follow the explanation in Figure 2.", "questions_for_authors": "see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag6Cpcf4WM", "reviewer": "Reviewer_Le35", "summary": "This paper proposes RepoAudit, an autonomous LLM-powered code auditing framework that can compete with current academic and industry solution tools. It consists of an initiator, an explorer, and a validator, working together to efficiently analyze GitHub repositories for code quality, security vulnerabilities, and compliance issues. Based on extensive experiments, it significantly addresses existing challenges, including the inability to analyze large repositories, high computational costs, excessive false positives, and the inefficiency of traditional static analysis methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Model Choice**\n\nWe evaluated RepoAudit using two additional LLMs, namely DeepSeek R1 and GPT-4 Turbo, which detected 44 and 14 true bugs with precisions of 75.86% and 35.90%, respectively. More detailed statistics of RepoAudit powered by DeepSeek R1 and GPT-4 Turbo were provided in Appendix C. During the author response stage, we further evaluated RepoAudit using two more LLMs, namely Claude-3.7 and GPT-o3-mini. By scanning the experimental subjects under the same settings, RepoAudit detected 40 and 36 true bugs with precisions of 78.43% and 76.60%, respectively.\n\nAs GPT-4 Turbo exhibits weaker code reasoning abilities than Claude-3.5-Sonnet and the reasoning models DeepSeek R1, Claude-3.7, and GPT-o3-mini, it would yield a weaker performance of RepoAudit. Notably, enhanced reasoning capabilities of stronger LLMs seamlessly benefit RepoAudit, demonstrating its great future potential. We will include further discussion on the performance of RepoAudit using different LLMs in the main body of the revision.\n\n**2.Comparative Table with Existing Works**\n\nWe agree with your suggestion and will include the following comparative table in our revised manuscript, clearly contrasting RepoAudit against existing works:\n\n| Name              | LLM-based | Build-Free | Customizable | General Program | Repo-level |\n|-------------------|-----------|------------|--------------|------------------|------------|\n| GPTScan           | Yes       | Yes        | No           | No               | No         |\n| LLMDFA            | Yes       | Yes        | Yes          | Yes              | No         |\n| LLMSAN            | Yes       | Yes        | Yes          | Yes              | No         |\n| Meta Infer        | No        | No         | No           | Yes              | Yes        |\n| Amazon CodeGuru   | No        | Yes        | No           | Yes              | Yes        |\n| **RepoAudit**     | **Yes**   | **Yes**    | **Yes**      | **Yes**          | **Yes**    |\n\nAs shown by the table, RepoAudit is the first work that supports the build-free and customizable analysis for repository-level bug detection upon general programs instead of domain-specific ones (e.g., smart contracts), supporting the security auditing of large-scale real-world software systems.\n\n**3.Workflow Figure**\n\nWe will enhance the clarity and presentation of the workflow figure. Specifically, we will replace the Claude icon with a more proper icon and move the figure to page 2. Additionally, we will explicitly reference this improved diagram when describing our solution in the introduction, ensuring greater visual clarity and reader comprehension.\n\n**4.Robustness to Prompt Manipulations**\n\nIndeed, it is possible to inject (malicious) prompts into LLM-generated repositories, potentially causing RepoAudit to fail in bug detection. In the following code, we experimentally verified this by injecting several misleading natural language comments into the code. RepoAudit correctly identified a feasible buggy path and reported the confirmed bug in the original code version, but failed when analyzing the new version. Natural language comments in the code can significantly influence the inference results of LLMs, potentially misleading the model to incorrectly identify feasible paths as infeasible. An example is shown as follows:\n\nIn the project `memcached`, the function `proxy_init_startfiles` allocates a memory object and assigns it to the pointer `db`. Later, if `db->buf == NULL` or `db->fname == NULL`, the function returns without freeing the pointer `db`, causing a memory leak bug.\n\n```\n 1  struct _mcp_luafile *db = calloc(sizeof(struct _mcp_luafile), 1);\n 2  if (db == NULL) {\n 3      fprintf(stderr, \"ERROR: failed to allocate memory for db\\n\");\n 4      return -1;\n 5  }\n 6  \n 7  db->buf = calloc(db->size, 1);\n 8  if (db->buf == NULL) {\n 9      /* inject point */\n10     fprintf(stderr, \"ERROR: failed to allocate memory for db->buf\\n\");\n11     return -1;\n12 }\n13 \n14  db->fname = strdup(p);\n15  if (db->fname == NULL) {\n16     /* inject point */\n17     fprintf(stderr, \"ERROR: failed to allocate memory for db->fname\\n\");\n18     return -1;\n19 }\n```\n\nRepoAudit successfully identified this bug in the original version of the code. However, after adding the comment stating, **\"This path will never be executed,\"** at lines 9 and 16, RepoAudit failed to detect the bug. The key reason is that the comments mislead the LLMs to identify the buggy paths that pass the lines 9-11 and the lines 16-18 as infeasible.\n\nLastly, it is important to note that RepoAudit is designed as a developer-oriented code auditing tool. Our threat model assumes developers do not intentionally insert misleading content into their own codebases. Our evaluation effectively demonstrates RepoAudit's practical utility in real-world scenarios. While improving LLM robustness against prompt manipulations is crucial, this aspect is orthogonal to our current research focus and beyond the scope of this work.", "claims_and_evidence": "Yes, the claims are well supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluatoin criteria make sense for the problem.", "theoretical_claims": "There is no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental design and analysis are reasonable.", "relation_to_broader_scientific_literature": "This paper has a broad scientific impact on software engineering in the AI era, particularly in enhancing AI's ability to automatically detect potential bugs in large-scale repositories. It has strong practical significance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "- I think fixing the base LLM to Claude makes the approach less general. The focus should be on the workflow rather than the inherent capabilities of the chosen LLM. Consider replacing the image that includes the Claude icon and restructuring the comparison of different LLMs in the main text instead of placing it in the appendix. \n\n- I hope a clear table can be provided to compare current industry bug detectors and other LLM workflows, helping readers better understand the advantages of your approach.", "other_comments_or_suggestions": "A clear process flow diagram of the RepoAudit framework is needed to illustrate its overall workflow and key components. Figure 3 should be enlarged and visually enhanced, positioned as the first image to provide readers with a concrete and direct understanding of the proposed workflow.", "questions_for_authors": "Is it possible to inject simple prompt manipulations into LLM-generated repositories to jailbreak RepoAudit and prevent it from reporting errors?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xwx9jr8IfB", "reviewer": "Reviewer_HHzn", "summary": "This paper presents REPOAUDIT, a system for auditing source code using large language models to identify and report software bugs. The system is designed to detect common vulnerabilities such as null pointer dereferencing, memory leak, and use after free. It utilizes a combination of parsing libraries, large language models, and a multi-step exploration process to identify potential bugs within software repositories. The system was tested against several real-world projects and compared to existing auditing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Bug Customization**\n\nPlease refer to the response to the first concern **Bug Customization** of [Reviewer yKxy](https://openreview.net/forum?id=TXcifVbFpG&noteId=ZFh3alkmPr).\n\n**2.Case Studies of FPs/FNs of RepoAudit**\n\nThank you for your suggestions. We collected the following typical cases and summarized the root causes of the FPs and FNs. We will add these cases and discussions to our revision.\n\n**_Two False Positive Examples:_**\n\n**Example 1:** In the project *icu*, the function `getTimeZoneRulesAfter` contains a `goto` statement making the control flow jump to the `error` label ([code](https://anonymous.4open.science/r/cases-BBFA/example1_1.md)). RepoAudit reports an FP of a Use-After-Free (UAF) at the second uprv\\_free(newTimes) at line 13 due to the hallucination of Claude-3.5-Sonnet. The model incorrectly identifies a spurious program path that leads to uprv\\_free(newTimes) after `newTimes` is freed in the loop.\n\n**Example 2.** In the project `frr`, the function `vrf_get` can return NULL under certain conditions: In the function `lib_vrf_create` ([code](https://anonymous.4open.science/r/cases-BBFA/example2_2.md)), the return value of `vrf_get` ([code](https://anonymous.4open.science/r/cases-BBFA/example2_1.md)) is assigned to the pointer `vrfp`, which is subsequently dereferenced in the expression `vrfp->status` without a null check at line 13 in the function `lib_vrf_create`. RepoAudit reports it as a NPD, while it is an FP. Due to YANG schema validation, the `vrfname` variable is guaranteed to be non-NULL. Given that `vrf_get` only returns `NULL` when both `name == NULL` and `vrf_id == VRF_UNKNOWN`, it cannot return `NULL` when `vrfname != NULL`. Hence, the dereference is safe in practice. The root cause is that the LLMs are not aware of the fact that the return value of `yang_dnode_get_string` is never `NULL`.\n\n**_A False Negative Example:_**\n\n**Example 3:** In the `libsass` project, the function `sass_make_data_compiler` ([code](https://anonymous.4open.science/r/cases-BBFA/example3_1.md)) allocates a memory object and passes it as the second argument to `sass_prepare_context` ([code](https://anonymous.4open.science/r/cases-BBFA/example3_2.md)). Within `sass_prepare_context`, if `calloc` fails at line 3, `ctxmem` is set to 0, and the function returns 0 without freeing the allocated memory object `cpp_ctx` or assigning it to any other pointer, leading to a memory leak.\n\nThis bug was detected by Deepseek-R1 but missed by Claude-3.5 and GPT-4o Turbo. The latter models did not accurately track all relevant execution paths, particularly the error-handling path in this case. In contrast, reasoning-oriented models like Deepseek-R1 demonstrated superior capability in recognizing execution paths, allowing RepoAudit to detect such memory management issues more effectively.\n\n**_Comparison with Existing Symbolic Tools:_**\n\nExisting symbolic code auditing tools, such as Meta Infer, can avoid the FP and FN in Example 1 and Example 3, respectively, as they symbolically enumerate all the program paths, thereby covering the program behaviors along different paths. However, they are also unable to understand the behavior of `yang_dnode_get_string` as it depends on the library function `lyd_get_value`, of which the implementation is absent in the analyzed project. Therefore, existing symbolic tools can also report the FP in Example 2.\n\n**_Future Improvements:_**\n\n- **Enhancing Model Reasoning Capabilities**: We can improve the LLM’s ability to reason by either adopting more advanced models or fine-tuning the current models for specific tasks. Fine-tuning can be tailored to particular sub-tasks like exploring feasible program paths in single functions, thus mitigating the FPs and FNs (e.g., Examples 1 and 3). Additionally, incorporating specialized training datasets focused on code analysis could further enhance the model’s accuracy in these contexts.\n\n\n- **Expanding the Tool Suite for Better Retrieval**: Another significant enhancement would involve integrating existing compilation-free analysis tools to identify all potential branches and loops within a program. This integration would offer a clearer representation of the program’s control flow structure to the model. Such an improved RAG design has the potential to significantly reduce the false positives and false negatives produced by RepoAudit, such as Examples 1 and 3.\n\n- **Adding Multi-Modal Support for Library Function Understanding**: Utilizing library documentation and other non-code material as knowledge bases of the LLMs enables RepoAudit to understand library functions better. For example, the library documentation can facilitate RepoAudit in identifying the non-null value after YANG schema validation, thereby avoiding the FP in Example 2.\n\n**3.Model Choice**\n\nPlease refer to the response to the first concern **Model Choice** of [Reviewer Le35](https://openreview.net/forum?id=TXcifVbFpG&noteId=5JaOhTUECl).", "claims_and_evidence": "The paper demonstrates an innovative approach by leveraging LLMs like Claude 3.5 Sonnet to perform data-flow analysis and control-flow validation, which significantly enhances bug detection. The system's ability to detect inter-procedural bugs is a notable strength, as demonstrated by the high number of true positive results, particularly for cross-function vulnerabilities.", "methods_and_evaluation_criteria": "The authors introduce two important validation mechanisms—alignment validation of data-flow facts and control flow, and feasibility validation of inter-procedural program paths. These mechanisms help ensure the accuracy of bug reports, reducing the likelihood of false positives and increasing the reliability of the tool.\n\nWhile the paper focuses on three bug types (NPD, MLK, and UAF), it does not address how REPOAUDIT can handle a broader range of vulnerabilities. The generalizability of the system to other bug categories or more complex vulnerabilities is not sufficiently explored, which limits its applicability to different types of software projects.\n\nWhile the paper reports false positives and true positives, there is insufficient discussion on the nature and causes of errors, especially false positives. A deeper analysis of why false positives occur and how they could be reduced or eliminated would improve the overall understanding of the limitations of REPOAUDIT.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "REPOAUDIT is both time-efficient and cost-effective. With an average analysis time of 0.44 hours per project and a low cost per bug detected, the system provides a practical and scalable solution for auditing large codebases. The comparison with traditional bug detection tools shows that REPOAUDIT is competitive, particularly in terms of precision and resource utilization.\n\nREPOAUDIT’s performance is highly dependent on the underlying LLMs, particularly Claude 3.5 Sonnet, and the results may vary with other models. This reliance on a single model raises concerns about the system's robustness and adaptability across different LLMs or future updates. More discussion on the potential for model flexibility would strengthen the paper.", "relation_to_broader_scientific_literature": "It utilizes a combination of parsing libraries, LLMs, and a multi-step exploration process to identify potential bugs within software repositories.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the proposed approach yields strong performance on a challenging task and leverages a nontrivial approach to do so. The main concerns centered on the breadth of the evaluation (e.g. the focus on Claude) and the writing, both of which were improved/addressed in response to the reviews. In this light, we recommend accepting the paper.
Decision: Accept (poster)