Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning
Authors: Brian Zhang
ZHUO ZHANG
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
rule based reasoning
smart contract
accounting bugs
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We develop ABAuditor, a hybrid LLM and rule-based reasoning system to detect bugs with substantial monetary consequence.

Abstract: Financial transactions are increasingly being handled by automated programs called *smart contracts*. 
However, one challenge in the adaptation of smart contracts is the presence of vulnerabilities, which can cause significant monetary loss.
In  2024, $247.88 M was lost in 20 smart contract exploits.
According to a recent study, accounting bugs (i.e., incorrect implementations of domain-specific financial models) are the most prevalent type of vulnerability, 
and are one of the most difficult to find, requiring substantial human efforts.
While Large Language Models (LLMs) have shown promise in identifying these bugs, they often suffer from lack of generalization of vulnerability types, hallucinations, and problems with representing smart contracts in limited token context space.
This paper proposes a hybrid system combining LLMs and rule-based reasoning to detect accounting error vulnerabilities in smart contracts. 
In particular, it utilizes the understanding capabilities of LLMs to annotate the financial meaning of variables in smart contracts, and employs rule-based reasoning to propagate the information throughout a contract's logic and to validate potential vulnerabilities.
To remedy hallucinations, we propose a feedback loop where validation is performed by providing the reasoning trace of vulnerabilities to the LLM for iterative self-reflection. 
We achieve 75.6% accuracy on the labelling of financial meanings against human annotations. 
Furthermore, we achieve a recall of 90.5% from running on 23 real-world smart contract projects containing 21 accounting error vulnerabilities.
Finally, we apply the automated technique on 8 recent projects, finding 4 known and 2 unknown bugs.

Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning
Brian Zhang
Zhuo Zhang Purdue University West Lafayette, IN 47906
University of Texas at Austin Austin, TX 78705 bz5346@utexas. edu
zhan3299@purdue... edu
Abstract
Financial transactions are increasingly being handled by automated programs called smart contracts. However, challenge in the adaptation of smart contracts is the presence of vulnerabilities, which can cause significant monetary loss. In 2024, $247.88 M lost in 20 smart contract exploits. According recent study, counting bugs (i.e., incorrect implementations of domain-specifer financial models) are the most prevalent type of vulnerability, and are of the most difficult find, requiring substantial human efforts. While Large Language Models (LLMs) have shown promise in identifying these bugs, they often suffer from lack of generaliza- tion of vulnerability types, hallucinations, and problems with representing smart contracts in limited token context space. This paper proposes a hybrid system com- bining LLMs and rule- based reasoning to detect accounting eгror vulnerabilities in smart contracts. In particular, it utilizes the understanding capabilities of LLMS to annotate the financial meaning of variables in smart contracts, and employs rule-based reasoning to propagate the information throughout a contract's logic and to validate potential vulnerabilities. To remedy hallucinations, we propose a feedback loop where validation is performed by providing the reasoning trace vulnerabilities to the LLM for iterative self-refiection. We achieve 75.6% accuracy on the labelling of financial meanings against human annotations. Furthermore, we achieve recall of 90.5% from running on 23 real-world smart contract projects containing 21 accounting error vulnerabilities. Finally, we apply the automated technique on 8 recent projects, finding 4 known and 2 unknown bugs.
Introduction
Blockchains are public, append-onl) ledgers that record data in a secure manner and provide foundation for many financial applications. Two such applications are tokens and smart contracts. Tokens are cryptocurrencies (e.g., Bitcoin Nakamoto [2008] WETHWETHH (024)). ). Smart contracts are programs that perform service once deployed to the blockchain, typically financial in nature. Example services include banking, marketplaces, and loaning. With the digitization of finance, smart contracts have seen increasing usage for financial transactions. For example, the Ethereum blockchain has over 40 million smart contracts deployed Bitkan [2024). Smart contracts are implemented in Java-like language known as Solidity (023) They are similar to traditional software applications in that they are composed of functions, and not all functions can be invoked (due to function modifiers). Entry functions are those that can be directly invoked by users, which are defined as any entities (e.g.. humans) interacting with the smart contract.
Our work focuses on one challenge in smart contracts, namely their vulnerability to accounting bugs. As defined by Zhang et al. [2023], accounting bugs are incorrect implementations of domain-specific business models. According to the study, accounting bugs are the most popular type of bug, as well as one of the most difficult for humans to identify. Furthermore, they cause significant damage exploited. 2024, more than $50 M dollars damages DefiLlama 2024 were caused by
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

accounting bugs so far. There are many existing vulnerability detection techniques for smart contracts.
including fuzzers Wüstholz and Christakis
[020]; Choi et al. 202T] program analysis Wang
et al. 2019:; Huang et al. [2022] verification Jiao et al. [2020] Tan et al. [022] and symbolic
CescutionCConsensee 2024] Bose et al. [2022]. However, we find that most cannot handle accounting
bugs. Recendly Zhang [2024] proposed program analysis based technique that relies on a type
system. In particular, it relies on manual annotation of initial types of variables, which include the currency unit, scaling factor, and correspondence to some functionality in bank. With such knowledge, it performs type checking to detect accounting bugs. However, the annotation extensive, in many cases requiring deep understanding of the code and even external documentation. Hence, there is a need to develop an automated technique for detecting accounting bugs in smart contracts.
According to Zhang et al. [202]], the reason that accounting bugs are difficult to identify is due to the necessity to first understand the complex business logic of smart contracts. We hypothesize that L.LMs like GPT can be used to perform the analysis, citing their capability to easily comprehend code. For example, [Sun et al. 2024) propose an L.MM based technique that performs analysis of smart contracts, through utilizing context from past vulnerabilities as well as techniques such Chain-of-thouge (COT) to improve the reasoning capability of the LLM. However, the approach does not focus on accounting bugs, and the analysis is function-level as opposed to file-level. Furthermore, significant concern with LLM--baed techniques is the generation of hallucinations. Hence, a naive approach focused on detecting accounting bugs providing high-level descriptions and few-shot examples of accounting bugs does not suffice, due to the observations that:
The code of smart contracts provided for analysis is often too large or too costly to fit in the limited token context space. While it is possible to filter non-relevant functions (i.e., functions which cannot be invoked by the public), the remaining code usually retains the same issues. Furthermore, reducing the scope of the analysis to function-level can miss accounting bugs that spread across multiple functions or even files.
Challenge 1:
Challenge 2:
Most warnings produced by the method are often hallucinations from the model, and there is no existing technique to validate the warnings without human analysis.
While the naive approach fails, we are able to develop 4 key insights. First, the task of detecting accounting bugs can be split into two subtasks: assigning financial meanings to variables and checking the correctness of operations. For example, there is an accounting bug in the operation z= + Y, where X has a financial meaning of balance (i.e., an amount of some currency) and has financial meaning of price (i.e., an exchange rate between two currencies), the reasoning being that such an operation does not result in meaningful output, hence is a violation of all business models. Second, LLMS such as GPT can easily assign financial meanings to variables through semantic analysis. However, prompting LLMs to annotate all variables is infeasible due to monetary cost and risks of misclassifications due to hallucinations. Hence our third insight: rule-based reasoning allows for propagation of financial meaning, efficient checking of operation correctness, and function-level analysis with file-level scope. Rule-based systems (Haee-Rothh [1985) are an approach to artificial intelligence that operates on set of predefined rules. In our context, inferences rule are utilized for the propagation, taking advantage of the deterministic results of financial meaning operations. Furthermore, the rules substantially reduce the workload on L.MSs to only annotating entry variables (i.e., parameters of entry functions and global variables), as all other variables can be assigned financial meanings from the propagation. Finally, inference rules can be enhanced to enable propagation through function invocations, reducing the analysis to function-level while maintaining file-level scope for detection. Fourth, while hallucinations are still an issue, a reasoning trace can be prompted to the LLM to detect hallucinations in annotations. Furthermore, the trace can be integrated with an iterative feedback loop to automatically remedy hallucinations.
Based on the insights, we develop A ABADDITO,, a hybrid L.LM and nule-based reasoning system to automatically detect accounting errors. It utilises the understanding capabilities of LLM (specifically, GPT 3.5-turbo OpenAl (2024) to provide financial meaning annotations for entry variables and rule-based reasoning to propagate meanings to other variables and to discover potential vulnerabilities. Furthermore, our technique reduces hallucinations by implementing an iterative feedback loop where reasoning traces of potential bugs are provided to the LLM for self-reflection. In sum, we make the following contributions.
We abstract detecting accounting bugs into two subtasks: identifying financial meanings of variables with LLM, and checking correctness of operations with rulc-based reasoning.
2

---

1 contract Vagiclp4ggregatorl
2. function getPairPrice() public returna (int256)
3 baseRegerve - baseCurrency. htt.urreeccc.ooluuupppp.ooo.e totalSupply(; O);
quoteReserve quoteCurrency. totalSapply()
priceOfBaselnquutee
5
totalShareSupply totalShareSupply() return ((baseReaerve quozeReaerva) totalShareSupply);
?
}}
Figure 1: Buggy Code from Abracadabra MIMSwap Abracadabra
[2024]
We develop method to detect hallucinations in GPT via generating reasoning trace and prompting it back to GPT for self-reflection.
We implement prototype ABAUDITOR based on Slither FFist et al. [2019] using the GPT-3.5 turbo model as our LLM. We evaluate on 34 real contracts from ZZhng et al. [2023] that have 40 reported accounting bugs, Among these bugs, we find that 19 of them belong to categories such as pure math errors (i.e., the operation is swapped with <) hence fall out of scope. Our method detects 19 of the remaining in-scope bugs (i.c., 90.5% recall). Furthermore, we achieve 75.6% accuracy in financial meaning annotations by our system versus by humans. We also measure the results without our reasoning trace-based hallucination reduction, and find that the trace reduces the false positive rate by 54.5%. Furthermore, we run our system on new smart contract projects, finding out of accounting bugs, two of which are zero-day (i.e. not discovered before) vulnerabilities.
2
Motivation
We demonstrate the effectiveness of our hybrid system with the following real-world example.
In January 2024, the smart contract project Abracadabra MIMSwap Mutual [2024) was exploited for $6.5 M dollars. The loss due to attack known flash loan attack, which when malicious actor takes advantage of a vulnerability to control a currency's 's price and make profit. Typically the attack begins via large loan which is used to skew the price through the vulnerability. The attacker then makes a profit based on the skewed price (by trading the exploited currency), and pays off the loan right after. Flash loan attacks have caused $33.35 M in damages in 2024 alone DefiLlama [2024). The developers fixed the vulnerability in their new version, and posted the code as a bug bounty on Code4rena March. Code4rena| (2024) is prestigious vendor for smart contract auditing competitions. However, the new implementation was also vulnerable to the aforementioned attack. The vulnerability was an accounting bug, and was only found by two security researchers, among hundreds. Figure I shows code for the function that contains the accounting bug. The function is written in Solidity, and has been simplified for demonstrative purposes. The code was taken from the released Code4rena bug report after publication, and the vulnerability has already been fixed.
In Figure I getPairPrice () function in the file Magi MagicLLpeggregator sol. The file im- plements an exchange of three cryptocurrencies: the base token, the quote token, and the share token. the exchange, users can trade base tokens with quote tokens, or vice versa. Intuitively, the functionality is analogous to an ATM. The share token represents share of the total base and quote token the exchange. It acts similar real-world stocks, where instead of a company, ownership designated to the amounts of base and quote tokens in the exchange. Users can buy sell the share token by paying or receiving amounts of both the base and quote token, respectively. The function getPairPrice is used to calculate the price of the share token, in units of the base token.
In the getPairPrice() function, the price of the share token is calculated as fol- lows. The total amount of base tokens is calculated on line 3 via the function call baseCurrency. totalSupplyl)) The total amount of quote tokens is calculated on line 4 via the function call quoteCurrency. totalSuppll (). is then converted to an equivalent amount of base tokens by multiplying the price of base tokens. The function calls baseCurrency. totalSupply O) and quoteCurrency. htt.eurrrrnccy.ttallalpppppppp totalSupply return the exact amount of each token owned by the smart contract. Then, the amount of available share tokens SootalharrSSpplly) 5. Finally, the share token price is calculated by adding the total amount of base currency, baseReserve, with the total amount of converted quote currency, quoteReserve, and dividing by the totalShareSupply.
obtained on line
3

---

The accounting bug is due to using baseReserve and quoteReserve to compute the share token price. Specifically, is due to the function calls baseCurrency. totalSupply ( and quoteCurrency. cotalSupply () which directly return the amounts of base and quote curren- cies owned by the smart contract. malicious actor can initiate a flash loan attack by taking large loan of the base or quote currency and paying it to the smart contract. This results in the amount of base or quote currency to be inflated, and when calling the function getPairPrice O a grossly incorrect share price is returned. The malicious actor can take advantage of the price to make profit.
When our technique is run on the smart contract, it identifies the vulnerability with no false positives. It can determine that the variables baseReserve, quoteReserve, and SotalPairSupply all repre- sent reserves the smart contract, and that the return value of the function priceofBase represents price. Reserve is defined as an amount of currency that is owned by the smart con- tract. Price is defined as the exchange rate of one currency to another. Furthermore, is able to determine that the addition operation on line 6, (baseReserve + quoteReserve), also has financial meaning of reserve. Intuitively, the result of the operation is considered reserve be- cause it represents the aggregation of two reserve values, thereby maintaining its definition. Our system reports an error for the division operation on line 6, where the code attempts to produce price by the division of two reserves, following a rule that disallows any division of two re- serves. Such divisions are problematic due to being a potential vulnerability for flash loan at- tacks, demonstrated above. A better alternative would be to obtain the price by querying off- chain authorities, which are trusted services that process data such as prices off the blockchain.
()
3 Design
Convervior SSA Airr
Lis of cxary Ganctions SSA
Armotation of pacamaters alabal
naive approach of directly prompting the
LLM does not work (Apppendix A). The under-performance of the naive approach leads to our proposal of ABAUDITOR. Fig- ure shoows the architecture of ABAUDI- TOR. The blue blocks represent actions taken by the system, while the grey blocks represent the intermediate and final results produced by the system. ABAUDITOR takes input singular smart contract file or an entire directory with many such files. First, ABAUDITOR extracts the control flow graph (CFG) of the targeted smart contracts and converts the individual operations to static single assignment (SSA) (SSA)Cytroo et al. 1986), a specific intermediate representation whose details not needed to understand our paper. From the extracted CFG, entry functions are identified. Recall entry functions are those that are accessible to all users. These are done using APIs from an existing analysis framework Slither Feist al. [2019). Once all entry functions are identified, the next step involves identifying and annotating the financial meanings of their parameters as well as global variables. This discussed in more detail in Section 3.1 After the initial labeling, propagation of financial meanings and verification of the correctness of all operations is done through rule-based rea- soning. This is discussed in more detail in Section 3.2 At this step, list of potential vulnerabilities is generated and validated as either a hallucination or a real bug. The validation process is discussed in more detail in Section 33
Frew
Fual ausouring enoe vulls
Yalidution kulucianto
Posettial accousting onor vilre.
Rule-
Figure 2: Architecture of Our System
We demonstrate each step of our tech- nique with an example found in Fig- ure 3 which shows a bug. The code contains two functions: tswap() and applyFee() (). tswap () is public function (i.e., it can be invoked by all users) which performs a swap of tAmount of one currency to another. During the swap, a fee is applied. Function applyFee is an internal function used to calculate the remain- ing amount after fee is applied in the exchange. The accounting bug is that
contract CurrencySuapt
uint256 netRate;
function Supitt2 _tAnount, uint256
_uetRate)
internal returne (uint256 [fAmount)
fAmount _timount * __aetaat;;
function (avnp(uin tAnount, wint256, price) public returns (uint256 avappedAmount))
// USAGE: deducts fee from 'tAmount'
10
thate - netRate;
not net_tAmount
PrplyPeeetAnouse fRate);
12
Ainal_tmmoutt net_ vet_uttount * #Bate;
auappedAnount rinal_ tAmount price;
14
>}
Figure 3: Demo smart contract with accounting bug
4

---

+ (Ri) 91 : rrwball y2 : rawbbal
(Ra)
reserve V2 reserve
x == y + y2 : rawbal
I. == y1 + 1/2 : reserve
y1 rawbal y2 rawbol
(Ra) y netbal y2 netbal
(Ra)
yn y2 rawbal
21 22 nelbal
rambbal y2 netrate
(Ra) y1 reserve y2 netrate
y1 netbal
y2 netrate
X (Rs)
(Rr)
x = x y/2 netbol
== x y2 [Error]
==
X y2 [Error]
91 rawbal y2: : rawbal
(Ro) y1 reserve y2 reserve
#: (Rs)
x = yn + ya : price
= y1 + ya [Error]
Figure 4: Subset of Rule Based Inference Rules
the remaining amount is less than intended. Particularly, this is because fee is applied to it twice, one instance on line 5 in applyFee(), and the other on line 12.
3.1 Initial Annotation by LLM
For every entry function parameter and global variable, GPT performs initial annotation of their financial meaning. In our implementation, we define 6 possible choices for initial financial meanings, which are: raw balance, net rate, interest rate, debt, price, and reserve. Raw balance is defined as an amount of currency that is strictly owned by user. Net rate is defined as the percentage left after applying fee. In the blockchain economy, many smart contracts deduct fee from users to earn profit. Net rate is the amount left after the fee. (e.g., if 10% fee was deducted, net rate is 90%). Interest rate represents the percentage charged on borrowed or lent funds. Debt refers to the amount of borrowed funds owed by user. Price represents the exchange rate from one currency to another. Reserve denotes the pool of funds strictly owned by the smart contract. These latter two financial meanings were depicted in Figure T] in the motivation section. We restrict the choices of initial annotations to these types as they are sufficient for commonly seen business models, although is very easy to expand the system.
GPT is prompted with high-level definitions of each financial meaning, as well as few-shot examples of real-world instances. To fit the few-shot examples as well as to account for the potential size of the entry function, each financial meaning is formatted in its own prompt. To perform the annotation for an entry function parameter, the name of the parameter and the code of the entry function are provided to the prompt. To perform the annotation for a global variable, a similar prompt crafted for every function it appears in, and the final assignment is done via the first financial meaning to appear times, or a majority vote. An example can be found in (nppendix B and Annenddi C
3.2 Rule-based Reasoning
We describe the inference rules used in reasoning. The rules are invariants (i.e. properties held across different business models) that we have manually summarized. Th invariants are summarized from a study of many business models and their variants Zhang et al. [023). A subset of these rules is included in Figure 4, There are in total 119 such rules. Within the table, T represents variable y having the financial meaning of r. The statement r := ... 7 represents that T is the resulting financial meaning from the statement that will be propagated to æ.
Rule RS specifies that when a variable y1 with the meaning rawbal, or raw balance, is added to variable y/2 which is also a raw balance variable, the result x is a raw balance. Rule R2 is similar for reserve. Rule R3 specifies when variable y1 is subtracted from a variable y2, and both have financial meanings of raw balance, the result is still a raw balance. Rule R4 specifies when a variable 91 is subtracted from variable y2, and both have financial meanings of net balance, the result is still a net balance. Net balance is defined as an amount of currency owned by user that has already had fee applied. Rule RG specifies when a raw balance variable y11 is multiplied by net rate y2, the result is a net balance. The operation represents collecting a fee from a user. Rule Rs specifies when a reserve variable y/1 multiplied by a net rate 1/2, the result is an error. The operation is deducting a fee from the smart contract, which is counterintuitive. Rule R7 specifies when net balance variable y1 multiplied by a net rate y/, the result an error. The operation represents collecting fee twice from user, which should not be allowed. Rule Rs specifies when a raw balance variable is divided
5

---

User:
are given reasoning trace an accounting bug. Ideetiify any possible mislabellngs. each operation, variable linancial meaniny. Firanciaal moaning defined ...
Here the inference rules for
"raw halance" "net rite" "net halance"
R2: "reserve" x "net rate": Froor
The trace is provided below. It contains all related operations to the accounting bug.
Fach line trace contains an operation following format:
Tree line: Operation: Propusation Repsont Line
Trace!
t:tAmount tswap() typed by the LLM entry function tswap() Reserve
2: rAmount in Fee() tAmount in tswap(s JAmount is Reserve due to parameter propagation line
___eeeRRtee (Net Rate)
4: Error in operation" _tAmuunt (Reserve)* _netRutee (Net rate)" in applyFec( on line 5 due to violation R2)
MaVarailallee "tAmount" should be classified as raw balance. is used to multiply netRate", which rate, the most likely candidate raw balance, according R1.)
Figure 5: Reasoning Trace Prompt and Response from Hallucination
by raw balance, the result price. Rule Rg specifies when a reserve variable y11 divided by another reserve variable y2, the result is an error.
The overall process of ABAUDITOR is the following. Given smart contract, it traverses the statements in the function invocation order and the control flow graph order. It prompts the LLM to assign the initial annotation when encountering a global variable or an entry function parameter that has not been prompted before. Otherwise, it invokes the inference engine that follows the aforementioned rules to propagate financial meanings and identify potential bugs. For each step of propagation, the system records its source. This is for later construction of reasoning traces. For example, _tAmount on line 5 in Figuue 3 its origin from t.mouut on line whose origin is in turn tmmount on line 11. whose origin IS line 7 by the LLM. The algorithm and an example of the whole process can be found in Appendix D
3.3 Iterative Validation of Accounting Bugs and Application of Remedy
At this point, our algorithm has produced a list of statements containing potential accounting bugs. However, some of the bugs are produced from mislabelled initial annotations (i.e., due to GPT hallucinations), while others are true accounting bugs. this subsection, we discuss how we detect hallucinations by providing reasoning trace for GPT to perform self-reflection. We also explain how this approach can be integrated into an itcrative feedback loop to automatically correct hallucinations and re-run the analysis, thereby improving the accuracy of bug detection.
For an accounting bug report regarding some buggy statement, its reasoning trace contains the list of operations involved in the propagation of financial meanings from the initial LLM annotations to the operands in the buggy statement. For each operation, the source statement and the rule applied in propagation are included. At the end, the explanation of why the system considers this a bug is also appended. provides a comprehensive context on which the LLM performs self-reflection. We then compose self-reflection prompt to the LLM,, providing the trace, the definitions of all financial meanings, together with a subset of related reasoning rules in natural language, and ask the LLM to consider if other initial annotations are possible for the involved entry/global variables. If so, the system continues with the new annotations. If the bug disappears with the new annotations, we consider it false positive caused by hallucination. The procedure is iterative. If the LLM considers there are no other annotations, and the bug still persists, we consider the bug real and include the final report. The trace construction algorithm is presented in Appendix E
Example. An example of a reasoning trace can be found in Figure 5) It was generated from an accounting bug produced by mislabeled inital annotations (due to hallucinations). We have reduced its content for sake of space, and only include necessary details for discussion. Here we discuss the content of the reasoning trace well as how GPT responds to it. As mentioned previously, the

---

definitions of all financial meanings are provided in the trace. Specifically, they can be found in the yellow box labeled with "1". The green box labeled with "2" contains the set of relevant rules. We have selected two such rules (R1: raw balance x net rate = net balance [a raw balance variable multiplied by a net rate results in a net balance] and R2: reserve x net rate Error). Finally, we provide the list of related operations to the reported accounting bug in the blue box labeled with "3". We go over the included operations as follows. In the first operation, we provide that the variable tmoount located in function tswap () was initially annotated by GPT as reserve. In the second operation, we provide that tAmount in tswap() is propagated to parameter _tAmount in function applyFee (), and hence the latter also has the financial meaning of reserve. The third line begins chain of related operations depicting how variable netRRte in function applyFee () is eventually assigned the financial meaning of net rate, however, we exclude them here for the sake of space. In the fourth operation in the figure, we provide the erroneous operation of ,t.Amount on line 5 in applyFee O) and explain how it is accounting bug due the rule R2. In particular, ..moount was assigned the meaning of reserve while _neeRaaee was assigned the meaning of net rate, and the result of their multiplication is error, as defined by R2.
_netRate
The response from GPT can be found in the orange box labelled with "4". In particular, the response details how the entry variable tAmount should have been classified as a raw balance. It supports this claim by referencing that if tAmouut (which had its meaning directly propagated from Amount) was classified as such, the fourth operation would succeed according to R1. Since GPT reports that reclassification should be made, the initial annotations are updated (i.e.. tAmount is classified to raw balance), and the maee reasoning algorithm is rerun. Although the annotations are now correct, the rerun still produces a potential accounting bug in line 12. We note that this accounting bug is a true bug, as discussed in Section 3.2 The system reports it accurately, and the LLM does not flag any mislabeling. For more details, refer to Appppddixx F.
4 Evaluation
We implement our technique on the Slither [13 framework. We chose GPT-3.5 turbo for our LLM. In this section, we conduct experiments to determine the effectiveness and efficiency of the technique. We aim to answer the following research questions.
Research Question 1: What is the effectiveness of our approach?
Research Question 2: What is the distribution of financial meanings assigned by the LLM?
Appendix G)
Research Question 3: What is the efficiency of the approach?
Research Question 4: What is the effect of fine-tuning and providing few-shot examples in
our system?
Research Question 5: Can less powerful model than GPT-3.5 be used in the approach?
Appendix H)
Research Question 6: How sensitive the LLM annotation to variable names?
(Appendix
4.1 Experimental Setup
Benchmark. We utilize the dataset provided by Zhang et al. 2023 which contains 513 smart contract bugs across 113 smart contract projects. 72 are categorized as accounting bugs. We preclude 15 of them as they cannot be loaded by Slither miss code. We further exclude 6 projects due to reasons such as code obfuscation. The list of remaining projects can be found in /aalle While a project may have many files, we only run ABAUDITOR on those with accounting bugs. We further analyze 8 new smart contract projects containing accounting bugs collected from the most recent Code4rena bug reports. The list can be found in Appendix Table 4
Baseline. For our baseline, we ran our rule-based reasoning algorithm using annotations provided by humans on the main dataset. Specifically, we performed manual annotation of financial meanings on every file containing accounting bugs in the benchmark. In each project, we only annotate the entry variables, leveraging the propagation ability of our rule-based reasoning. We find that such effort is substantial, sometimes requiring significant time and attention to detail. We also run our benchmark with two recent tools, namely GPTScan (Sun et al. (2023) and ItyFuzz (Shou et al. (2023) )) We do not run with older tools, since according to Zhhan et al. 2023 published in 2023, accounting bugs were beyond existing tools at the time.
7

---

Table 1: Accuracy of Initial Financial Type Labelling
Labelling
Baseline
Ours
No Rem.
Project name
NIS
Tokens
Fune.
Time
Req.
Baseline Ours Iters.
TP
FP
TP
FP
TP
FP
MarginSwap
3
3
3
1
0
0
1
37558
23
24.47s
18
VaderMath
4
4
2
1
2
2
165284
60
219.74s
45
PoolTogether
2
1
0
51517
10.73s
16
YieldMicro
1
59592
25
10.23s
yAxís
3
2
1
2
1
22178
121
73.11s
17
BadgerDao
4
3
1
0
0
28379
22
154.06s
17
WildCredit
3
I
2
0
23058
2143
87.91s
14
Pogethg v4
0
O
O
198600
60
14.75s
53
Badger Dao p2
4
0
0
100079
13.71s
43
yAxis p2
0
0
1
5655
5
15.2s
3
Malt Finance
2
0
1
2
139665
24
51.69s
45
Perennial
3
0
4106
21
146.89s
7
2
Sublime
9
27263
35.36s
Yeti Finance
1
14256
247
48.81s
14
Vader Protocol p3
1
3
I
90713
84
121.9ss
47
Dao
3
0
0
587797
32
74.06s
25
Racket Joe
3
1
1
0
96258
41
178.76s
47
Concur Finance
0
0
62089
184.32s
40
Biconomy Hyphen
5
4
1
0
0
105347
248
102.63s
30
Sublime p2
4
4
0
0
0
0
1
109168
40
34.70s
37
Volt Finance
1
0
0
135919
79
34.96s
16
Badger Dao p3
0
129386
518.63s
Tigris Trade
5
4
0
599294
3493
492.87s
471
Total
82
62
19
7
19
10
22
19
The experiments are conducted on a machine with AMD Ryzen 3975x and 512GB RAM,
4.2
Research Question Effectiveness of approach
We ran our system and the baseline technique on the benchmark. We manually inspect all results reported by both methods and categorize them as true positives (TP) or false positives (FP). We also compared the annotations produced by the LLM (i.e., from our system) with the human annotations. We collected the number of invocations of the reasoning trace feedback loop the number of tokens passed to GPT, the number total functions analyzed, the run time and the number of GPT requests. The manual evaluation involved two researchers. Two independently categorized the results and provided annotations while an external judge resolved any inconsistencies.
The results can be found in Table 1 leftmost column. The column labelled "Baseline" shows the total human annotations for each project. The column labelled "Ours" shows the number of correct annotations by GPT. The column labelled "Iters" shows the number of times the iterative feedback loop was invoked. The following three headers contain the results from running the baseline (labelled "Baseline"), our technique with the feedback loop (labelled "Ours"), and our technique without (labelled "No Rem."). "). For each header, the column labeled with "TP" contains the number of true positive bugs. The column labeled with "FP" contains the number of false positive reports. Accounting bugs that are out of the scope of our system can be found in the column labeled with "NIS" or Not-in-Scope. Such accounting bugs belong to other categorics such as pure-math-erree (i.c., a '>' sign is swapped with a '<' sign). The following data unique to our system. The number of tokens prompted can be found in the column labeled with "Tokens". The number function analyzed can be found the column labeled with "Func.". The run time per project can be found in the column labeled with "Time", The number of requests set can be found in the column labeled with "Req.".
The name of the smart contract project can be found in the
Observations. Both our system and our baseline were run on 23 smart contract projects, containing 40 accounting errors. regards to the effectiveness, our system reports 19 TP and FP, while the baseline reports 19 TP and 7 FP. While both techniques are unable to detect 21 of the total accounting errors, we find that and our baseline are 19/(40-19) 90.5%. In regards to labelling, technique can correctly annotate
them are beyond scope. Hence, the recalls both our system

---

Table 2: Results of GPTScan
Bug Type
Included Projects
Total Instances
Wrong Order Interest
2
2
Flashloan Price
6
4
First Deposit
0
0
Approval Not Revoked
Table 3: Results Using Fine-tuned GPTs
Model
TP
FP
Iters.
Correct Annotations
Baseline (Manual)
19
7
N/A
82/82
ABAuditor (Gpt3.5 w/ few-shot)
19
62/82
GPT3.5 no few-shot
17
31
14
32/82
Finc-tuned GPT3.5 no few-shot
17
16
9
39/82
Fine-tuned GP53.5 w/ few-shot
19
9
7
64/82
Fine-tuned GPT4.0 mini w/ few-shot
19
2
71/82
62 out of the 82 human annotations performed in the baseline, achieving an aceuracy of 62/82 75.6%. We find that most misclassifications were due to hallucinations in the initial annotation not resulting in accounting bug warnings. As a result, our feedback loop was not run, hence leaving the misclassifications as is. Also observe that using the reasoning traces reduces the number of FPs from 22 to 10, denoting a 54.5% reduction, without losing any TPs. In most cases, fixing these hallucination requires less than 3 iterations.
False Positives. We manually inspect the false positives produced by the system. We find that of the 7 FP common to both our full system and our baseline, the most common issue is the path-insensitive nature of our rule-based reasoning. For example, our rules are unable to deal with the scenario where a variable can potentially have two financial meanings due to an if conditional. Such scenarios require path-sensitive analysis such as symbolic execution, or enhancing with expressive reasoning methods such Symbolic Finite Automata (SFA), which can model symbolic transitions between states to model stateful behaviors of business models. Of the 3 FP unique to our full technique, we find that the reason is due to hallucinations in the initial annotation. Particularly, when performing the hallucination remedy procedure, GPT fails to recognize or fails to fix the hallucination. Solving this problem requires improving upon the reasoning trace generation. We will leave resolving both to our future work.
Further experiments. To further test the effectiveness of our technique, we run our system on 8 very recent real world smart contracts released in 2024 containing accounting bugs. We do not perform all of the
any human annotation, and run system directly, allowing GPT to perform annotation entry variables. Our technique can detect 6 of the accounting bugs that are in scope, including zero-day bugs. We have made reports the zero-day bugs and submitted them to the developers. Our system reports 6 FPs. We argue that this demonstrates the effectiveness of our system even in latest projects. Details can be found in Amppeddixx J
4.3 Research Question What the efficiency of the technique?
To evaluate the efficiency of the system, measure the cost of evaluation the main dataset. Regarding the monetary cost of the system, each project requires an average of 110645 tokens to run. Using the API pricing of the GPT3.5-Turbb model, we find that the average cost to project is 110645- x $0.5 1/100000000 $0.05. Regarding the time to run the system, we find that each project requires an average of 115.19 seconds run, approximately 2 minutes.
such
4.4 Research Question 4: What is the effect of fine-tuning and providing few-shot examples in our system?
We examined the impact of fine-tuning GPT-3.5 and providing few-shot examples for our system to identify accounting bugs. We also studied the benefits of using fine-tuned GPT-4.o mini model instead of the GPT-3.5 turbo model. We used 50 fine-tuning examples covering all the supported financial types and those without financial meanings. Then, we evaluated our system with different settings, namely with and without fine-tuning, with and without few-shot examples in prompting..

---

We include only one setting for GPT-4 due to the high cost of fine-tuning and using it. The results can be found in Table 3] In the table, the first column labelled "Model" shows the settings used for each experiment. The second column labeled "True Pos." shows the true positive bugs detected during each experiment. The column labeled "False Pos." " lists the number of false positive warnings reported. The column labeled "Iters." shows the total number of times that the reflection process was run. Finally, the column labeled "Correct Annotations" shows the total number of annotations that match the human-labeled annotations.
Observe that the experiment on row 5 with both fine-tuning and providing few-shot examples to the GPT 3.5 model improved annotation accuracy from 75.6% (62/82; the annotation accuracy of our system located on row 2) to 78% (64/82). Only fine-tuning with no few shots on row 4 performs worse than our default setting. No fine-tuning and no few shots, simply using the GPT 3.5 model directly, on row generates many more false positives and requires more calls of the reflection process. Using the fine-tuned GPT4.o model has the best performance in terms of higher annotation accuracy of 86.6% (71/82) at the expense of higher fine-tuning and inference costs. Note that the annotation accuracy changes lead to changes of downstream bug finding. However, the influence may not be proportional because the financial types involved in the bugs are not evenly distributed. That is, the incorrect annotations lie in variables unrelated the bugs.
5 Limitations (see Appendix K
Related Work
Rule-based Systems and Rule-base reasoning. Rule-based systems are long-established field in artificial intelligence. Early work such as [HayesRothh 1985] and Golding and Rosenbloom [1991] derive rule-based systems as way to modularize knowlede wewle Rules-based systems have been used to enhance LLMs, such as in generating business insights Verteel and Rumiantsau (2024) and
visual data processing [Shrann et al. [2023]. J. In the context of analysis, rule-based reasoning can be particularly effective, such as in anomaly detection Preece et al. [1992), computer penetration
detection Ilgun et al. [1955], and reverse engineering of code Alnusair et al. [2014).
LLMs in Program Analysis The use of LLMs program analysis has gained significant attention in recent years. LLMs leverage extensive datasets and sophisticated neural network architectures to understand and generate human-like text. They have seen usage in tasks such as code understanding
Nam et al. [2244, ). testing [eng et al. [2023] and detecting vulnerabilities Cheshkov et al. [2023). In relation to our work, LLMs such as GPT have even been used to detect vulnerabilities in smart
contracts. Sun et al. [2024] evaluate the vulnerability reasoning capability of LLMs through prompting approaches such as Chain-of-Thought (CoT). Xia et al. [2024] utilize GPT to verify adherence of smart contracts to set of standards known as the Ethereum Request for Comment (ERC). Sun et al. [2023] perform vulnerability detection via function-leved abstraction. In comparison, our system focuses on the detection of accounting bugs in smart contracts.
Accounting Bugs in Smart Contracts. recent technique was proposed in Zhang 2024 to detect accounting bugs using type system. It relies on intensive manual annotations of initial variable types. A lot of human efforts are hence needed to apply it to a new contract. The technique has inspired our rule-based reasoning part. To some extent, it corresponds to the baseline presented in our evaluation section. In contrast, our technique proposes the novel integration between LLM annotation and rule-based reasoning, enabling automation. It also uses reasoning traces to reduce hallucinations.
7 Conclusion
We develop a hybrid LLM and rule-based reasoning system for the detection of accounting bugs in smart contracts. We utilize the understanding capability of LLMs like GPT to perform annotation of financial meanings for variables. We further utilize based reasoning to propagate financial meanings and check for the correctness of operations. We implement a remedy technique for hallucinations, which relies on the generation of reasoning traces. Our results achieve 75.6% accuracy in annotations against those performed by humans, and detects 90.5% of the accounting bugs. Furthermore, our system detects 2 zero-day accounting bugs, which have been reported to the developers.
10

---

References
Abracadabra. 2024. Abracadabra Smart Contract. abracadabra- -oney
Awny Alnusair, Tian Zhao, and Gongjun Yan. 2014. Rule-based detection of design patterns in program code. International Journal on Software Tools for Technology Transfer 16 (2014), 315- 334.
Luca Keurer-Kellner, Mark Niklas Müller, Marc Fischer, and Martin Vechev. 2023, Prompt Sketching for Large Language Models. Ntv::311..09923 http:://aax...r.rrrr//////.1...0
Bitkan. 2024. Number Smart Contracts, httg:://bttaaa.ccoo/lar///oooo-a many-smart, otw.://--oo-.iiiiaaaa-a---------
Contracts on-
Priyanka Bose, Dipanjan Das, Yanju Chen. Yu Feng. Christopher Kruegel, and Giovanni Vigna. 2022. Sailfish: Vetting smart contract intiiiiiiionnnnnc bugs in seconds. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE.
Anton Cheshkov, Pavel Zadorozhny, and Rodion Levichev. 2023. Evaluation of ChatGPT Model for Vulnerability Detection. arXiv:2304.07233 [es.CR]
Jaeseung Choi, Doyeon Kim, Soomin Kim, Gustavo Grieco, Alex Groce, and Sang Kil Cha. 2021. SMARTIAN: Enhancing smart contract fuzzing with static and dynamic data-flow analyses. In ASE. IEEE.
Code4rena. 2024. Code4rena. htttp://coderrrea.ccmm
Consensys. 2024. Blockchain Technology Solutions.
httts::/consensss..et
Ron Cytron, Andy Lowry, and Kenneth Zadeck. 1986. Code motion control structures in high- level languages. In Proceedings of the 13th ACM SIGACT-SIGPLAN symposium on Principles of programming languages. 70-85.
DefiLlama. 2024. Defillama.
attps://deffs httee///ddelllaaaaccmmmtattii
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. arXiv:2212.14834 [cs.SE]
Josselin Feist, Gustavo Grieco, and Alex Groce. 2019. Slither: a static analysis framework for smart contracts. In WETSEB @ ICSE. IEEE/ ACM.
Andrew R Golding and Paul S Rosenbloom. 1991. Improving Rule-Based Systems Through Case- Based Reasoning.. AAAI, Vol. 1. 22-27.
Frederick Hayes-Roth, 1985. Rule-based systems. Commun. ACM 28, (sep 1985), 921-932. httt::://6o..orr/111145 00t.:0111///24444220000
Jing Huang, Kuo Zhou, Ao Xiong, and Dongmeng Li. 2022. Smart contract vulnerability detection model based on multi-task learning. Sensors (2022).
Koral Ilgun, Richard Kemmerer, and Phillip A Porras. 1995. State transition analysis: A rule-based intrusion detection approach. IEEE transactions software engineering 21, (1995), 181-199,
Jiao Jiao, Shang- Lin, and Jun Sun. 2020. A Generalized Formal Semantic Framework for Smart Contracts. Fundamental Approaches to Software Engineering 12076 (2020), 75 96.
Neptune Mutual. 2024. Abracadabra Exploit.
httpp::/meppturentttall.oom////
how-war
Satoshi Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system. (2008).
Daye Nam, Andrew Macvean, Vincent Hellendoom, Bogdan Vasilescu, and Brad Myers. 2024. Using an Il to help with code understanding. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1-13.

---

OpenAl. 2024. 2024. OpenAl GPT T.rboo http:://jaatttmmnooooaa..//oooop 3-b-turbo
Alun D Preece, Rajjan Shinghal, and Aida Batarekh. 1992. Principles and practice in verifying
rule-based systems. The Knowledge Engineering Review 7, 2 (1992), 115-141.
Akond Rahman, Chris Parnin, and Laurie Williams. 2019. The seven sins: Security smells in infrastructure as code scripts. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 165-175.
SP Sharan, Francesco Pittaluga, Mammohan Chandraker, al. 2023, Llm-assist: Enhancing closed- loop planning with language-based reasoning. arXiv preprint 2XXi:24011.00255 (2023).
Chaofan Shou, Jing Liu, Doudou Lu, and Koushik Sen. 2024. LLM4Fuzz: Guided Fuzzing of Smart Contracts with Large Language Models. arXiv preprint arXiv:2401. 11 (2024).
Chaofan Shou, Shangyin Tan, and Koushik Sen. 2023, ItyFuzz: Snapshot-Bssheeee Fuzzer for Smart Contract. urXiv:2306.17135 [cs.CR] httpge//aarai.oorraa///33.111112
Solidity. 2023. Solidity.
Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, and Yang Liu. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning, arXiv preprint arXiv:2401. 16188 (2024).
Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. 2023. When gpt meets program analysis: Towards intelligent detection of smart contract logic vulnerabilities in gptscan. arXiv preprint arXiv:2308.0330 003314 (2023).
Bryan Tan, Benjamin Mariano, Shuvendu Lahiri, Isil Dillig, and Yu Feng. 2022. SolTyye: refinement types for arithmetic overflow in solidity. Proceedings of the ACM on Programming Languages 6, POPL (2022), 1-29.
Aliaksei Vertsel and Mikhail Rumiantsau. 2024. Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Data. arXiv preprint arXiv:2404.1564 (2024).
Shuai Wang, Chengyu Zhang, and Zhendong Su. 2019. Detecting nondeterministic payment bugs in Ethereum smart contracts. Proceedings of the ACM on Programming Languages 3, OOPSLA (2019).
WETH. 2024. WETH. https://weth.io/
Valentin Wüstholz and Maria Christakis. 2020. Harvey: a greybox fuzzer for smart contracts. In ESEC/SIGSOFT FSE. ACM, 1398-1409.
Shihao Xia, Shuai Shao, Mengting He, Tingting Yu, Linhai Song, and Yiying Zhang. 2024. AuditGPT:
Auditing Smart Contracts with ChatGPT. arXiv preprint arXiv:2404.04306 (2024).
Brian Zhang. 2024. Towards Finding Accounting Errors in Smart Contracts. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1-13.
Zhuo Zhang, Brian Zhang, Wen Xu, and Zhiqiang Lin. 2023. Demystifying Exploitable Bugs in Smart Contracts. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (2023), 615-627.
12

---

User: You are given an assistant for identifying accounting bugs in smart contracts.
Accouniing bugs are detined as violations of deraain specitic business models.
following are csamples of accounting, bugs smart contracis, along with the reason that they are ertonenuss Determine such parterns can be found the new smart contract.
Example I: Code:f
[Code Motivating Example]
return ((baseReserve
quoteReserve)
totalShareSupplyys <Line 10)
Rensen: Line 10 is vulnerable flashloun attack due to ...]
Example 2: Code: {...
Please check for accunting buga in this smart contract: ( [Code for target smart contract
Figure 6: Format for Naive Prompt
User:
You given assistant for assigning finuncial meaning variables. Variables in conmect often have financial meaning.
The financial meaning "reserve" amount currency that strictly owned by the amart contract. The following are examples of "reserve" variables in smart contracts.
Example I: Code:]
quotoReserve
Currency Suppll pricoorraseennuuttcc
(Line
...} Variable: (quoteReserve) Reason: (Variable "quotcReserve""k"
Example 2: Code: f...
Answer YESINO if "reserve" is the financial meaning of Variable: (tAmount)] in Function: {
function tsuap (uint25b tAnount uint25b.
price) public returns (uint256
sxappedknount)....
Figure 7: Format for Initial Prompt for Raw Balance
Supplementary Material
A A Naive Approach
naive approach to directly prompt the LLM with few-shot accounting bug examples and ask it to find new bugs of a similar nature. Figure 6) shows the format of the prompt used for the naive approach. Similar to existing works utilizing GPT to detect vulnerabilities smart contracts [Sun et al. 2024]; Shou et al. [2024), high-level descriptions and few-shot examples of accounting bugs are provided within the prompt. Specifically, the blue box labeled "1" contains the definition of accounting bugs, while the red box labeled "2" shows the few-shot examples. Each few shot example follows two-part format: first, related code is provided, followed by the reason that it contains an accounting bug (i.e., what domain-specific property is violated in the code). The purple box labeled "3" where the code of a target smart contract provided as input to the L.M..
However, this approach does not suffice to detect accounting bugs, due to the observations that:
The code of smart contracts provided for analysis is often too large too costly to fit in the limited token context space. While is possible to filter non-relevant functions (i.e., functions which cannot be invoked by the public), the remaining code usually still retains the same issues. Furthermore, reducing the scope of the analysis to function-level analysis can miss accounting bugs that spread across multiple functions even files.
Challenge 1:
Challenge Most warnings produced by the method are often hallucinations from the model, and there is no existing technique to validate the warnings without human analysis.
13

---

B Prompt for LLM to Annotate Variable with Finnancial Meanings
Figure demonstrates the prompt format for the financial meaning "reserve". The text in the blue box labelled with "1" contains the definition of a "reserve". The text the red box labelled with "2" contains few-shot examples for "reserve" variable. The text in the orange box labelled with "3" contains the prompt used to determine whether or not the new variable (tAmount) should be assigned "reserve".
C Example of Initial LLM Annotation
Example. Referring to the example provided in Figure 3 there is only one entry function, tswap (). tswap() contains two parameters: tAmount and price, which are introduced on line 7. There is also one global variable, netRate, which is defined on line 2. Only these variables would be selected for initial annotation, and the correct annotations for the variables are "raw balance", "price", and "net rate", respectively.
D Rule-based Reasoning Algorithm and Example
Data: p the list of all operations in the smart contract, ordered according the CFG in SSA Result: nodeserror + a list of the operations in SSA form containing accounting error vulnerabilities
nodes error = D:
while p is not empty do nodeeur p.pop(); for x in nodeeur..pperanee do
if (x: is entry function parameter or global) and (a.. fin is None) then
x. x.fin LLMGGeFinanneeyyeecc))
end end
haserr, nodeseur. dest. fin RubBBsselifferene/oddcuaagoogppp otwp://wwo..o...oooooooooorrrror nodecur.dest finition = nodes if haseer is True then nodescrror aarer.dddndn (nodeeeee)rr)::
10
12
13
end
end
Algorithm 1: Rule-based reasoning pseudocode
We explain the usage of the rule set for analysis and propagation through the pseudocode presented
in in|Algorithm 1 The pseudocode shows the Algorithm for both sections 3.1 and 3.2 represents the list of operations in the smart contract in CFG order in SSA, and is provided as input. According to Figure is the first grey box. The output is list of nodes that potentially contain accounting bugs. According to Foiiurre 2] it is the second grey box. A node is defined as either an assignment x = or a computational operation x y/1 op y2. In the pseudocode, node.dest represents the destination of the node (e.g., æt), node.op represents the operation performed in a node (e.g., +, -, ..... and node.operaruds represent the operands (e.g., 91, y2) that appear in the node. For any variable æ; its financial meaning is denoted by x..ín and its parent node (i.e., the node that assigned its financial meaning) is denoted æ. de finition.
The loop from lines 2-14 traverse in order, and for each node, performs the following operations. An inner loop from line 4-13 traverses each variable (dubbed x) that appears in the node. If variable is an entry function parameter or global variable, the initial annotation discussed in 3.1 performed (lines 5-7). Lines 9-10 show the propagation. The financial meaning of the variable nodecur-dess the destination variable of the current node, is the result of rule-based inference given the operation of the node (nodecur-op) and its operands (nodeexx--operani) operation aligns with a rule in the rule set), the destination variable is assigned the resulting financial meaning of the operation. If there is no right operand (i.e., the node an assignment), the financial meaning of the destination variable is assigned the financial meaning of the singular operand. If the
or
If the operation is valid (i.e., the
14

---

operation is invalid (i.e., does not satisfy a rule), the node is added to the list of potential accounting bugs, nodes The definition of variable node assigned node
Example. We demonstrate how rule-based reasoning is used to propagate financial meaning as well identify potential accounting bugs using the example in Figure 3] We use the initial annotations of variable tAmount + reserve, price < price, and netRate < net rate to demonstrate how misclassified initial annotation (i.e., tAmount should be raw balance) can lead false postive reports.
We begin with the first operation in function tswap() of fRate getFeeRate line 6. Chaining the SSA nodes the operation results in: fRate [in tswap getFeeRate () feeRate [in getFeeRate ()] feeRate [Global] Since all of the SSA nodes are assign- ments, the financial meaning of fRate in tswap() is simply that of global variable feeRate, which fee rate (as assigned from the initial annotation.
Continuing to the second operation in function tswap() () of net bet_tAmount fRate) on line 11, we find that there is an improper usage of financial meanings, hence resulting in a potential accounting bug. Specifically, the violation of the rule-based reasoning occurs in the operation Amount _tAmmoutt * _netRate on line 5 within function applyFee(). In tswap() O), function applyFee() is called with two parameters, tAmount (initially annotated as a reserve), and fRate (annotated as net rate). Correspondingly, parameters t.mount and _netRate defined on line 4 are assigned the financial meanings of reserve and net rate, respectively. However, the rule-based reasoning is violated because multiplying a reserve and a net rate is not allowed, as demonstrated by rule Rg in FFigure 4 Intuitively, the net rate should be applied smart contract to earn profit, yet this operation shows it being deducting from the smart contract's balance (i.e., reserve), instead. Hence, this node is added the list of potential accounting bugs.
applyFeeetAmmunn, (tAmount,
line 5 during the operation t.mount * ..neRate. This is
user-owned balance (i.e., balance) in order for the
However, this accounting bug is present due to the misclassification of variable tAmount in the initial annotation. By simply replacing the misclassified financial meaning (i.e., reserve) with the correct one (i.e., raw balance), there will be no violation on line 5. Kule-based reasoning applied to the operation _tAmuunt net rate, as specified by Rs in the rules, leading to the assignment of net balance to the variable .f Amount. Additionally, the true accounting bug the operation net tmmuntt 12 can be discovered via continuing the analysis. Variable net net_ tAmount on line 11 is assigned a net balance, it is assigned to be Amount by the return statement net_tAmounts
_.neRRate yields valid computation of raw balance multiplied by
fRate on line
line 6. In the next operation, fRate on line 11, there is an accounting bug due to the operation net balance net rate. It is represented by RG in the rules. Intuitively, the fee has already been deducted from the users balance, as represented by the variable net tAmount being a net balance, hence the operation on line 12 is applying the fee twice, which is not allowed.
E Reasoning Trace Construction Algorithm
Data: e t an operation that results in an accounting error vulnerability in SSA Result: nodestrace + list of operation in SSA that represent the trace ofe nodestrace =
2 nodeswkkistt
3 nodeswkkistt aad((e) while nodeswkkistt is not empty do
nodecur = nodes witsssspssss:pp()):
modestrace-addd modes for z in node do nodes Suilist addd (x finnitio))
end end
Algorithm 2: Trace generation pseudocode
To obtain the list of related operations for the response trace, we develop a recursive algorithm that traverses through and tracks each node along with the nodes defining its operands. The pseudocode for the algorithm can be found in Algorithm 2.. The input is a node that potentially contains an accounting error. The output rodestrace is a list of nodes in the reasoning trace of e.
15

---

User:
are given reasoning accomnting bug. Identify any possible mislabelingss
Here are the inference riles for "x"
RI: "nw balanne"' xx "net rate": "net ballene"
R3: "net balance"s "not rite": Frnoe
Trace:
tAmouutt in tswap was typed by the LLM in entry function tswap( as a Raw Bal.
tAmount applyFee() tAmount (swapO, (Amouut Bal. parameter prop, on
netRattee (Net Rate) in
4: [Amooutt = tAmount _aetRate" ", [..mmmnn is Net Bal. in applyPeet) on line 5
due nale RI (Raw bal. x net rate)
net tAmount tswap() fAmount applyFee((, tAmount
due prop. line 11.
Error in operation "met tAmount (Net Bal.) .Rate (Net Rate)" in tswap( on line 12 due violation R3)
LLM: [MM::TTeee There misclussifiaaions. Variable "tAmount" is classified zaw balance, is used in operation with net rato produce net balance according to RI.)
Figure 8: Reasoning Trace Prompt and Response for Valid Bug
The loop from lines 3-8 contains the procedure applied to a worklist queue that initially contains e. First, the current node is popped into node The node is added to nodestrace- Then, for every variable x in continues until the worklist is empty.
operands, the parent node d. finition is added to the worklist. This process
LLM Reflection on a Real Bug Trace
The second reasoning trace in Figure ilis generated for the accounting bug discovered by the correct initial annotations. While the majority of the content is shared with that in Figiure 5]. the green box labeled with "1" representing inference rules contains the rule R3 net balance X net rate = Error instead of R2. In the trace, note that there more operations. First, with the relabelling of "reserve" to "raw balance", " we find that the fourth operation is success. The result that the variable fAmount in applyFee is now categorized is propagated to net ttnotunttt in function tswap() () on line 11 due to the return statement. In the sixth operation, we provide the new erroneous operation of net_ tAmount explain how it is an accounting bug due to the rule R3. In particular, net_tAmount was assigned the meaning of balance while fRate was assigned the meaning of net rate, and the result of their multiplication is an error, as defined by R3.
net balance. In the fifth operation, fAmouunt
fRate on line 12, and
The response from GPT can be found in the orange box labelled with "4". In particular, the response concludes that no reclassification should be performed. As evidence, provides the success of operation 4 due rule R1. As such, the accounting bug is considered real.
G Research Question 2: What is the distribution of financial meanings?
Figure 9 shows the distribution of the 6 financial meanings in the human annotations performed in the first experiment. Notably, the largest category comprises raw balance variables. This outcome aligns with our expectations. An entry function, defined as one that any user can invoke, often serves as the entrance for users to interact with the smart contract's underlying business model. Hence is natural that such functions have parameters representing raw balances or amounts of currency owned by users. All other financial meanings are less common in the sense that they are not usually controllable by users.
16

---

Raw Bsllnce
Reserve
Debt
15%
(10%
(8%)
(5%4
Figure Distributions of financial types
Table 4: Results on New Projects
Project Name
TP
FP
FN
NIS
Abracadabra
Thruster
0
Acala
0
0
Lybra
1
Polynomial
Trait Forge
0
N/A
Munchables
1
2
N/A
N/A
Basin
N/A
N/A
H Research Question 5: Can less powerful model than GPT-3.5 be used in the approach?
We replaced GPT-3.5 in our system with less powerful models such as CodeLlama-7BB -B-ins--t-ittttttbb and 138-instruct--ff. CodeL.lama-7B failed to produce formatted outputs many cases such that our pipeline could not parse its results properly. Similar problems as BBurrer-Keee Kellner et al. [2023]. 13B performs better in this matter. However, does not seem to understand the nuances of various financial meanings
observed in existing works such
with the few-shot examples. As such, it produced much worse annotation results than our default setting (41.9% vs 75.6%). This indicates the level of intelligence of the underlying LLM is quite important.
I Research Question 6: How sensitive is the LLM annotation to variable names?
We conducted an experiment in which we leveraged the Solidity compiler to rewrite variable names (to something as plain as vl, v2), without changing program semantics, to see how the annotation affected. We kept the function names. We then reran our pipeline with the modified code. Our results showed that the annotation accuracy degraded from 75.6% to 31.7%. The true positives (TPs) degrade from 19 to 14, and the FPs increase from 9 to 21. It is not surprising because without proper variable names, it is extremely difficult even for humans to decide the financial meaning of an operation, e.g., simple addition. Note that the substantial annotation degradation does not yield in proportional loss in TPs. This because for TPs, many incorrect annotations do not happen in the variables that are involved in the bugs. For FPs, the same operations (e.g., simple additions) multiple financial types such that even though a variable mis-annotated, and the system may not flag its operation an eπror.
allowed for
J Results on Eight New Smart Contracts
The results rumning our system on the 8 new smart contracts can found in Table 4 The name of the project can be found in the leftmost column. The number of true positive accounting errors can be found in the column labeled with "TP". The number of false positive accounting errors can be found in the column labeled with "FP" The number of false negative accounting errors can be found the column labeled with "FN". The number of accounting emoor belonging to other categories (i.e., pure math errors) can be found in the column labeled with "NIS" (Not-in-scope).
17

---

Out of the accounting bugs in the new projects, the technique was able to detect 6. Furthermore, one of the undetected accounting errors due an incorrect integer value, hence belonging the pure math category. Of the 6 accounting bugs, 2 are zero-day vulnerabilities, meaning that they were previously undiscovered. Our technique fails to catch 1 bug (FN) and reports false positive bugs (FP). It analyzed total of 10,097 additional functions. We find that the FN bug is due to the lack of modeling of a specific financial meaning (i.e., a reward rate, which represents the percentage of loan that is earned reward for the loaner). We find that the FP bugs due to similar reasons, where financial meanings are incorrectly assigned to variables which have financial meanings that are not modeled. We address both in the limitations section of our paper.
K Limitations
We note four main limitations: lucinations, inability to handle all accounting bugs, and inability scale other programming languages. The reason for the first is that we only model financial meanings in our system. However, real-world smart contracts may contain more financial meanings. As such our technique may miss some accounting bugs. This can be improved by expanding the system to accommodate a wider range of financial meanings. The reason for the second that we only apply the hallucination detection on potential accounting bugs. This can be improved by developing alternative methods to detecting hallucinations. The reason for the third is that our system cannot handle accounting bugs such as pure-math errors. To address this, more comprehensive approach that combines rule-based reasoning with additional analysis techniques may be necessary to capture these types of errors effectively. For example, our rules could be extended to capture problematic smells, as demonstrated in Raahman et al. 2019). The reason for the fourth that Solidity is the most popular programming language for smart contracts, and our implementation currently only supports that. Our system is implemented inside of Slither, a Solidity analysis engine that generates intermediate representations (IR) of smart contracts and provides rich set of APIs manipulate the IRS. That said, it is possible to implement the rule checker at the source level (using just a parser), which would allow easy extension to other languages. We leave this future work.
insufficient financial meaning coverage, inability to detect all hal-
18

---

The claims made in the abstract and introduction match the results reported in NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are provided
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification: Our work does not include theoretical results, all results have been achieved by our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The design of our architecture can be found in contains the main algorithm, well pseudocode to help explain certain algorithms (i.e. for how the trace for our hallucination remedy is generated).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have submitted our code, well as instructions how set up technique. Due to space limitations (the entire benchmark requires > 20 GB), we only submit subset of our benchmark.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Cade/SubeeSbiccccc cocbbbisssioon ssisssiiinn ssionollicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our work does not perform training, but we provide instructions on how to understand the test output.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We did not perform studies of statistical significance in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the specifications of the computer resources to run experiments in
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and determine that we conform with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the dangers of accounting bugs the real world
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks. We do not release any new models nor data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original owners of assets in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwtpsppppphhhoooooooeee com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have submitted our code, along with documentation on how to setup and run the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not involve crowdsourcing, nor performed research with human
subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not involve crowdsoureing, nor performed research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "CiNKvu15DT", "reviewer": "Reviewer_zmms", "summary": "This paper proposes a system to detect accounting error vulnerabilities in smart contracts. The key idea is a hybrid approach that combines LLMs and rule-based reasoning. In particular, it prompts LLMs to annotate the financial meaning of variables in smart contracts, and employs rule-based reasoning to propagate the information throughout a contract’s logic and to validate potential vulnerabilities.  To mitigate hallucinations, the system employs a feedback loop that provides reasoning traces of detected vulnerabilities to the LLM for iterative self-reflection.\n\nThe system is applied to 34 smart contract programs written in Solidity which are known to contain 21 accounting bugs out of a total of 40 bugs.  It is able to find 19 of the 21 bugs for a recall of 90.5%. It also achieves 75.6% accuracy in financial meaning annotations compared to humans. The reasoning trace-based hallucination mitigation reduces the false positive rate by 54.5%.", "questions": "Please see the Major points listed under Weaknesses, especially 2 and 3.\n\nI don't really expect much to change in terms of 1 even though I view it as the single most significant weakness of the work.  However, you could possibly comment on whether your simple rule-based reasoning could be replaced with a more sophisticated reasoning engine, such as an SMT solver.\n\nAnd while 4 would be nice to address, it is understandable if you did not have the time to apply the system to discover new bugs.", "limitations": "Yes, limitations are discussed (although in the Appendix), namely: 1) insufficient financial meaning coverage, 2) inability to detect all hallucinations, and 3) inability to handle all accounting bugs.\n\nHowever, I don't view these as the most significant limitations (unless by \"insufficient financial meaning coverage\" the authors are referring to the fact that they target only accounting bugs).  Likewise, 2 and 3 aren't significant since the approach has very high precision and recall.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The overall methodology is very elegant: it combines the strengths of LLMs and logic reasoning, using LLMs to overcome two key weaknesses of rule-based reasoning: annotation inference (pre-reasoning) and trace validation (post-reasoning).\n\n2. The overall empirical results are very strong with both high recall and high precision on a suite of real-world smart contracts.\n\n3. The paper is well-organized and easy to follow (with the exception of the flow being interrupted by frequent references to the appendices).", "weaknesses": "Major:\n\n1. It is well known that correctness is critical in the domain of smart contracts; as such, I was expecting a more comprehensive approach in terms of a) coverage of the kinds of bugs detected, and b) expressiveness of the reasoning engine.\n\nThe proposed system is very limited in both these aspects: it targets the relatively narrow class of \"accounting\" bugs -- which cover only 21 out of 40 in the authors' benchmark suite, and the checking is performed using very simple accounting rules.  The system is really a lightweight type qualifier checking engine that is more like a linter than a verifier.\n\n2. There aren't any realistic baselines in the evaluation, namely, existing static analysis and fuzzing techniques.\n\n3. The system uses prompting only as opposed to fine-tuning; the ~ 75% accuracy of annotations indicates that there is significant room to improve, and in any case it would have been nice to see the benefits of fine-tuning.\n\n4. The system has not been applied to find any previously unknown bugs.  The abstract talks about discovering 4 bugs in 5 recent projects, which made me think initially that these were new bugs; but it is later revealed that they were known bugs.\n\nMinor:\n\n- The presentation flow is interrupted by constant references to material in the the appendices.\n\n- Abstract: \"Finally, we apply the automated technique on 5 recent projects and discover 4 bugs\".  It is a bit misleading to say \"discover\" since these are known bugs.\n\n- line 3 of Fig 1: Currencey -> Currency\n\n- line 137: totalPairSupply -> totalShareSupply\n\n- I found the Example in Sec 2 to be simplistic; perhaps due to the fact that the approach itself does relatively shallow reasoning.\n\n- line 177: fianncial", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer zmms\n### Answers to Questions of Reviewer zmms\n**Q1. Replacing with rule-based reasoning with SMT solving and expressiveness of the reasoning technique**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;It is possible to enhance our system with an SMT solver. In lines 346-348 of our submission, we mentioned that the most common reason for FPs is the lack of path feasibility reasoning. This can be addressed by using a solver. From our experience, the difficulty of handling accounting bugs may not lie in using inference rules or the much more expressive SMT solving. Instead, it lies in deriving a set of rules/invariants that generalize across a wide spectrum of business models. Complex reasoning rules often run the risk of being too specific to a subset of business models. That said, we do think more expressive reasoning methods such as Symbolic Finite Automata (SFA), which can model symbolic transitions between states could be quite useful in modeling stateful behaviors of business models. It belongs to our future work. \n\n**Q2. Baselines**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C1 in the global response.\n\n**Q3. Benefits of finetuning**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C1 in the global response.\n\n**Q4. Unknown bugs**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C2 in the global response.\n\n**Covering more bug types**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Accounting bugs are quite different from other bugs as they are closely coupled with application specific business logics. Other bugs such as reentrancy and frontrunning have precise definitions and general oracles. To some extent, accounting bugs are analogous to the traditional functional bugs, whereas other smart contract bugs are analogous to buffer-overflow and divided-by-zero bugs. In traditional bug detection, functional bugs are a lot more difficult to deal with than others. In our work, we use type rules, which essentially denote the invariants that all business models should respect, to construct a general detector. Although there may be future designs different from ours, we believe detectors for accounting bugs would likely be different from those for other bug types. Moreover, just like functional bugs, accounting bugs are quite diverse and can be further classified to many sub-categories such as normalization errors, unit errors, interest errors, etc.\n\n**Example in Sec-2 is too simplistic**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;In general, accounting bugs are much more complex than that in Sec-2. We chose it for readability. The average reasoning trace for a real bug is 20 lines, where the trace for that example is only 4 lines. \n\n**Minor changes**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thanks for pointing them out. We will change all of them accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52PiHK7Qhu", "reviewer": "Reviewer_F88y", "summary": "The paper introduces ABAUDITOR, a hybrid system that combines LLMs and rule-based reasoning to detect accounting bugs in smart contracts. It leverages the semantic understanding capabilities of LLMs and rule-based logic for validating operations.", "questions": "- How many times were the experiments run?\n- Please clarify the origin of the 119 rules.\n- The manual evaluation process is not sufficiently explained; for example, the number of people involved and how conflicts were handled are not detailed. Please provide more information on these aspects.", "limitations": "- In addition to the concerns above, the authors need to justify how scalable the technique is across various programming languages, such as dynamically typed languages like python.\n- Please clarify if these 119 rules address all accounting bugs or just a selection of them.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is computationally inexpensive, facilitating the use of the model on large source code.\n- The combination of logical rule-based reasoning with LLMs reduces the model's tendency to produce inaccurate outputs.", "weaknesses": "- The primary weakness is that the only baseline used is human annotation; for example, it's not clear how the method compares to those in [1] and [2]. Therefore, the significance of the results remains uncertain.\n\n[1] Zhang, Zhuo, et al. \"Demystifying exploitable bugs in smart contracts.\" 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023.\n\n[2] Sun, Yuqiang, et al. \"Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis.\" Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer F88y\n### Answers to Questions of Reviewer F88y\n**Q1. How many times were the experiments run?**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Three due to the cost involved. We will clarify.\n\n**Q2. Origin of the rules**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We are smart contract auditors with years of experience. We have studied many business models (and their variants). These rules are invariants (i.e., properties held across different business models) we manually summarized. We are currently exploring using LLMs to extract a more comprehensive set of such properties. We will elaborate in the paper.\n\n**Q3. Manual evaluation process**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Three people were involved in the manual process. Two were annotating the variables independently, and the third served as a judge when inconsistencies occurred. \n\n**Comparison with [1] and [2]**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We cited [1] and the arxiv version of [2]  in our submission (lines 472-474 and 454-456). [1] is an empirical study of exploitable bugs in real-world  smart contracts and the effectiveness of existing tools on these bugs. It inspired us to work on accounting bugs as the authors found that such bugs are difficult, critical (as they may cause substantial financial loss), and beyond existing tools. [2] used prompt engineering (with Chain-of-Thought) to detect ten bug patterns. Most of them are not related to accounting bugs.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please also see C1 in the global response.\n\n**Scaling to other programming languages**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Solidity is the most popular programming language for smart contracts, and our implementation currently only supports that. The rule checker is implemented inside Slither, a Solidity analysis engine that generates intermediate representations (IR) of smart contracts and provides a rich set of APIs to manipulate the IRs. That said, it is possible to implement the rule checker at the source level (using just a parser), which would allow easy extension to other languages.\n\n**Comprehensiveness of the 119 rules**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Although these rules cover common business models, as stated in our limitations (lines 612-621), they are not comprehensive. For example, we currently do not model options in future contracts. We leave it for our future work.\n\n### References\n>[1] Zhang, Zhuo, et al. \"Demystifying exploitable bugs in smart contracts.\" ICSE’23.\n>\n>[2] Sun, Yuqiang, et al. \"Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis.\" ICSE’24.\n>", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z6xMBmtkU4", "reviewer": "Reviewer_uXwt", "summary": "The work proposes a system to detect accounting bugs in smart contracts by combining large language models (LLMs) and rule-based reasoning. The system first annotates the source code with LLMs to identify the parameters and global variables that are relevant to accounting bugs. Then, the system uses rule-based reasoning to detect accounting bugs by checking the annotated variables against a set of accounting rules. The system is evaluated on 23 real-world projects, and further on 5 recent projects from this year, showing that the system is able to detect accounting bugs effectively.", "questions": "* It is mentioned that the system only runs on those source code files with accounting bugs; however, in a real-world scenario, the vulnerability is often not known in the first place. What are the false positive rates of the system given a whole project's source code files, assuming the vulnerability location or existence is unknown?\n* What portion of the code is included when prompting the model to do annotations of parameters and global variables? Are the relevant code snippets or function implementations included in the prompt?\n* How does this work compare with other automatic or semi-automatic bug detection systems? The results table only shows the performance of the proposed system comparing the LLM annotation with human annotation and with/without reflection, and does not compare to other baseline systems.\n* How sensitive is the LLM variable annotation to variable names, function names, or comments in the code? Would removing or altering the natural language, such as renaming the variables, affect the annotation accuracy?\n* From Table 1, it seems that for some projects, the system didn't annotate correctly for some variables, and still performed the same as the human annotation in terms of TP and FP. Could you provide more insights on why this is the case? What is essential for the system to detect accounting bugs effectively if the annotation is not perfect?\n* The reflection process of fixing the annotation may iterate several times. How is it determined when to stop?", "limitations": "The limitations are properly addressed in the appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Ths system combines LLMs and rule-based reasoning to build an effective accounting bug detection system, overcoming the limitations of LLMs.\n* The system is able to detect accounting bugs effectively and is further demonstrated by a study of 5 recent projects released this year.\n* The technique is fast and cost-effective to run without developers manually annotating the source code, making it potentially useful for a wide range of adoption.", "weaknesses": "* The system is not compared with other bug detection systems, and the narrow focus on accounting bugs may limit the applicability of the system. For example, there are about 9 projects out of 23 that are considered \"not in the scope\" for the system, and even with human annotation, the system cannot detect the bugs. It is not clear how the limitations of the system compare to other bug detection systems.\n* The paper would benefit from more analysis on how the LLM is able to perform the annotation. For example, how sensitive is it to the syntax of the code, and is a model less capable than GPT-3.5 enough to do the annotation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer uXwt\n### Answers to Questions of Reviewer F88y\n\n**Q1. False positive (FP) rate when bugs are unknown**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C2 in the global response.\n\n**Q2. Code snippets included during prompting**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We prompt functions individually. We parse the output of each variable (including global variables). Our rules will propagate such information across functions (e.g., parameter passing from a caller to a callee and uses of the same global variable). In other words, we do not have to prompt a large code body (e.g., including relevant functions) because of the use of rules.\n\n**Q3. Other baselines**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C1 in the global response.\n\n**Q4. Sensitivity to variable names**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We conducted an experiment in which we leveraged the Solidity compiler to rewrite variable names (to something as plain as v1, v2), without changing program semantics. We kept the function names. We then reran our pipeline with the modified code. Our results showed that the annotation accuracy degraded from 75.6% to 31.7%. The true positives (TPs) degrade from 19 to 14, and the FPs increase from 9 to 21.  It is not surprising because without proper variable names, it is extremely difficult even for humans to decide the financial meaning of an operation, e.g., a simple addition. Please also see Q5 to find the explanation of why substantial annotation degradation may not yield proportional loss in TPs.\n \n\n**Q5. Incorrect annotation may not affect TP or FP, and What is essential for the system to detect accounting bugs effectively if the annotation is not perfect?**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;For TPs, many incorrect annotations do not happen in the variables that are involved in the bugs. For FPs, the same operations (e.g., simple additions) are allowed for multiple financial types such that even though a variable is mis-annotated, the system may not flag its operation as an error. We will elaborate. \n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;To detect an accounting bug, the annotations for the variables involved in the business operation should be correct. \n\n**Q6. Termination of reflection**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;It terminates when the LLM does not propose any new annotations for variables. Note that there are finite types of annotations and hence termination is guaranteed.\n\n**Can a model less powerful than GPT3.5 work?**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We tried CodeLlama-7B-instruct-hf and 13B-instruct-hf during rebuttal. CodeLlama-7B failed to produce formatted outputs in many cases such that our pipeline could not parse its results properly. Similar problems were observed in existing works [1,2]. 13B performs better in this matter. However, it does not seem to understand the nuances of various financial meanings even with the few-shot examples. As such, it produced much worse annotation results than our default setting (41.9% vs 75.6%). This indicates the level of intelligence of the underlying LLM is quite important. Please also see C1 in the global response, in which GPT4 has better performance than our default setting (using GPT3.5), namely, 86.6% annotation accuracy (versus 75.6%), fewer FPs, and a much smaller number of iterations before termination (2 vs 12), indicating that it has much fewer hallucinations. However, it also comes with a much higher cost.\n\n### References\n> [1] Beurer-Kellner, Luca, et al. \"Prompt Sketching for Large Language Models.\" ICML’24.\n>\n> [2] Beurer-Kellner, Luca, et al. \"Prompting is programming: A query language for large language models.\", PLDI’23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fi18VCTHuH", "reviewer": "Reviewer_2yGP", "summary": "This study is motivated by accounting bugs in a real-world setting. The authors focused primarily on the Flash Loan attack in smart contracts, which cost $50 million due to eight accounting bugs in DefiLlama. \n\nThe proposed method uses a Large Language Model (LLM), GPT3.5-Turbo, to trace a smart contract in the order of the control flow graph (CFG) and generate statements that include possible errors or LLM hallucinations.\n\n\nA follow-up step involves iterative self-reflection of the GPT via reasoning trace.", "questions": "1. How were the 119 rules derived? Open coding is a common method for extracting rules from software artifacts, followed by inter-rater agreement. See a sample paper [1].\n \n \nI am happy to raise my score if the authors address my questions and points mentioned in **Weakness*.*\n\nReferences:\n1. Rahman, A., Parnin, C. and Williams, L., 2019, May. The seven sins: Security smells in infrastructure as code scripts. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) (pp. 164-175). IEEE.\n2. Reis, S., Abreu, R., d'Amorim, M. and Fortunato, D., 2022, October. Leveraging practitioners’ feedback to improve a security linter. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (pp. 1-12).\n3. Rahman, A., Farhana, E., Parnin, C. and Williams, L., 2020, June. Gang of eight: A defect taxonomy for infrastructure as code scripts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 752-764).\n4. https://ai.meta.com/blog/code-llama-large-language-model-coding/\n5. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D. and Zhou, M., 2020, November. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1536-1547).", "limitations": "This paper mentions limitations of the proposed approach in Appendix I.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Existing vulnerability detection tools can not detect accounting vulnerabilities in smart contracts\n- Leverage LLMs to understand the business logic of smart contract\n- Analysis of reducing false positives, false positives are a major limitation of static analysis tool\n- New bugs were investigated (see Section 4.1).", "weaknesses": "- The proposed approach's generalizability is not clearly proved. \n The evaluation of the new smart contact bug is rather minor, with only six true positive bugs.\n \n- The proposed technique uses GPT3.5-Turbo rather than LLM trained on codes like Code Llama[4] or CodeBERT [5].\n\n **Minor Notes** \n\n-Reducing false positive alarms is a desirable property for security static analysis tools, otherwise, developers will be discouraged from using the tool because of the frequent false positive alarms. The authors could consider the paper [2] to reduce false positive rates of a static analysis tool.\n\n-The authors did not explain how to scale their approach. The authors may potentially scale their technique by taking into account the paper [3].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 2yGP\n### Answers to Questions of Reviewer 2yGP\n\n**Q1. How were the rules derived?**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We are smart contract auditors with years of experience. We have studied many business models (and their variants). These rules are invariants (i.e., properties held across different business models) we manually summarized. We really appreciate the pointer to the inspiring “seven sins” paper [1].  As mentioned in the limitations of our submission (lines 613-615), our rules are not comprehensive. A large empirical study involving more domain experts would help us complete the rule set. In addition, our rules could be extended to capture smells, as according to the paper, smells are quite problematic. \n\n**Generalization to unknown bugs**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Please see C2 of the global response.\n\n**Using CodeLlama and CodeBert**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We tried CodeLlama-7B-instruct-hf and 13B-instruct-hf during rebuttal. CodeLlama-7B failed to produce formatted outputs in many cases such that our pipeline could not parse its results properly. Similar problems were observed in existing works [2,3]. 13B performs better in this matter. However, it does not seem to understand the nuances of various financial meanings even with the few-shot examples. As such, it produced much worse annotation results than our default setting (41.9% vs 75.6%). This indicates the level of intelligence of the underlying LLM is quite important. Please also see C1 in the global response, in which GPT4 has better performance than our default setting (using GPT3.5), namely, 86.6% annotation accuracy (versus 75.6%), fewer FPs, and a much smaller number of iterations before termination (2 vs 12), indicating that it has much fewer hallucinations. However, it also comes with a much higher cost.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CodeBert generates embeddings for both code and natural languages and does not operate like an LLM that can answer our questions. As such, it cannot be used in our pipeline. Exploring integration of code-language models and LLMs is part of our future work.\n\n**Reducing FPs**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thanks for your pointer to the ASE’22 paper [4]. Although our tool does not produce that many FPs, reducing them is always highly desirable. We will open-source the tool upon publication and plan to solicit feedback from the community through a large-scale study like the one in the paper. The authors have strong connections with both the blockchain industry and the auditing community. These connections will be instrumental for the study. \n\n**How to scale our approach**\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Thank you for the pointer to the “Gang of Eight” paper [5]. The scale of the study in the paper and the impact are exemplary. We aim to achieve similar ramifications. Our work mainly followed the taxonomy proposed in [6], which studied 500 or so critical/exploitable real-world smart contract bugs in 2021-2022. We can clearly see the benefits of having a larger study than and a more thorough taxonomy than [6]. From our experience, accounting bugs continue to be a very important bug category, causing substantial monetary loss, and the current tool support is still insufficient. Moreover, we observe that recent smart contracts have increasingly complex business models which demand a more fine-grained study. We plan to conduct a larger scale study focusing on accounting bugs in the past five years and hopefully come up with a fine-grained taxonomy and a comprehensive set of invariants (similar to the rules in our submission). Similar to [1,4,5], we will leverage efforts/feedback from the community.\n\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;We will cite [1,4,5] and include the above discussion in the paper.\n\n\n### References\n> [1] Rahman, A., et al.  “The seven sins: Security smells in infrastructure as code scripts.” ICSE’19.\n>\n> [2] Beurer-Kellner, Luca, et al. \"Prompt Sketching for Large Language Models.\" ICML’24.\n>\n> [3] Beurer-Kellner, Luca, et al. \"Prompting is programming: A query language for large language models.\", PLDI’23.\n>\n> [4] Reis, S., et al. “Leveraging practitioners’ feedback to improve a security linter.”, ASE’22\n>\n> [5] Rahman, A., et al. “Gang of eight: A defect taxonomy for infrastructure as code scripts.” ICSE’2064).\n>\n>[6]  Zhang, Zhuo, et al. \"Demystifying exploitable bugs in smart contracts.\" ICSE’23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments.
## Common Concerns
**C1. Baselines**

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;According to [1] published in 2023, accounting bugs are beyond existing tools. That is why we did not compare our tool with others. Following the reviewers’ suggestions, we found two recent tools and performed empirical comparison during rebuttal. First, we compared ours with GPTScan [2], a recent LLM-based linter for smart contract bugs. It supports ten common bug patterns, including interest related accounting bugs. The results are shown in Table 1. The first column presents the bug types that it finds at least one instance. The second column lists the number of applications with such bugs, and the third the number of instances found. Only the first row (wrong-order-interest) belongs to accounting bugs, and the two reports in that row are false positives upon inspection. In other words, GPTScan could not find any of the bugs ours found.

**Table 1 GPT-Scan Results**:

| Bug Type      | Included Projects           | Total Instances                | 
|-------------|----------------------|---------------------|
| Wrong Order Interest  | 2       | 2    |
| Flashloan Price |  6     | 4         |
| First Deposit  | 3        | 3       | 
| Approval Not Revoked | 1 | 1|


We also compared our tool with ItyFuzz [3], a SOTA public smart contract fuzzer. Due to the limited time we have, we ran it for 1-4 hours for each project, corresponding to hundreds of millions of executions per-project. We observed that the coverage reached 32% on average. However, the fuzzer did not report any bugs.

In addition, we compared our tool with fine-tuned GPT3.5 and fine-tuned GPT4 mini. We used 50 fine-tuning examples covering all the supported financial types and those without financial meanings. We then evaluated our system with different settings, namely, with and without fine-tuning, with and without few-shot examples in prompting. The results are shown in Table 2. Due to the high cost of fine-tuning/using GPT4, we only include one setting for it. Observe that finetuned-fewshot (row 5) improved the accuracy of annotations from 75.6%(62/82 in row 2) to 78%(64/82); finetuned-no-fewshot (row 4) performed worse than our default setting (row 2); and no-finetuned-no-fewshot (row 3) has a lot more false positives (31 vs. 10) and more iterations. GPT4 has the best performance with annotation accuracy of 86.6% (71/82). However, its fine-tuning and inference costs are much higher than GPT3.5. Note that the annotation accuracy changes lead to changes of downstream bug finding. However, the influence may not be proportional because the financial types involved in the bugs are not evenly distributed. That is, the incorrect annotations lie in variables unrelated to the bugs. 

**Table 2 Results using Finetuned GPTs**

| Model       | True Pos.           | False Pos.  |         Iters.      |  Correct Annotations| 
|-------------|----------------------|---------------------|-----------------|------------|
| **Baseline** (_Manual_) | 19| 7| N/A| 82/82|
| **ABAuditor** (_Gpt3.5 w/ few-shot_)  | 19       | 10    | 12| 62/82|
| _Gpt3.5 no few-shot_ |  17  | 31    | 14 |  32/82 |
|  _Fine-tuned GPT3.5 no few-shot_ | 17   | 16  | 9| 39/82  |
| _Fine-tuned Gpt3.5 w/ few-shot_ | 19 | 9 | 7 | 64/82|
| _Fine-tuned Gpt4 mini w/ few-shot_ | 19 | 9 | 2 | 71/82 |


These results support our unique contributions. We will include them in the paper.

**C2. Finding unknown bugs**

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;During rebuttal, we further scanned 8721 additional functions in the five recent projects used in our submission (lines 355-359). In the submission, we did not scan them as they are not in the files that contain the known bugs. The tool generated 3 reports. However, manual inspection showed that they are not real bugs. On the positive side, our tool does not generate many false warnings in such a large-scale scanning. 

In addition, we scanned 75 functions in 3 new projects Munchables [4], Basin[5], and TraitForge[6] from the very recent Code4rena audit competitions whose results are still unknown. We chose them as they are business related. Due to the time constraints, we only scanned the files that define core business logics. We found 2 zero-day (unknown) accounting bugs. The first one  is in function _farmPlots() of file LandManager.sol in project Munchables. It adds a variable of a normalized balance type to another variable of an unnormalized type, which is problematic. The second is in function calcReserve() of file Stable2.sol in project Basin. To validate that they are real, we generated exploits (or POCs) that caused monetary loss. We have submitted the bug reports, which can be found in the one-page supplementary material. In addition to the two bugs, the scanning yielded 4 false positives.

To some extent, these illustrate the level of automation and the effectiveness of our tool.

### References
>[1] Zhang, Zhuo, et al. "Demystifying exploitable bugs in smart contracts." ICSE’23.
>
>[2] Sun, Yuqiang, et al. "Gptscan: Detecting logic vulnerabilities in smart contracts by combining gpt with program analysis." ICSE’24.
>
>[3] Shou, Chaofan, Shangyin Tan, and Koushik Sen. "Ityfuzz: Snapshot-based fuzzer for smart contract." ISSTA’23.
>
>[4] https://code4rena.com/audits/2024-07-munchables#top
>
>[5] https://code4rena.com/audits/2024-07-basin#top
>
>[6] https://code4rena.com/audits/2024-07-traitforge#top

Meta-review: Thanks for your submission to NeurIPS 2024. Overall, the reviewers recognize that the paper presents a novel hybrid approach that combines LLMs and rule-based reasoning to detect accounting error vulnerabilities in smart contracts. The experiment results demonstrate the effectiveness of the proposed method in vulnerability detection in real-world smart contract projects. The reviewers agree that this paper makes important contributions to the community and is valuable to be accepted.
Decision: Accept (poster)