RCABench: Open Benchmarking Platform for Root
Cause Analysis
Keisuke Nishimura, Yuichi Sugiyama, Yuki Koike,
Masaya Motoda, Tomoya Kitagawa, Toshiki Takatera, and Yuma Kurogome
Ricerca Security, Inc.
Abstract—Fuzzing has contributed to automatically identifying bugs and vulnerabilities in the software testing ﬁeld. Although
it can efﬁciently generate crashing inputs, these inputs are usually
analyzed manually. Several root cause analysis (RCA) techniques
have been proposed to automatically analyze the root causes ofcrashes to mitigate this cost. However, outstanding challenges
for realizing more elaborate RCA techniques remain unknown
owing to the lack of extensive evaluation methods over existing
techniques. With this problem in mind, we developed an endto-end benchmarking platform, RCABench, that can evaluate
RCA techniques for various targeted programs in a detailed
and comprehensive manner. Our experiments with RCABench
indicated that the evaluations in previous studies were not enoughto fully support their claims. Moreover, this platform can beleveraged to evaluate emerging RCA techniques by comparing
them with existing techniques.

I.

I NTRODUCTION

Fuzzing has contributed to automatically identifying bugs
and vulnerabilities in the software testing ﬁeld; it is theprocess of randomly generating inputs and providing them to a
program to ﬁnd crashing inputs [1], [2]. Fuzzing is simple andeasy to deploy compared to other testing methods, and it canalso automatically and efﬁciently ﬁnd crashing inputs, helping
an enormous number of software programs improve their
quality. In fact, OSS-Fuzz [3], a popular fuzzing infrastructure,
is used in more than 650 open-source software (OSS) projects
and has found more than 40,500 bugs and vulnerabilities [4].
However, crash analysis, which is required after fuzzing,is difﬁcult and can be a bottleneck in making software testing
scalable. We must manually analyze these inputs later because
fuzzers only generate crashing inputs. The cost of analyzing
the crashing inputs generated by fuzzers is very high for two
reasons. First, fuzzers sometimes generate numerous crashing
inputs, the root causes of which are the same. For example,
a fuzzer generated more than 254,000 crashing inputs for
39 unique bugs in an experiment conducted in a previous
study [5]. Second, fuzzers randomly generate crashing inputs,
which means that the inputs can contain a signiﬁcant amount ofnoise. In other words, these inputs can include byte sequencesthat are not essentially related to the crash causes and hence

Workshop on Binary Analysis Research (BAR) 2023
3 March 2023, San Diego, CA, USA
ISBN 1-891562-84-3
https://dx.doi.org/10.14722/bar.2023.23004
www.ndss-symposium.org

can be removed. During crash analysis, analysts need todetermine which parts of the inputs are related to the causes,
which costs signiﬁcantly as the noise increases.
To reduce such costs in crash analysis, various automatedtechniques have been proposed in the ﬁeld of triage [2], [5],[6], [7], [8]. Triage is the process of analyzing and reporting
inputs that cause crashes [2]. In triage research, root cause
analysis (RCA) has attracted particular attention in recent
years. RCA, also known as localization or fault localization,is the process of identifying lines, basic blocks, or conditions
related to the root cause of a crash caused by a crashing input;
it provides developers with hints about the root cause. While
there are several different RCA approaches [9], [10], [11], thisstudy focuses on statistical fault localization, which infers the
program states correlated with the crash causes by contrasting
the execution of crashing and non-crashing inputs.
However, RCA is relatively underdeveloped among triage
topics. As discussed in detail later, state-of-the-art RCA techniques are not infallible. For example, in DeFault [8], theaverage false positive rate was 9.2 %. Furthermore, RCA is
not widely applied in industry, whereas deduplication, another triage technique, is integrated into major fuzzers, suchas AFL [12] and honggfuzz [13]. Thus, the existing RCA
techniques are neither accurate nor practical enough to be
widely used in the real world; this is because RCA techniques
have not been thoroughly improved owing to undiscovered
outstanding challenges, that is, the problems to be solved torealize more elaborate RCA techniques. One of the causesthat make such challenges still unknown would be the lackof extensive evaluation methods over existing techniques. We
found that the following three points were unnoticed andshould be considered to realize extensive evaluation methods:
Non-uniqueness of root cause deﬁnition There can beseveral possible patches to ﬁx a complex bug. If we deﬁne (thelocation in the source code of) the root cause as the locationthat should be ﬁxed, there can be multiple candidates for them.
It is not obvious for evaluators to deﬁne where root causes lie,
while it is certainly necessary for evaluating RCA techniques.
Despite this vagueness, existing techniques [6], [7], [8] did not
fully disclose the ground truth of their evaluations. This makes
it difﬁcult to reproduce the experiments in existing studies.
Decoupling RCA steps We found that the existing techniques
consist of two separable steps: data augmentation and feature
extraction. However, these techniques have not been evaluatedseparately to determine the performance of each step.

B. Feature Extraction

Variance-aware evaluation for data augmentation The existing techniques augment data using various fuzzing methods,
particularly those that are altered for data augmentation. The
evaluations need to consider the random nature of fuzzing.

Feature extraction analyzes the root cause using a dataset
generated by data augmentation. This process consists of two
steps. First, an analyzer records the state of the targeted
program at runtime while executing it with each input in thedataset. For example, in Aurora, executed instructions andvariable values are recorded. Next, the analyzer compares
traces between the crashing and non-crashing inputs andstatistically infers their differences. This difference is indicatedas the root cause. The existing techniques estimate the lines orbasic blocks related to the root cause. Aurora also estimates
predicates, that is, simple Boolean expressions that represent
the conditions to be met before a crash occurs. When actuallyused, analyzers do not report only the most likely root cause
candidate but instead multiple candidates in descending orderof the level of conﬁdence that analyzers assign to them. In thisstudy, we denote VulnLoc and Aurora analyzers VulnLocFEand AuroraFE, respectively.

Considering these three points, we developed RCABench,
an end-to-end benchmarking platform, to reveal the challengesof RCA. We provide a detailed and comprehensive evaluationof existing techniques for various targets and ﬁnd some cases
where exsting techniques cannot correctly analyze. Moreover,this platform can be used to evaluate the new RCA techniques
proposed in the future by comparing them with existing
techniques.
Overall, the main contributions of this work are as follows:•

We present three problems in the evaluation methodsof existing RCA studies.

•

We developed RCABench, an open-source benchmarking platform1 ; it provides a more standardized
evaluation and helps to summarize the outstanding
challenges in RCA.

•

To better illustrate this step, we take as an example CVE2016-10094 in LibTIFF, an open-source library. As shownin Listing 1, this vulnerability causes a heap buffer overﬂow
owing to an off-by-one error. Speciﬁcally, the program crashes
when the variable count is four and the statements inside thepatched if statement are executed. Generally, we refer to this if
statement as the root cause location and “count == 4 ” as theroot cause predicate. If this if statement appears frequently
in the program traces for crashing inputs and infrequently for
non-crashing inputs, the statement can be identiﬁed as the root
cause location. Similarly, if there is a distinguishable differencein the value of the variable count between two sets of traces,
the predicate can be identiﬁed.

Through experiments with RCABench, we identiﬁed
several insights into the pitfalls of existing techniques
and provided examples to motivate further research.
II.

ROOT C AUSE A NALYSIS

Root cause analysis (RCA), also known as localization or
fault localization, is a process of automatically identifying
lines, basic blocks, or conditions related to the root cause ofa crash; it aids in debugging and reduces the cost of the crashanalysis. We analyzed state-of-the-art RCA techniques [6], [7],[8] and identiﬁed two separable processes that are common toall. In this study, we refer to these as data augmentation andfeature extraction2 . In this section, we describe these in detail.

Listing 1: Developer patch for CVE-2016-10094 in LibTIFF.
−+

A. Data Augmentation
Data augmentation is the process of generating new crashing and non-crashing inputs from a given crashing input; thisis the ﬁrst process in RCA, and the generated inputs are usedas datasets for the feature extraction. Therefore, the quality ofthe dataset affects the RCA results.

i f ( T I F F G e t F i e l d ( i n p u t , TIFFTAG JPEGTABLES ,&c o u n t , &j p t ) ! = 0 ) {i f ( c o u n t >= 4 ) {i f ( count > 4) {i n t retTIFFReadRawTile ;TIFFmemcpy ( b u f f e r , j p t , c o u n t − 2 ) ;III.

C HALLENGES IN RCA E VALUATIONIn this section, we describe three previously unconsidered
points which are imperative to extensive RCA evaluations.

The existing techniques augment inputs using various
fuzzing methods that are specially altered for data augmentation. For example, Aurora [6] uses the crash exploration mode
provided by AFL [12], a typical coverage-guided fuzzer. In
this study, we refer to this as AFLcem. VulnLoc [7] proposed
ConcFuzz, a directed fuzzer for efﬁciently generating inputs
that exercise execution paths in the neighborhood of the pathtaken by a given crashing input, aiming at augmentation ofhigher quality. These methods use a single crashing input as theinitial seed and automatically generate crashing/non-crashing
inputs by randomly mutating it. In fuzzing, an initial seed isan input provided at the beginning of a fuzzing campaign.

A. Non-uniqueness of Root Cause Deﬁnition
Sometimes, there are multiple ways to ﬁx a bug; supposethat function B triggers a bug when it processes the data
produced by function A because the produced data conform torule X, whereas B expects rule Y. In this case, we can makeA comply with Y or B comply with X.
In such cases, if we deﬁne the root cause locations as thelocations in the source code that should be ﬁxed, there can be
multiple candidates for root cause locations. Therefore, it is
difﬁcult to correctly include all of them as the ground truth in
RCA evaluations. The evaluators currently deﬁne the groundtruth manually by coming up with all the possible patches, andhence, the evaluators sometimes miss some of the root cause
locations and use different ground truths.

1 https://github.com/RICSecLab/RCABench
2 Although these processes were not explicitly deﬁned in the RCA study,
inspired by similar efforts in the machine learning ﬁeld, we refer to them asdata augmentation and feature extraction.

2

Data augmentation time The existing studies did not evaluate
RCA techniques with various values of the time spent in data
augmentation. For example, in Aurora, the data augmentation
time was ﬁxed to only one value, either 2 or 12 h, dependingon targeted programs. However, the data augmentation time
can affect RCA results in multiple ways. We can generate adataset with a larger amount of crashing/non-crashing inputs
by spending more time in fuzzing; this may increase the dataset
diversity and improve the accuracy of feature extraction. However, it is also plausible that overﬁtting occurs, similarly to data
augmentation in machine learning, making the accuracy worse.

To illustrate more simply that there are multiple root cause
locations for a bug, we take CVE-2017-15232 in Libjpeg as
an example; this vulnerability causes a null pointer dereference
owing to the lack of a code to check for a null pointer. There
are several possible ﬁxes for this vulnerability, as shown in
Listings 2 and 3. The ﬁrst method, as shown in Listing 2,
is to insert a code to check for a null pointer before the forstatement. Another way, as shown in Listing 3, is to do thesame at the beginning within the for statement. Thus, the root
cause location is not uniquely determined, and identifying all
the candidates is difﬁcult; this can occur frequently with bugs
whose root cause is the absence of code.

Initial seed The existing studies prepared only one speciﬁccrashing input as an initial seed for fuzzing in data augmentation. For example, in VulnLoc, the initial seed is the input
used as a proof-of-concept when reporting vulnerabilities. In
fuzzing, the difference in the initial seeds is known to affectperformance, such as coverage and bug ﬁnding [14], [15].
Data augmentation using fuzzing may also affect the accuracyof feature extraction; therefore, the evaluator should prepare
several initial seeds. In addition, the existing studies have notfocused on the characteristics of initial seeds. For example, theinitial seed generated by a fuzzer tends to be noisier and more
complex than that generated by an analyst manually.

The existence of multiple root cause locations makes it difﬁcult to determine the ground truth and prevents the evaluation
results from being identical. In addition, the existing studies
did not fully disclose the ground truth, making evaluators have
difﬁculty reproducing the existing experiments accurately.
Listing 2: Developer patch for CVE-2017-15232 in Libjpeg.
++

i f ( o u t p u t b u f == NULL && num rows )ERREXIT ( c i n f o , JERR BAD PARAM ) ;f o r ( row = 0 ; row < num rows ; row ++) {j z e r o f a r ( ( v o i d * ) o u t p u t b u f [ row ] ,( s i z e t ) ( w i d t h * s i z e o f ( JSAMPLE ) ) ) ;Fuzzing randomness The existing studies have not considered the randomness of fuzzing in data augmentation. Theyevaluated each method using only a dataset from a single
fuzzing run. However, fuzzing is a highly stochastic process.
Hence, the generated dataset changes with each run, which canaffect the results of RCA. RCA techniques must be evaluatedmultiple times to address this problem. In fuzzing studies, it
has already been standard practice to run fuzzers multiple times
and evaluate the results statistically if possible [16]. The same
approach is required in RCA studies.

Listing 3: Another developer patch for CVE-2017-15232 in
Libjpeg.
++f o r ( row = 0 ; row < num rows ; row ++) {i f ( o u t p u t b u f == NULL)ERREXIT ( c i n f o , JERR BAD PARAM ) ;j z e r o f a r ( ( v o i d * ) o u t p u t b u f [ row ] ,( s i z e t ) ( w i d t h * s i z e o f ( JSAMPLE ) ) ) ;B. Decoupling Data Augmentation and Feature ExtractionIV.

As described in Section II, we found that state-of-the-art
RCA methods [6], [7], [8] consist of two separable steps: data
augmentation and feature extraction. However, the evaluationsin these previous studies did not decouple the data augmentation and feature extraction. Evaluators should investigate
the performance of each process independently because these
are two separable steps. For example, VulnLoc [7] proposed
ConcFuzz as a data augmentation method but did not evaluate
its relative performance by replacing it with AFLcem, an
existing alternative algorithm. In other words, it has not been
fully conﬁrmed that ConcFuzz generates datasets of higher
quality than AFLcem. Thus, the pure performance achievedby each step of proposed methods was not measured.

P ROPOSAL : RCAB ENCH

We propose RCABench, an end-to-end benchmarking platform that can run RCA techniques on selected bugs and check
whether their results match the predeﬁned locations of root
causes3 . The design of the RCABench was motivated by theinsights described in Section III. For each RCA technique,the data augmentation and feature extraction steps were decoupled, which enabled the comparison and evaluation of theaugmentation and extraction methods separately.
Currently, RCABench supports two augmentation methods and two extraction methods. The available augmentation
methods are AFLcem, used in Aurora [6], and ConcFuzz4 ,proposed in VulnLoc [7]. The available extraction methods are
the AuroraFE and VulnLocFE. We decoupled the augmentationand extraction steps and abstracted their interfaces so that eachaugmentation method could be connected to each extraction
method interchangeably since the original implementationsof AuroraFE and VulnLocFE are incompatible with ConcFuzz and AFLcem, respectively. Consequently, RCABench

C. Variance-aware Evaluation of Data Augmentation
In existing studies [6], [7], [8], evaluations did not consider
the variable characteristics of data augmentation. As describedin Section II, data augmentation generates a dataset for feature
extraction using fuzzing. Therefore, the quality of the dataset
may depend on the conﬁguration of fuzzers, such as the initialseeds and duration of a fuzzing campaign. The existing studies
have not dealt with this concern and have not been able toevaluate the impact of data augmentation on RCA results
in a variance-aware manner. Speciﬁcally, the following threevariables should be considered:

3 Some RCA techniques (e.g., VulnLoc and Aurora) indicate the candidates
for root cause locations as pairs of addresses of an assembly instruction andtheir corresponding source line numbers. RCABench uses line numbers forthe check because the addresses of instructions are too ﬁne-grained to decide
whether the address is a root cause location.
4 For ConcFuzz, the time spent in saving its internal data at the end is notincluded in the augmentation time.

3

can evaluate previously untested combinations of AFLcem ×
VulnLocFE and ConcFuzz × AuroraFE. Note that we used
only the root cause locations inferred by AuroraFE to compare
the performance of the techniques. Supporting and evaluating
the root cause predicates included in the outputs of AuroraFE
are left for future work.

TABLE I: Results of four RCA techniques in different
data augmentation times.

RCABench provides multiple popular real-world programs
containing actual bugs and vulnerabilities as targets of RCA.
Currently, seven targets have been prepared, all of which wereused in the evaluations of Aurora and VulnLoc [6], [7]. Weshow the lists and summary of their root causes in Table II. Ourcriteria for selecting targets are the availability of the sourcecode and the diversity of root and crash causes. As discussed in
the previous section, the selection of root cause locations canvary and be biased. Therefore, we ﬁrst registered to RCABenchseveral reasonable candidates for root cause location as groundtruth for each target. For stable re-evaluation, RCABench
publicly exposes these root cause locations, along with theirbrief explanations. RCABench also includes one or more initialseeds for each target to support augmentation methods that
require a crashing input as an initial seed. For targets withmultiple seeds available, we selected a crashing input used inbug disclosure or explanation to the developers as the baseline.
V.

This section describes the results of the proposed benchmark RCABench. Through benchmarking, we answered thefollowing questions:
RQ1: Which RCA techniques can perform accurate
analysis on each bug?

•

RQ2: Does the increase in data augmentation time
improve accuracy?

•

RQ3: Do initial seeds affect accuracy?

•

RQ4: Does the randomness of data augmentation
affect accuracy?

D.A. Time

A×A

C×A

A×V

C×V

#1

LibTIFF

15 m
2h
4h

15
9
9

9
33
47

2
2
2

13
12
12

#2

Libjpeg

15 m
2h
4h

–
–
–

–
15
14

23
12
12

32
23
17

#3

Libjpeg

15 m
2h
4h

16
7
6

–
–
–

2
2
2

1
1
1

#4

Libxml2

15 m
2h
4h

28
28
27

–
–
–

19
23
19

16
16
17

#5

mruby

15 m
4h
12 h

41
30
31

105
60
60

59
–
–

11
45
45

#6

readelf

15 m
2h
4h

1
1
1

4
1
1

4
4
4

4
4
4

#7

Lua

15 m
4h
12 h

–
–
32

–
N/A
N/A

1
–
–

1
N/A
N/A

“C × A” means ConcFuzz × AuroraFE and “A × V” means AFLcem ×
VulnLocFE.
“N/A”: No data were obtained. “–”: The root cause location did not appear in
the candidates reported by an RCA technique.

B ENCHMARK R ESULTS•

Programcandidates, in accordance with the original paper. “–” indicatesthat the correct answer was not included in the candidates
produced by the RCA technique. “N/A” indicates that no data
could be obtained because the technique tried to produce
quite huge ﬁles or took a long time for ﬁle I/O, which were
impossible to handle with our limited machine resources.
Table I indicates that no technique can predict the root
cause locations with high accuracy for all targets: while ConcFuzz × VulnLocFE successfully inferred the correct locationof the root cause in Target #3 with the highest accuracy, itspredictions for Targets #6 were less accurate than AFLcem
× AuroraFE. Our newly tested combination, AFLcem ×
VulnLocFE, outperformed the existing methods for Targets#1 and #2. However, they failed to ﬁnd the root causes ofsome other targets. This result implies that the characteristicsof the targets, which can be analyzed with high accuracy,
would be different for each RCA technique. This up-anddown situation depending on the targeted programs is similar
to fuzzer benchmarking, as seen in some results [17], [18].

All results shown here were obtained on a 256-CPU (AMDEPYC 7742) machine with 2TB memory and Ubuntu 20.04
operating system. To investigate the relationship between thedata augmentation time and the accuracy of RCA techniques,
we ran the data augmentation process up to an imposed time
limit and, in each of the ﬁrst 5, 15, 30, and 45 minutes andevery hour thereafter during the execution, RCABench saved
the dataset produced by the data augmentation at that time andanalyzed root cause with the dataset. For Targets #1-4,6, we
set the time limit to 4h, and for Targets #5,7, we extended it
to 12h in accordance with the evaluation of Aurora [6].

Answer: The technique that gave the highest rank to thecorrect root cause was different for each bug, and there wasno universal technique that was most accurate.

A. Which RCA techniques can perform accurate analysis oneach bug? (RQ1)

B. Does the increase in data augmentation time improveaccuracy? (RQ2)

Table I presents the overall comparisons of RCA techniques
for each target. The numbers in the table indicate the rankof the correct answer (actual root cause location) among thelocation candidates reported by each RCA technique, orderedby the level of conﬁdence assigned by the technique. A lower
rank indicates that the RCA technique can infer the root
cause location more accurately (“1” is the best score). In
this experiment, we set VulnLocFE to report up to Top-200

Next, we compared the results of each technique for eachtarget with different data augmentation times. Table I lists
the results in three data augmentation times (15m, 2h, 4h forTargets #1-4,6 and 15m, 4h, 12h for Targets #5,7) for eachtechnique and target pair. We also show two detailed examples
of how accuracy changes with time in Figure 1. In 19 cases,out of the 26 results excluding “N/A”, the accuracy remained
the same (e.g., Target #6 except ConcFuzz × AuroraFE)
4

1

AFLcem AuroraFE

30
40
50

20

Ranking

20

30
40

30m 1h2h3hData Augmentation Time30m 1h2h3h(b) Target #1

A (Crashes)
A (Non Crashes)
A (Crashes)
A (Non Crashes)# of (Non-)Crashing Inputs# of (Non-)Crashing Inputs

1500

A
A
C
C

1000
500
0

30m

1h

2h

3h

Data Augmentation Time(a) Target #1

4h

240

A, 366 bytes
A, 803 bytes
A, 3000 bytes
V, 366 bytes
V, 803 bytes
V, 3000 bytes

7000
6000
5000
4000

A
A
C
C

15

A
A
A
C
C
C

20

30m 1h2h3hData Augmentation Time4h

3030m 1h2hA, 324 bytes
A, 1247 bytes
A, 4825 bytes
V, 324 bytes
V, 1247 bytes
V, 4825 bytes

3h

Data Augmentation Time4h(b) Target #6

Fig. 3: The transitions of the accuracy over time for different
initial seeds. A × A, and C × V denote AFLcem × AuroraFEand ConcFuzz × VulnLocFE, respectively. In Target #4 and#6, the 803-byte and 324-byte seeds are their original seed,
respectively.

A (Crashes)
A (Non Crashes)
A (Crashes)
A (Non Crashes)3000#6 even in 15m, considering that a large number of samples
usually leads to high accuracy in classifying tasks.

2000
1000
0

10

25

(a) Target #4

Fig. 1: Accuracy vs. data augmentation time.

2000

180

360

4h

Data Augmentation Time(a) Target #3

2500

5
A
A
A
C
C
C

120

300

50

4h

1

60

10

Ranking

Ranking

10

1

ConcFuzz AuroraFE

Ranking

1

30m

1h

2h

3h

Data Augmentation Time

4h

C. Do initial seeds affect accuracy? (RQ3)(b) Target #6

To answer RQ3, we investigated whether the accuracychanged depending on the initial seed. For this purpose, we
ﬁrst ran AFLcem against Target #4 and #6 to produce various
crashing inputs. Then, for each target, we randomly selected
two different-length inputs from the produced inputs as initialseeds and evaluated the RCA techniques with them. We
selected these two targets because we could ﬁnd the crashing
inputs that were much smaller or larger than the original
crashing input. Thus, we believe that the newly produced initialseeds were very different from the original ones, in terms ofseed size and the method of producing them. Note that mostof the original seeds were created manually, which may makea signiﬁcant difference between the original and new seeds
with regard to whether noise exists in them, as described in
Section I.

Fig. 2: The number of generated inputs over time. A × A,
and C × A denote AFLcem × AuroraFE, and ConcFuzz ×
AuroraFE, respectively.
or improved as the data augmentation time increased (e.g.,
AFLcem × AuroraFE on Target #3 in Figure 1a).
However, the increase in data augmentation time worsened
the accuracy in some pairs of RCA techniques and targets,
such as ConcFuzz × VulnLocFE on Target #5. Figure 1b
shows an example of the deteriorating trend of ConcFuzz ×
AuroraFE on Target #1. In this example, the highest accuracy
was achieved up to 15m, and its accuracy declined thereafter.
Answer: While the accuracy improved or did not changeover time in many cases, there were a few cases in which theaccuracy was degraded.

Figure 3 shows accuracy versus data augmentation time
for different initial seeds. While, in Target #6, the accuracies
are little affected by the difference of initial seeds in both
AFLcem × AuroraFE and ConcFuzz × VulnLocFE, the two
added initial seeds of Target #4 affected the accuracy of
ConcFuzz × VulnLocFE, and one of them had a signiﬁcant
impact in particular. This result is consistent with the factthat the performance of a fuzzer can be affected by the initialseeds [14], [15].

Thus, sometimes data augmentation time eventually affects the accuracy. This fact indicates that somehow data
augmentation time affects the quality of datasets producedby data augmentation. To analyze how it affects the quality,
we inspected how the number and ratio of samples (i.e.,crashing/non-crashing inputs) in a dataset changes as data
augmentation time increases. Figure 2 plots the number ofsamples versus data augmentation time for Target #1 and #6.

Answer: The difference in initial seeds sometimes affects
accuracy. This implies that evaluators should make the initialseeds public to avoid cherry-picking and for reproducibility.

When looking at the ratio of samples produced by ConcFuzz × AuroraFE in Figure 2a, we see that the number of
non-crashing inputs starts exceeding that of crashing inputs
considerably in one hour. This would force feature extraction
methods to ﬁnd out root cause locations with imbalanced
datasets of crashing/non-crashing inputs, which is very similar
to a situation called imbalanced data classiﬁcation in themachine learning ﬁeld [19]. Generally, imbalanced data cancause poor accuracy in these classifying tasks. Actually, inTable I, the accuracy of ConcFuzz × AuroraFE perceptibly
decreases as the ratio gets imbalanced.

D. Does the randomness of data augmentation affect accuracy? (RQ4)
We observed the randomness effect on the accuracy by
evaluating the techniques ﬁve times. For ConcFuzz, different
seeds of its random number generator were set in each trial.
Consequently, we observed some non-negligible variances
as predicted in Section III, while the accuracy was very stablein some targets. Figure 4 shows the results of AFLcem ×
AuroraFE and ConcFuzz × VulnLocFE in Target #1 and #6.
While both RCA techniques in Target #6 and ConcFuzz ×
VulnLocFE in Target #1 had little divergence in their accuracy,

Another implicative fact is that the numbers of samples forTarget #6 shown in Figure 2b are large from the beginning,
compared to those of Target #1. Perhaps, this would have madeall the RCA techniques achieve the high accuracy in Target
5

1

5

5

10

10

Ranking

Ranking

1

15
20
25
30

30m 1h2h3hData Augmentation Time(a) Target #1

15
20
25

AFLcem AuroraFE
ConcFuzz VulnLocFE

4hprograms. For preparing more target programs with diverseroot causes, the programs constituting benchmarks known in
the fuzzing ﬁeld [20], [21], [17], [22] would be reasonablecandidates. In particular, Magma [21] and FuzzBench [22]provide a suite of programs that contain bugs found in the realworld and are widely used. Evaluating techniques with these
programs would further clarify their practical effectiveness.

30

AFLcem AuroraFE
ConcFuzz VulnLocFE

30m 1h2h3hData Augmentation Time4h

B. Possible Improvement of Existing Techniques(b) Target #6

In our experiments, we found some cases where RCA
techniques failed to analyze the root causes with high accuracy,
owing to their nature. A striking example is that ConcFuzz ×
AuroraFE ranked the root cause locations of Target #1 (CVE2016-10094) very low in RQ1 at data augmentation timesof 2h and 4h; this is probably because the ideal difference
that should exist between its generated crashing/non-crashing
inputs is whether or not a certain variable is at a certain value,
while ConcFuzz focuses on control ﬂow and the generated
inputs did not have enough diversity of values. This suggests
the possibility of further improvement of data augmentation
methods by considering features other than the control ﬂow.

Fig. 4: The effect of the randomness in data augmentation on
the accuracy. Five attempts were performed.
AFLcem × AuroraFE showed signiﬁcant divergence in theaccuracy in Target #1. Speciﬁcally, one of the ﬁve trials outperformed the others. If one cherry-picked only this trial, it could
be concluded that AFLcem × AuroraFE was more accurate
than ConcFuzz × VulnLocFE, although the two techniquesachieved similar accuracies on average; this suggests that it isimportant to evaluate RCA techniques multiple times and, if
possible, perform statistical analysis of the results.
Answer: For some combinations of techniques and targets, randomness in data augmentation leads to non-negligible
variances in accuracy. This result suggests that experiments toevaluate RCA techniques should be conducted multiple times
to reduce the effect of randomness as much as possible.
VI.

Another concern about the existing techniques is that
implementation designs differ among them. For example, totrace a program, Aurora [6] uses Intel PIN [23] and VulnLoc
uses DynamoRIO [24]. In addition, the data augmentation
method of VulnLoc is written in the programming language
Python, whereas that of Aurora in C/C++. Thus, there is apossibility that they have different performances owing to theirimplementation methods. If VulnLoc were written in C/C++,
VulnLoc would be able to run faster. In the fuzzing ﬁeld,
some frameworks have already been proposed so that different
algorithms can be implemented in a uniform way to solve
such a problem [25], [26]. For example, LibAFL [25] is aframework for building fuzzers in a modular manner. LibAFL
can reduce the cost of combining multiple fuzzing algorithms
into a single fuzzer and can fairly and objectively evaluate thealgorithms within the common implementation. If a similar
modular framework is presented for RCA, researchers would
be able to take fairer evaluations. Also, it would allow themto implement and evaluate a new algorithm more easily.

D ISCUSSION AND F UTURE W ORK

A. Threats to Validity
Although the evaluations on RCABench and our researchquestions provided thought-provoking claims in Section V, we
admit that two major threats may spoil some of the claims.
The ﬁrst one is the non-uniqueness of root cause deﬁnition.
As previously noted, we have been aware that multiple root
cause locations can be the ground truth, and mitigated risk bymaking our deﬁnition public and upgradable. However, this isjust a temporary countermeasure in the sense that particulartechniques can be underestimated still; it is possible that sometechniques report different valid root cause locations than ourdeﬁnition while the others report ours. A more robust andbetter deﬁnition and evaluation method of accuracy is desired.

VII.

The second one is fuzzing randomness. While RQ4 revealed that it undoubtedly threatens the evaluation validity inprevious studies, it also threatens the validity of our evaluation,
especially for RQ1 and RQ3 (note that RQ2 should be caredabout even within one trial, and hence its claim is valid inthat sense regardless of the threats). We acknowledge that our
results do not have a statistical signiﬁcance due to our limited
computational resources and should be carefully reviewed
by others. Nevertheless, it is sure that the effectiveness andsuperiority of the existing techniques are at least not so obvious
as the evaluations in the existing studies claimed. Moreover,the lack of a statistical signiﬁcance can be eventually resolvedin the future since we release RCABench as an OSS platform
and other researchers are also able to take benchmarks.

C ONCLUSION

Although fuzzing is a mature method for automaticallyﬁnding bugs, root cause analysis (RCA) techniques for discovered bugs are not full-grown. One of its causes is that theenvironment for a comprehensive evaluation of existing RCA
techniques was inadequate, making it difﬁcult to discover theoutstanding problems. Therefore, we developed a benchmark
platform, RCABench, for automatic and extensive evaluation.
Our experiments indicated that the evaluations in previous
studies were not enough to fully support their claims andfound some cases where the representative techniques failed
to analyze with high accuracy.
We believe that this initiative fosters future RCA researchby assisting researchers to propose and evaluate emerging RCA
techniques, as this study gives a glimpse of it. To shed light on
and help resolve the hidden challenges of RCA, we would liketo continue adding various targets and techniques and making
RCABench a more insightful platform.

While the above two points threaten the internal validity,
the external validity is another concern because seven programs are not enough to fully understand the behavior and performance of RCA techniques against a wide variety of targeted
6

ACKNOWLEDGMENT
We would like to thank the anonymous reviewers for theirhelpful feedback. We also gratefully acknowledge the authorsof Aurora and VulnLoc, who made their implementations
and experiment conﬁgurations publically available. This work
was supported by the Acquisition, Technology & Logistics
Agency (ATLA) under the Innovative Science and Technology
Initiative for Security 2020 (JPJ004596).
R EFERENCES[1] H. Liang, X. Pei, X. Jia, W. Shen, and J. Zhang, “Fuzzing: State of
the art,” IEEE Trans. Reliab., vol. 67, no. 3, pp. 1199–1218, 2018.
[Online]. Available: https://doi.org/10.1109/TR.2018.2834476[2] V. J. M. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz,
and M. Woo, “The art, science, and engineering of fuzzing: A survey,”
IEEE Trans. Software Eng., vol. 47, no. 11, pp. 2312–2331, 2021.
[Online]. Available: https://doi.org/10.1109/TSE.2019.2946563[3] K. Serebryany, “OSS-Fuzz - Google’s continuous fuzzing service foropen source software,” Talk at the 26th USENIX Security Symposium,
2017.
[4] Google,“Oss-fuzz,”
2022.
[Online].
Available:
https://github.com/google/oss-fuzz#trophies[5] Z. Jiang, X. Jiang, A. Hazimeh, C. Tang, C. Zhang, and
M. Payer, “Igor: Crash deduplication through root-cause clustering,”
in CCS ’21: 2021 ACM SIGSAC Conference on Computer andCommunications Security, Virtual Event, Republic of Korea, November
15 - 19, 2021. ACM, 2021, pp. 3318–3336. [Online]. Available:
https://doi.org/10.1145/3460120.3485364[6] T. Blazytko, M. Schlögel, C. Aschermann, A. Abbasi, J. Frank,
S. Wörner, and T. Holz, “AURORA: statistical crash analysis
for automated root cause explanation,” in 29th USENIX Security
Symposium, USENIX Security 2020, August 12-14, 2020.
USENIX Association, 2020, pp. 235–252. [Online]. Available:
https://www.usenix.org/conference/usenixsecurity20/presentation/blazytko[7] S. Shen, A. Kolluri, Z. Dong, P. Saxena, and A. Roychoudhury,“Localizing vulnerabilities statistically from one exploit,” in ASIA
CCS ’21: ACM Asia Conference on Computer and Communications
Security, Virtual Event, Hong Kong, June 7-11, 2021. ACM, 2021, pp.
537–549. [Online]. Available: https://doi.org/10.1145/3433210.3437528[8] X. Zhang, J. Chen, C. Feng, R. Li, W. Diao, K. Zhang, J. Lei,and C. Tang, “Default: Mutual information-based crash triage formassive crashes,” in 44th IEEE/ACM 44th International Conference
on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May
25-27, 2022. ACM, 2022, pp. 635–646. [Online]. Available:
https://doi.org/10.1145/3510003.3512760[9] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa,“A survey on software fault localization,” IEEE Trans. SoftwareEng., vol. 42, no. 8, pp. 707–740, 2016. [Online]. Available:
https://doi.org/10.1109/TSE.2016.2521368[10] H. A. de Souza, M. L. Chaim, and F. Kon, “Spectrum-based
software fault localization: A survey of techniques, advances, andchallenges,” CoRR, vol. abs/1607.04347, 2016. [Online]. Available:
http://arxiv.org/abs/1607.04347[11] P. Agarwal and A. P. Agrawal, “Fault-localization techniques forsoftware systems: a literature review,” ACM SIGSOFT Softw. Eng.
Notes, vol. 39, no. 5, pp. 5:1–5:8, 2014. [Online]. Available:
https://doi.org/10.1145/2659118.2659125[12] “American fuzzy lop (aﬂ),” https://lcamtuf.coredump.cx/aﬂ/.
[13] “honggfuzz,” https://honggfuzz.dev/.
[14] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L.
Hosking, “Seed selection for successful fuzzing,” in ISSTA ’21: 30th
ACM SIGSOFT International Symposium on Software Testing andAnalysis, Virtual Event, Denmark, July 11-17, 2021. ACM, 2021, pp.
230–243. [Online]. Available: https://doi.org/10.1145/3460319.3464795[15] D. Wolff, M. Böhme, and A. Roychoudhury, “Explainable fuzzer
evaluation,” CoRR, vol. abs/2212.09519, 2022. [Online]. Available:
https://doi.org/10.48550/arXiv.2212.09519

7[16] M. Böhme, L. Szekeres, and J. Metzman, “On the reliabilityof coverage-based fuzzer benchmarking,” in 44th IEEE/ACM 44th
International Conference on Software Engineering, ICSE 2022,
Pittsburgh, PA, USA, May 25-27, 2022. ACM, 2022, pp. 1621–1633.
[Online]. Available: https://doi.org/10.1145/3510003.3510230[17] Y. Li, S. Ji, Y. Chen, S. Liang, W. Lee, Y. Chen, C. Lyu, C. Wu,
R. Beyah, P. Cheng, K. Lu, and T. Wang, “UNIFUZZ: A holistic
and pragmatic metrics-driven platform for evaluating fuzzers,” in 30th
USENIX Security Symposium, USENIX Security 2021, August 11-13,
2021. USENIX Association, 2021, pp. 2777–2794. [Online]. Available:
https://www.usenix.org/conference/usenixsecurity21/presentation/liyuwei[18] Google,“Fuzzbenchreports,”
2020.
[Online].
Available:
https://www.fuzzbench.com/reports/[19] H. Kaur, H. S. Pannu, and A. K. Malhi, “A systematic review on
imbalanced data challenges in machine learning: Applications andsolutions,” ACM Comput. Surv., vol. 52, no. 4, pp. 79:1–79:36, 2019.
[Online]. Available: https://doi.org/10.1145/3343440[20] B. Dolan-Gavitt, P. Hulin, E. Kirda, T. Leek, A. Mambretti,
W. K. Robertson, F. Ulrich, and R. Whelan, “LAVA: large-scale
automated vulnerability addition,” in IEEE Symposium on Security
and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016.
IEEE Computer Society, 2016, pp. 110–121. [Online]. Available:
https://doi.org/10.1109/SP.2016.15[21] A. Hazimeh, A. Herrera, and M. Payer, “Magma: A groundtruth fuzzing benchmark,” Proc. ACM Meas. Anal. Comput.
Syst., vol. 4, no. 3, pp. 49:1–49:29, 2020. [Online]. Available:
https://doi.org/10.1145/3428334[22] J. Metzman, L. Szekeres, L. Simon, R. Sprabery, and
A. Arya, “FuzzBench: an open fuzzer benchmarking platform
and service,” in ESEC/FSE ’21: 29th ACM Joint European
Software Engineering Conference and Symposium on theFoundations of Software Engineering, Athens, Greece, August23-28, 2021. ACM, 2021, pp. 1393–1403. [Online]. Available:
https://doi.org/10.1145/3468264.3473932[23] C. Luk, R. S. Cohn, R. Muth, H. Patil, A. Klauser, P. G. Lowney,
S. Wallace, V. J. Reddi, and K. M. Hazelwood, “Pin: building
customized program analysis tools with dynamic instrumentation,” inProceedings of the ACM SIGPLAN 2005 Conference on Programming
Language Design and Implementation, Chicago, IL, USA, June
12-15, 2005. ACM, 2005, pp. 190–200. [Online]. Available:
https://doi.org/10.1145/1065010.1065034[24] D. Bruening, Q. Zhao, and S. P. Amarasinghe, “Transparent dynamic
instrumentation,” in Proceedings of the 8th International Conference
on Virtual Execution Environments, VEE 2012, London, UK, March
3-4, 2012 (co-located with ASPLOS 2012). ACM, 2012, pp. 133–144.
[Online]. Available: https://doi.org/10.1145/2151024.2151043[25] A. Fioraldi, D. C. Maier, D. Zhang, and D. Balzarotti, “LibAFL: A
framework to build modular and reusable fuzzers,” in Proceedings ofthe 2022 ACM SIGSAC Conference on Computer and Communications
Security, CCS 2022, Los Angeles, CA, USA, November 711, 2022. ACM, 2022, pp. 1051–1065. [Online]. Available:
https://doi.org/10.1145/3548606.3560602[26] “Fuzzing uniﬁcation framework,” https://github.com/fuzzuf/fuzzuf.

A PPENDIXTABLE II: Details of targeted vulnerabilities.
Program

CVE ID

Root Cause

Crash Causeheap buffer overﬂow#1

LibTIFF

CVE-2016-10094

off-by-one error#2

Libjpeg

CVE-2018-19664

incomplete check

heap buffer overﬂow#3

Libjpeg

CVE-2017-15232

missing checknull pointer dereference#4

Libxml2

CVE-2017-5969

incomplete checknull pointer dereference

#5

mruby

None

missing checktype confusion#6

readelf

CVE-2019-9077

missing check

heap buffer overﬂow

#7

Lua

CVE-2019-6706

missing checkuse-after-freeTarget #5 was not assigned a CVE ID but was assigned ID 185041 in the
HackerOne platform.

8