RCABench: Open Benchmarking Platform
for Root Cause Analysis
Keisuke Nishimura, Yuichi Sugiyama, Yuki Koike, Masaya Motoda,Tomoya Kitagawa, Toshiki Takatera, Yuma Kurogome{keisuken, yuichis, yukik, masayam, tomoyak, toshikit, yumak}@ricsec.co.jp
Ricerca Security, Inc., Tokyo, Japan

2Fuzzers find a lot of bugs automatically● OSS-Fuzz: 8,900+ vulnerabilities and 28,000+ bugs
● ClusterFuzz: ~27,000 bugs in Google“We got inputs that cause crashes automatically.”“How do we process them? Manual analysis?”

3RCA (Root Cause Analysis) a.k.a Fault Localization
Automatic crash analysis

Candidates of Root Cause
Program(Souce Code)

Crashing Input

RCAtool○ Lines○ Basic Blocks○ …

4

Internal components of RCA tools
Crashing input

Generating more inputs
by fuzzing / from testcases

Non-Crashing input

Crashing Input

FeatureExtraction

Data
Augmentation

RCCandidates

User

Input datasets
Program(Souce Code)Statistically inferring
root causes

5

Evaluation of RCA techniques is challenging…
#1: Non-uniqueness of root cause definition#2: Tightly coupled RCA steps#3: Variance of Data Augmenatation

6

Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232

for (row = 0; row < num_rows; row++) {jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Original souce code

NULL-able

7

Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232

for (row = 0; row < num_rows; row++) {NULL-able

jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Original souce code

Where is/are theRC location(s)?

8Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232
RC location is…

HERE
for (row = 0; row < num_rows; row++) {jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Original souce code

RCA Tool #1

9Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232+ if (output_buf == NULL) {+ERREXIT();+ }for (row = 0; row < num_rows; row++) {jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Possible Patch #1

10

Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232
for (row = 0; row < num_rows; row++) {RC location is…

HERE
jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Original souce code

RCA Tool #2

11

Challenge #1: Non-uniqueness of root cause location
Multiple possible patches for CVE-2017-15232
for (row = 0; row < num_rows; row++) {+ if (output_buf == NULL) {+ERREXIT();+ }jzero_far((void *) output_buf[row],(size_t) (width * sizeof(JSAMPLE)));

Possible Patch #2

12

Challenge #1: Non-uniqueness of root cause location
Many possible candidates for root cause locations
The ground truth is …
Line 10 in function A

Line 50 in function A
Line 10 in function B

Line 80 in function A
Line 50 in function A
Line 10 in function B

Evaluator 1

Evaluator 2

Evaluator 3

13

Challenge #2: Tightly coupled RCA stepsAurora[Security ‘20]

AFLcemAuroraFE

ConcFuzz

VulnLocFE

Crashing input

VulnLoc[AsiaCCS ‘21]

Data
Augmentation

FeatureExtraction

Collecting inputs

Inferring root causes

Crasing input
Non-Crasing input

14

Challenge #2: Tightly coupled RCA steps
Aurora[Security ‘20]

AuroraFE

AFLcemNot Fully Modularized
Crashing input

VulnLoc

VulnLocFE

ConcFuzz[AsiaCCS ‘21]

Data
Augmentation

FeatureExtraction

Collecting inputs

Inferring root causes

Crasing input
Non-Crasing input

15

Challenge #3: Variance of Data Augmentation● Data Augmentation (internal fuzzing) TimeLonger time = More inputs = More accurate ??

● Initial seeds (crashing inputs) of Data Augmentation
Do seeds affect accuracy, like fuzzing *?

● Randomness of Data Augmentation
Quality of generated dataset may change?
* A. Herrera, et al. “Seed selection for successful fuzzing,” ISSTA ’21

16

RCABench: Open Benchmarking Platform for RCA
RCABench supports:

Existing challenges
1. Non-uniqueness RC

Predefined public RC locations

2. Tightly coupled RCA steps

Decoupled RCA steps

3. Variance of D.A.

Variance-aware evaluation

17

RCABench: Open Benchmarking Platform for RCA
7 real-world bugs/vulnerabilities with predefined public RC
from VulnLoc/Aurora’s
evaluations
But no public RC…

We plan to add more targets….

18

RCABench: Open Benchmarking Platform for RCA
Decoupling and modularization of D.A and F.E.
AuroraAbstraction Layer

AFLcem ✕ AuroraFE
AFLcemAuroraFE

ConcFuzz ✕ AuroraFE

VulnLoc
AFLcem ✕ VulnLocFE

Crash Input

ConcFuzz

Crasing input

VulnLocFE

Non-Crasing input

ConcFuzz ✕ VulnLocFE19

RCABench: Open Benchmarking Platform for RCA
Supporting variance-aware evaluation- Multiple initial crashing inputs for some targets- Multiple Data Augmentation times- Configuration based easy multiple benchmarking

20

Results of RCABench
RQ1: Which RCA technique is most accurate?

RQ2: Does D.A. time length affect accuracy?
RQ3: Do initial seeds affect accuracy?
RQ4: Does the randomness of D.A. affect accuracy?
Please see the paper for the detail 😀

21

RQ1: Which RCA technique is most accurate?
RCA techniques:●●●●AFLcem x AuroraFE = Aurora[Security ‘20]
ConcFuzz x VulnLocFE = VulnLoc[AsiaCCS ‘21]
AFLcem x VulnLocFE
Newly tested
ConcFuzz x AuroraFE

Answer:There was no obviously universal
technique that was most accurate for all targets.

22

RQ2: Does D.A. time length affect accuracy?
Answer:More accurate● Accuracy improved or did not change over time in many cases.
● There were a few cases in which the accuracy was degraded.

worsenimprove

23

RQ3: Do initial seeds affect accuracy?
Answer: Initial seeds sometimes affect accuracy.

24

RQ4: Does the randomness of D.A. affect accuracy?
Answer: Randomness in DA can lead to non-negligible
variances in accuracy.

Target #6

Target #1

25

Limitations and future work
● Mores statistical evaluation considering randomness○ D.A. randomness affected the RCA results (RQ4).
○ This threatens the validity of previous RCA evaluations.

● More abundant targets with diverse root causes○We plan to add more diverse targets■ Fuzzing benchmark (Magma, FuzzBench…)■ Real-world vulnerabilities

26Conclusion● Motivation: Evaluation of RCA techniques are challenging● RCABench (end-to-end benchmarking platform)○ Predefined and public root cause locations for seven targets○ Decoupling RCA steps (D.A. and F.E)
○ Variance-aware evaluation for Data Augmentation(DA time/initial seed/fuzzing randomness)

https://github.com/RICSecLab/RCABench

27

28

Limitations and future work
● Modular framework for fair and objective RCA evaluation○ Implementation differences can spoil fair comparisons.
■ Tracing: Intel PIN, DynamoRIO…
■ Language: Python, C++■ Misc: parallelization, file I/O, log…

○ Basic Blocks for implementation is needed.
c.f. modular framework for fuzzing [LibAFL, fuzzuf]

LibAFL: https://github.com/AFLplusplus/LibAFL fuzzuf: https://github.com/fuzzuf/fuzzuf

29

What affects the quality of the DA’s results?
● Number of inputs● Ratio of crashing/non-crashing inputs

Target #1

depends on combinationsof targets and methods.

Target #6

30

Target selection- Diverse Root Cause (Missing check, Incomplete check)- Diverse crash causes (heap overflow, UAF …)- Real-world software

Any contributions are welcome.

31

Question: What about targets with poor accuracy?
if statement at the patch point is executed regardless of the value of count.

if(TIFFGetField(input, TIFFTAG_JPEGTABLES, &count, &jpt) != 0) {- if (count >= 4) {+ if (count > 4) {int retTIFFReadRawTile;_TIFFmemcpy(buffer, jpt, count - 2);

Target #1: CVE-2016-10094

32

More precise evaluation for randomness- Average of rankings- User’s perspective- Is 1000 candidates of RC practical- Internal thresholds to reduce the output cadidates- More fundamental solution such as formalization is needed.

33

Question: Number of figures● RCA techniques shows the candidatesof root causes ordered by the level ofconfidence.
● The number means the ranks of theactual root cause we defined.
Candidate

1. a.c:100
2. b.c:200
3. a.c:105

Root cause
b.c:200
Rank 2 a.c:500

34

Non-uniqueness of root cause location
Multiple possibilities of root causes locations
function buggy(size) {data = malloc(size);
return data;}
function crashable(idx) {data = buggy(size);data[idx] = 1;}

Patch 1 in function buggy()if (data == NULL) {exception();Missing check ofmalloc()’s return value}

Patch 2 in function crashable()if (data == NULL) {exception();}Missing check of
buggy()’s return value