An Evaluation of Bug Finding Approaches

Bugs in software, such as buffer overflows, can often become entry points for breaches in software security. These bugs are often difficult to find manually. They can also be extremely costly to fix after software is released. To alleviate this there are automated bug finding tools. Several different bug finding methods have been developed from existing approaches. We test several implementations of these methods using the automated tools including bounded model checking (CBMC), static analysis (Coverity) with BugBench, and compare and evaluate the results. Introduction Software bugs are a very common entry point for security exploits. Integer overflows, buffer overflows, pointer errors, and any bug that affects memory can enable malicious users to force a program to run arbitrary code. Part of the problem with bugs is that they are quite difficult in practice to find manually. Due to their complexity nearly all software has bugs of various types. While these bugs may be very difficult to encounter within the intended use of the program malicious attackers may be able to craft inputs that exploit these bugs in a way that is potentially damaging. To compound this issue, bugs are much harder and more costly to fix once software has been deployed. One proposed solution to this problem is to create bug-checking tools to automatically detect and report bugs in software. Several different approaches have been taken to achieve this goal of automated bug finding. The two approaches we examine in this paper are Bounded Model Checking (BMC) and Abstract Static Analysis (ASA). BMC comes from hardware verification, and ASA comes from program analysis from the perspective of software testing. While these tools have several similarities in how they operate, they all employ theorem provers. For example, they have historical and implementational differences. We examine but do not analyze KLEE a Dynamic Symbolic Execution tool as grounds for continued work. The goal of this paper is to help increase knowledge on the effectiveness of these tools. We attempt to do this by testing each of the approaches, in the form of software tools, on a neutral software testing suite, BugBench [2]. We use CBMC, a bounded model checker; and Coverity, a static analysis tool; Klee, a dynamic symbolic executor, as representatives of their corresponding approaches. This paper is organized as follows: In section two we offer brief explanations of the approaches each tool uses as well as specific points on each of the tools. In section three we describe our method of evaluation as well as outline the current results of the project. In section four we discuss results as well as describe how the project means to expand in the future. Finally, in section five we conclude the paper. 2. Overview of Bug Finding Approaches Coverity’s Static Analysis software was created by and is constantly monitored by a professional team of engineers, and is licensed out to other companies to test their own software. Klee is rather successful implementations of Dynamic Analysis, and was used to find bugs that manual testing has not found in 15 years. CBMC is considered the best Bounded Model Checker, and is used within other verification and bug finding approaches. In the following sections we give brief descriptions of BMC, ASA, and DCE as well as interesting specifics on the tools we use, CBMC, Coverity, and Klee respectively.