Fuzz testing for design assurance levels

With safety critical software, it is important that the application is safe and stable. While this software can be quality tested with manual testing, automated testing has the potential to catch errors that manual testing will not. In addition there is also the possibility to save time and cost by automating the testing process. This matters when it comes to avionics components, as much time and cost is spent testing and ensuring the software does not crash or behave faulty. This research paper will focus on exploring the usefulness of automated testing when combining it with fuzz testing. It will also focus on how to fuzzy test applications classified into DAL-classifications. INTRODUCTION In modern airplanes there are parts that need to communicate with each other. In this environment, data needs to be able to flow securely from one part to another. ARINC 653 is a specification for avionics real-time operating systems (RTOS) that explains how data can flow between these parts [1]. Applications resides inside partitions. When a partition is executing, it does not need to finish executing before sending data to a component. The components are parts of a interface which links the partitions with the hardware and operating system. The partitions, interface, operating system and hardware makes up a module. Modules has a component for monitoring and handling software errors and crashes, this is provided by system partitions. This is done to prevent component failures from propagating to other partitions or modules. Since it is important that the airplane can be controlled in a secure and comfortable way, identifying critical sectors is very important. This can be done by using Design Assurance Level (DAL). DAL is used to classify applications in avionics systems [2]. Classification is based on a risk analysis of the application. There are different DAL-classifications. Higher DAL-classifications has stricter requirements on software reliability, an error in the highest DAL-classification can cause wreckage. Contrary to the lowest DAL-classifications where a software failure will only have a minimal affect to the safety, control over the airplane nor does it strain the crew [3]. Software that is being tested will be referred to as the Software Under Test (SUT). While studies suggests that manual testing is widely used today [4, 5, 6], fuzz testing can make it possible to do more automated and broader testing of the SUT. Fuzz testing is usually not specified to a certain part of the SUT, but instead tests random paths throughout the whole SUT which might give unexpected results [7]. This kind of testing is achieved by using a variety of strategies and algorithms to mutate the input data fed to the SUT. American Fuzzy Lop (AFL)1 is a tool which uses a deterministic, whitebox variant of fuzz testing. This variant of fuzz testing tries to understand how the SUT works by using certain strategies and a feedback loop. It can then target specific paths in the software. Nondeterministic fuzzers2 will not use deterministic steps during mutation of input data but instead just randomize input. If a fuzzer can recognize important bytes in input data, it categorizes as a smart fuzzer [8]. Likewise, if it is aware of the application structure it will categorize as a whitebox fuzzer. AFL for example learns the application structure when instrumenting the SUT during compilation. This separates AFL from other fuzzers as it is categorized as a smart, deterministic and whitebox fuzzer. When the deterministic strategies are exhausted, AFL will dive into completely non-deterministic, random behaviour. Since deterministic strategies are employed in AFL, the input will be random to a lesser extent than what it would be for a non-deterministic fuzzer. Purpose The goal of this research paper is to evaluate what fuzz testing can contribute in testing of different DAL-classifications and outcome in amounts of errors, and implement a solution that can: 1. Generate code from C++-header files and XML-files. 2. Create a layer which links the SUT with AFL where data can flow continuously. 3. Test the generated code with AFL. 4. Log eventual errors found in the SUT. We are also evaluating how the architecture of the SUT affects the usefulness of fuzz testing and draw conclusions from the correlation between code coverage and outcome. Research questions Our main research question is if fuzz testing can be useful for finding errors in software that is written and tested according to different DAL-classifications, and can be broken down into following: • What is the correlation between amount of errors found with fuzz testing and code coverage? • What is the the relation between amount of faults found and DAL? 1http://lcamtuf.coredump. cx/afl/ 2A fuzzer is a program which employs fuzz testing