Automatic Collection of Failure Traces

Software bugs are a major cause of system downtime. To improve the quality of software, we have to find costeffective ways to locate and fix software bugs. Automatic tracing of real software executions in which software bugs are triggered would be most helpful. This would, for example, help us to focus on patching bugs that are triggered in practice and would speed up locating software bugs. Software bugs can roughtly be classified as either Bohror Heisenbugs. Collecting failure traces for Bohrbugs is easier in the sense that Bohr bugs are replayable. After a user detects a bug, the user could use an existing tool to record the application execution and for Bohrbugs should be able to reproduce the failure behavior. Such tracing would fail for Heisenbugs because by definition it is very unlikely to trigger the bug during retry. Our proposed approach is the following: (1) Software needs to be designed to be able to detect error states. (2) When an error is detected, we can retry a failed sequence of operations and switch on tracing during that retry. (3) If the new sequence discovers an error state, we have a valid error trace. If we do not trigger the bug again (i.e., this was a Heisenbug), we masked it (i.e., it is invisible to the user) and hence, there is no need to send a trace. Up to the early 80s, the MTTF of systems was dominated by the MTTF of the underlying hardware. While hardware reliability is still an issue, e.g., in large cluster systems, software has become the dominating factor on the MTTF that can be achieved. For example, Gray showed for the late 80s that software unreliability has become a growing influence on the MTTF of systems [2]. Software is still one of the dominating factors for system unavailibility. For example, [3] studied three Internet services and concluded that between 24% and 27% of the service failures can be attributed to software bugs. Economical issues (like time to market and the cost of program development) and technical issues (like the increasing software complexity) make it unlikely that the reliability of newly released software will increase substantially. A reasonable way too improve the “real-world” reliability of software is to use (1) software fault tolerance mechanisms to mask the effects of software bugs whenever possible and (2) to remove the most pressing bugs, e.g., the once that cannot be masked by the applied software fault tolerance mechanisms and that have non-benign effects. 1 What data to collect? The main objective of the reliability analysis of system failure data is to increase the availability and reliability of systems. Many bugs will stay in software because (R0) the developers have not been able to locate the bug. However, even if the developers are aware of bugs, there are several reasons why these bugs are not removed, e.g., (R1) a bug might be considered unlikely to be activated, (R2) the effects of the bugs might be considered benign, and (R3) the risk of introducing new bugs by patching a known bug might be considered too high (e.g., because major architectural changes might be required). We need to collect system failure data that permit us to (D0) locate bugs, (D1) show how often certain bugs are activated and (D3) determine the effects of these bugs. Locating bugs is not easy because during runtime we can only detect error states, i.e., we can only detect the symptoms of bugs. The fundamental issue of locating bugs is that the distance between the activation of a bug and the detection of an error state can be very long. For example, [1] studied the propagation latency of injected errors. It is not uncommon that it takes more than 100,000s CPU cycles until an error manifests itself, e.g., in form of a crash. The longer the delay between the activation of the bug and the detection of an error or failure, the more difficult is it to locate the bug. Software has to be designed and implemented such that the latency between the activation of bugs and the detection of an error is minimized. One promising approach to do that is to use assertion-driven development which is similar to test-driven development except that developers write assertions (instead of test cases) before they start writing code.