Choosing Error Models for OS Robustness Evaluations

The increasing use of software (SW) and its increasing complexity, which are necessitated by the need to flexibly control more systems of increasing complexity, raise the demand for rapid SW development. In order to reduce the development effort, large fractions of SW systems are constructed from reusable SW components. Due to their complexity, operating systems (OSs) are for instance frequently acquired as commercial-off-the-shelf (COTS) products. Acting as a mediator and abstraction layer between the system’s hardware and SW applications, the OS is a highly critical component in virtually any computer system. The problem that results from including reusable COTS OSs into safety critical systems is that the respective components are initially developed for specific applications or operational environments, but are subsequently reused in a broader range of application scenarios. Implicit application context assumptions persisting in the components’s implementation constitute severe dependability threats. Furthermore, COTS OSs are usually provided by their manufacturers without source code or sufficient documentation to derive adequate dependability evidence. For most SW safety standards, evidence based on measured product attributes without source code consideration does not suffice. However, black-box assessment techniques for the dependability assessment of COTS SW components and the identification of their vulnerabilities are continuously being discussed in the literature. This thesis addresses several open research questions related to the validity and efficiency of experimental SW robustness evaluations using fault injection (FI). FI is a technique that exposes the SW component under evaluation to unspecified operational conditions and monitors its behavior to ensure the absence of catastrophic failures in cases of extreme or unexpected conditions. The questions targeted in this thesis are related to the efficient modeling of such perturbations. A notion of error model efficiency is developed by proposing a set of metrics whose relevance is evaluated in an extensive case study, assessing the robustness of Windows CE . NET 4.2. It is demonstrated how the proposed set of metrics prove useful comparing a set of four different error models and choosing the most efficient one for different robustness requirements. The evaluated error models were carefully selected in order to provide useful insights into multiple theoretical issues related to FI-based evaluations, such as the differences and suitability of code-mutationand state-mutation-based FI. The obtained experimental results reveal significant differences among the applied error models. They suggest that codeand state-mutation-based models complement each other in experimental risk assessment and therefore should be used in conjunction. Code-mutation-based error models appear to prove more effective in detecting system outage vulnerabilities, but are more expensive in terms of implementation and execution time. A detailed analysis reveals previously unnoticed error propagation paths, which may render the application of fault containment wrappers (one of the most widely applied robustness hardening techniques) ineffective.