Model-Based Mutation Testing with Constraint and SMT Solvers

Today, software is omnipresent in our everyday life. Computers and embedded systems are used naturally in telephones, cars, railway signalling systems, etc. For some of these applications, a malfunctioning is inconvenient – for others life-threatening. Hence, a thorough verification and validation of software systems is essential. In practice, it is mostly conducted via software testing. In the last decade, model-based testing has been identified as a promising approach to software testing as it allows for automated test case generation. Instead of manually designing many individual test cases, the test engineer creates a formal model of the system under test and uses tools to automatically derive test cases. A crucial matter in model-based testing is the choice of the test criterion specifying which test cases shall be generated. In this work, we follow a fault-centred approach. The original test model is syntactically altered to produce a set of mutated models. We perform a conformance check between the original and the mutated models. In case of non-conformance, we automatically generate a test case that reveals the non-conforming behaviour of the mutant. When executed on the system under test, these test cases will detect whether one of the faulty models has been implemented instead of the correct, original model. Hence, the generated test suite covers all of the failures caused by the model mutation operators and has a high chance of covering many additional similar failures. As a modelling formalism, we rely on action systems, which are well-suited to model reactive and non-deterministic systems. A main factor influencing the quality and performance of model-based mutation testing concerns the underlying conformance check, which is the main focus of this thesis. We designed and implemented an efficient test case generator for action systems based on two conformance relations: refinement and Input-Output Conformance (ioco). For refinement checking, we defined a predicative semantics for action systems that is close to a constraint satisfaction problem. This allows for utilising modern constraint and SMT solvers. We optimised our refinement checker to efficiently analyse a large number of mutated models. Experiments showed that our optimisations reduced the runtimes by up to 90% compared to our first implementation. Despite its efficiency, our notion of refinement is not completely satisfying the needs of model-based mutation testing. A more suitable conformance relation is ioco. However, previous work has shown that ioco checking is rather demanding in terms of runtime and memory consumption. Therefore, we use our optimised refinement check as a preprocessing step for the efficient computation of an underapproximation of an ioco test suite. Instead of performing a full ioco check between the original and a mutated model, we first perform a refinement check. Only in case of non-refinement, the ioco check is initiated from the point where non-refinement has been detected. In this way, the subsequent ioco check is more target-oriented. Our test case generator has been integrated into a tool chain that generates test cases from UML state machines. It has been applied to two case studies conducted with industrial partners from the automotive domain in the context of two research projects.