How strong is weak mutation?

Mutation testing is a fault-based technique for unit level testing of software. Weak mutation was proposed as a way to reduce the expense of mutation testing. Unfortunately, weak mutation is also expected to provide a weaker test of the software than mutation testing. This paper presents results from an implementation of weak mutation, where we compared the effectiveness versus the efficiency of weak mutation. Additionally, we examined several options in an attempt to find the most appropriate way to implement weak mutation. Our results indicate that weak mutation can be applied in a manner that is almost as effective as mutation testing, and with significant computational savings. Indez Terms: fault-based testing, mutation testing, weak mutation, firm mutation, interpreters. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. This paper presents an experimental comparison of strong mutation testing to weak mutation. Weak mutation testing was proposed by Howden [How82] as a refinement and extension of his earlier work on algebraic testing [How78] and Fosters work [Fos80]. He suggested a modification to mutation testing [DLS78] that is computationally more efficient, but provides a less stringent test. The effectiveness of weak mutation tests as compared to those of strong mutation has been open to question, and we present experimental work to try to answer this question. Experimental work of this kind involves two stages: implementation of software to support the work and execution of the software to obtain results. Our weak mutation system is based on the Mothra mutation system [D GK+88]. We describe our implementation by presenting the major differences between Mothra and our system, the details of which are found in Lee’s thesis [Lee91]. The experiment is described sufficiently so that the results and the method can be evaluated and the experiment repeated. Since mutation is primarily a unit testing technique, we performed this experiment on program units (subroutines and functions). We feel that unit testing is the most productive phase to find faults in software and where the bulk of our automated efforts should be focused. Testing large, monolithic soft ware syst ems is less effective and @ 1991 ACM 089791-449-X191/000910200 $1.50 200 may be impractical for detailed, white-box, techniques. Additionally, even if the questions we pose for weak mutation are applicable to integration testing, it is important that they first be answered for unit testing. In this introduction, we summarize strong mutation testing, describe weak mutation testing, and then review some related work. 1.1 Strong Mutation Testing Mutation testing is a fault-based for unit testing of software that technique has been widely studied in recent years [DGK+88, DLS78, Ham77, K091, 0ff89]. Fault-based testing strategies are based on the notion of testing for specific kinds of faults. Mutation testing describes faults as simple syntactic changes to a program, called mutations. These mutations are used to create mutant versions of a test program, and test data is measured by the percentage of mutants that are shown to be incorrect (killed) by the test data. Some mutants are functionally equivalent to the original program and cannot be killed. A mutation score is the percentage of non-equivalent mutants that a test set killed. A test set is mutation-adequate if its mutation score is 10070. Although the number of possible faults for a given program can be large, mutation testing uses two principles to restrict the classes of mutations that are crested: the Competent Programmer Hypothesis [ABD+ 79] and the C’oupling Effect [DLS78]. The Competent Programmer Hypothesis states that competent programmers tend to write programs that are “close” to being correct. Although a program written by a competent programmer may be incorrect, it will only differ from a correct version by relatively few faults. The Coupling Effect states that a test data set that distinguishes all programs with simple faults is so sensitive that it also will distinguish programs with more complex faults [DLS78]. Neither the Competent Programmer Hypothesis nor the Coupling Effect are always true, but are probabilistic guidelines used by mutation testing, The coupling effect has been supported analytically by Morell [Mor84] and experimentally by Offutt [Of? B9]. Although the competent programmer hypothesis and coupling effect restrict the number of mutants generated by mutation systems, the number is still large. Acree [Acr80] derives a formula giving the number of mutations for a program that is bounded by iV2, where N is the number of variable references in the program. Mothra, for example, creates 951 mutants for the commonly studied 30 line triangle classification program TRITYP. Although current mutation systems automate most of the mutation process [DGK+88], executing this many mutants for such a small program is a significant computational expense. Howden [How82] suggested weak mutation as a way to reduce this computation. 1.2 Weak Mutation Testing In Howden’s terminology, P is a program, c is a simple component of I’, C’ is a mutated version of C’, and P’ is the mutated version of P containing C’. Weak mutation testing requires that a test case t cause C’ to compute a different (incorrect) value than C does on at least one execution of C’. Even though C’ may produce a different outcome, P’ may still produce the same results as P. Since strong mutation requires programs P and P’ to produce different output, weak mutation requires a less stringent, or weaker, test set than strong mutation. Howden’s original paper does not provide a precise definition of program component, but describes components as “normally corresponding to elementary computational structures in a program”. He gives five types: variable reference, variable assignment, arithmetic expression, relational expression, and Boolean expression. Although the idea is further refined in Howden’s book on functional testing [How87], components are still not defined precisely enough for an experiment of this type. It seems that just as the definition of strong mutation depends on the specific mutation operators implement ed, which in