Estimating the Survival Rate of Mutants

Mutation testing is often used to assess the quality of a test suite by analyzing its ability to distinguish between a base program and its mutants. The main threat to the validity/ reliability of this assessment approach is that many mutants may be syntactically distinct from the base, yet functionally equivalent to it. The problem of identifying equivalent mutants and excluding them from consideration is the focus of much recent research. In this paper we argue that it is not necessary to identify individual equivalent mutants and count them; rather it is sufficient to estimate their number. To do so, we consider the question: what makes a program prone to produce equivalent mutants? Our answer is: redundancy does. Consequently, we introduce a number of program metrics that capture various dimensions of redundancy in a program, and show empirically that they are statistically linked to the rate of equivalent mutants. 1 EQUIVALENT MUTANTS Mutation testing is typically used to assess the effectiveness of test suites, by analyzing to what extent a test suite T can distinguish between a base program P and a set of mutants thereof, say P1, P2, . PN (Debroy and Wong, 2010; Debroy and Wong, 2013; V. and W.E., 2010; Ma et al., 2005; Zemı́n et al., 2015). A recurring source of aggravation in mutation testing is the presence of equivalent mutants: some mutants may be syntactically distinct from the base program, yet be functionally indistinguishable from it (i.e. compute the same function). Equivalent mutants distort the analysis of the test suite’s effectiveness, because when a test suite fails to distinguish a mutant Pi from the base program P, we do not know whether this is because Pi is equivalent to P, or because T is not sufficiently effective at detecting faults. Ideally, we want to quantify the effectiveness of a test suite T , not by the ratio of the mutants it distinguishes over the total number of mutants (N), but rather by the ratio of the mutants it distinguishes over the number of non-equivalent (distinguishable) mutants. Many researchers (Aadamopoulos et al., 2004; Papadakis et al., 2014; Schuler and Zeller, 2010; Just et al., 2013; Nica and Wotawa, 2012) have addressed this problem by proposing means to identify (and exclude from consideration) mutants that are equivalent to the base program. In this paper, we propose an alternative approach, which does not seek to identify which mutants are equivalent to the base, but merely to estimate their number. To do so, we consider the research question (RQ3) raised by Yao et al (Yao et al., 2014): What are the causes of mutant equivalence? Our answer: Redundancy in the program. Consequently, we define a number of software metrics that capture various forms of redundancy, discuss why we believe they are prone to produce equivalent mutants, then run an experiment that appears to bear out conjecture out. This is work in progress; we are fairly confident that our analytical arguments are sound, and we are encouraged by the preliminary empirical results. Following common usage, we say that a mutant has been killed by a test data set T if and only if execution of the mutant on T exhibits a distinct behavior from the original program P; consequently, when a mutant goes through test data T and shows the same behavior as P, we say that it has survived the test. Also, we use the term survival rate of a program P to designate the ratio (or percentage) of mutants of P that are found to survive the execution of test data T . For the purpose of our study, we use the semantic metrics introduced in (Mili et al., 2014), which we briefly review in section 2. In section 3 we discuss why we feel that the semantic metrics introduced in section 2 are good indicators of the number of potentially equivalent mutants that a program P may yield.