# Mutation Analysis Approach to Develop Reliable Object-Oriented Software

Authors: Monalisa Sarma
Journal: Advances in Software Engineering
 (2014)

---

## Abstract

In general, modern programs are large and complex and it is essential that they should be highly reliable in applications. In order to develop highly reliable software, Java programming language developer provides a rich set of exceptions and exception handling mechanisms. Exception handling mechanisms are intended to help developers build robust programs. Given a program with exception handling constructs, for an effective testing, we are to detect whether all possible exceptions are raised and caught or not. However, complex exception handling constructs make it tedious to trace which exceptions are handled and where and which exceptions are passed on. In this paper, we address this problem and propose a mutation analysis approach to develop reliable object-oriented programs. We have applied a number of mutation operators to create a large set of mutant programs with different type of faults. We then generate test cases and test data to uncover exception related faults. The test suite so obtained is applied to the mutant programs measuring the mutation score and hence verifying whether mutant programs are effective or not. We have tested our approach with a number of case studies to substantiate the efficacy of the proposed mutation analysis technique.


---

## Body

## 1. Introduction

Of late, computer applications have permeated heavily into every sphere of the daily life of an average person. Many of these applications are large, complex, and safety critical, requiring the software to be extremely reliable. An exception occurrence during a program execution is an abnormal computation state [1] and thus is a threat to the reliability of software. It is advocated that software should be tested adequately to trace if there is any exception in intended behavior [2]. To realize this software developers prefer exception handling mechanisms to be embedded inside code so that software can arrest and raise alarm whenever there is any exception and then cause of exception can be eliminated. Many reasons may be attributed to causes of exceptions such as erroneous input, hardware faults, logical errors in the code, semantic violation, linking errors, and limitation of system resources. It is reported that more than 50% of the operational failures that may occur in a system are due to faults in exception handling and recovery [3]. To cope with the system failures due to exceptions, most modern programming languages such as Ada, C++ and Java incorporate explicit mechanisms for exception handling. Syntactically, an exception handling mechanism consists of a means to explicitly raise an exceptional condition at a program point and a means of expressing a block of code to handle one or more exceptional conditions. In other words, an exception handling mechanism provides a way to separate code that deals with unusual situations and abnormal program termination.Exception handling constructs are responsible for the detection and handling of system conditions that could potentially lead to failure. Recent studies of Java programs indicate that exception handling constructs occur frequently and approximately 15%–25% of the classes contained exception handling constructs [4, 5]. Despite the frequency of their occurrences, the behavior of exception handling constructs is often the least understood and poorly tested part of a program [6]. Testing strategy to test object-oriented programs, in general, and exception handling constructs, in particular, is scarcely reported [7]. For testing exception handling constructs, some works have been reported [4, 6, 8]. Sinha and Harrold [9] proposed a class of adequacy criteria that can be used to test the behavior of exception handling constructs. Sinha and Harrold [9] described a methodology for applying the criteria to unit and integration testing of programs that contain exception handling constructs. However, to the best of our literature survey, no approach has been reported so far to generate exception handling related faulty programs and then to test the adequacy of test suites. Adequacy testing is an important issue because a test suite may not locate all exception points. For example, in Java, an exception can be raised implicitly. In that case, it is not possible to draw proper control flow graph [4, 10, 11]. Further, even if all exception paths are known from the control flow graph [4] and satisfy exception-coverage criteria [9], it needs to design test data manually for raising all possible potential exceptions. This manual test case generation is not only tedious [6] but raising exceptions is not always guaranteed. As a consequence we need to determine the adequacy of test cases to raise exceptions. In general, determining the path of exceptions is a difficult problem and manual synthesis of test data may not be sufficient to raise all exceptions. In this context, efficiency of a test suite is necessary to be evaluated in order to assure the reliability of a system under test. In summary, given a program, we are to test whether the exception handling constructs are sufficient to throw and catch all possible exceptions or not.Mutation analysis is a practical method to evaluate the quality of test suites [12–14]. In this work, we propose a mutation analysis method to evaluate test suites for testing exception handling constructs. We consider a set of mutation operators with which different types of faults can be injected to the original programs. The mutation operators are adopted from the work of Ji et al. [15]. The mutant programs are then analyzed with some test suites. We consider test suites generated according to the control flow based structural analysis approach proposed in [4, 9] for a given Java program. We also analyzed the program in JUnit 4.6 [16] and Jumble 1.0 [17] testing environment and their mutant programs. Our experimental results substantiate the efficiency of the proposed mutation analysis approach.The rest of the paper is organized as follows. We discuss basic concepts, related terminologies, and tools in Section2. In Section 3, we discuss the mutation operators followed in this work. Mutation analysis is discussed in Section 4. Our experimental results are presented in Section 5. Related work and comparison of our work with others are discussed in Section 6. Finally, Section 7 concludes the paper.


## 2. Basic Concepts

In this section, we discuss some basic concepts and terminologies which are referred to in our subsequent discussions. We also briefly introduce the tools used in our work.


### 2.1. Exception Handling Constructs in Java

Java [18] programming language is considered as a model language in this work. In this subsection, we provide an overview of exception handling constructs in Java. Details of exception handling constructs in Java can be found in [19, 20].In Java, exceptions are first-class objects. Each exception is an instance of a class that is derived from the classjava.lang.Throwable, which is defined in the standard Java API [19]. An exception can be raised at any point in the program through athrow statement. Athrow statement is associated with an expression denoting an exception object, which may be a variable (e.g.,throw e), a method call (e.g.,throw m()) or a new instantiation (e.g.,throw new E()), and so forth. Athrow statement can appear anywhere in the program. Java exception handling construct includes three blocks:try,catch andfinally (see Figure 1(a)). Code can be guarded for exceptions within atry block. Exceptions raised through execution of code (implicitly) or athrow statement (explicitly) within atry block may be caught in one or morecatch clause(s) declared immediately following thetry block. Thefinally block is executed independently of what happens in thetry block. It may be noted that atry statement consists of atry block and optionally one or morecatch blocks andfinally block (see Figure 1(a)). The valid instances of atry statement aretry{…}catch (…){…}[catch (…){…}][…] [finally{…}], where the blocks in square brackets indicate optional. Here, atry block contains statement whose execution is monitored for exception occurrences. Acatch block specifies the type of exception it handles andexception handler; that is, it contains a block of code that is executed when an exception of that type is raised in the associatedtry block. If afinally block is associated with atry block, then thefinally block is always executed, irrespective of how control transfers out of thetry block.Exception handling constructs and exceptions in Java [19].
(a)
Exception handling constructs in Java
(b)
Exception class hierarchy in Java


### 2.2. Java Exception Hierarchy

Similar to other Java objects, exceptions are typed, and types are organized into a hierarchy. All exceptions inherit from the class typeThrowable defined injava.lang API. The exception type hierarchy defines three different groups of exceptions:errors,runtime exceptions, andchecked exception (defined inJava.lang.Exception) (also see Figure 1(b)). Note that errors and runtime exceptions are unchecked exceptions. Unchecked exceptions can be thrown at any point in a Java program and, if uncaught, may propagate back to the program entry point, causing the Java Virtual Machine [20] to terminate. The classError is responsible for giving errors in some catastrophic failures. The exceptions of typeRuntimeException are typically created automatically during runtime in response to some execution errors. The exceptions of the typeException are checked exceptions and are used for exceptional conditions that user programs can catch. A list of exceptions that a Java programmer can catch in programs is available in [19]. There are 11 runtime exceptions, 17 errors, and 7 checked exceptions defined inJava.lang API [19] as per the release of Java 6. Programmers can define and extend existing exception type in Java API to customize their needs. Some new exception types can also be defined [20].


### 2.3. Call Exception Hierarchy

In this subsection, we discuss the flow of exceptions within the context of the structure of a given program. This flow of exceptions, we specially term it ascall exception hierarchy (CEH), is useful to understand and evaluate how exceptions are handled within a method, which exceptions might arise during the execution of a method, which exceptions are handled and where, and which exceptions are passed on. Alternatively, a call exception hierarchy specifies which exceptions may propagate to its caller. In essence, the call exception hierarchy is precisely specifying the behavior of the program as far as the exceptions and handling exceptions are concerned [21].We identify three basic control flow constructs to obtain the call exception hierarchy in any program. These are method calls, nesting, and multicatch. Any complex and arbitrary call exception hierarchy can be resolved with the help of the above three basic constructs. We discuss the three basic constructs as follows.


#### 2.3.1. Method Calls

A method sayMain can call a method mi, which in turn may call another method say mj and so on. Any method can throw and catch exceptions. The call exception hierarchy, in this case, implies the parent-child relationship among exceptions. An exception E2 is a child of another exception, say E1, if E2 occurs in a method that is called by the method where E1 may occur. The call exception hierarchy with method calls is illustrated with an example in Figure 2(a). In Figure 2(a), the methodMain calls the method M1() and is supposed to catch an exception E. Now, the method M1() calls method M2() and may throw two exceptions, namely, E11 and E12. The method M2() in turns may throw exception E21 and E22. In this case, the exception E should be at the root of the hierarchy because E is supposed to represent all instances of exceptions when the invocation of the method M1 occurs. The exceptions E11 and E12 are E’s children because they are instances, which may propagate to E. Similarly, E1 is a child of E and E21, E22 are the children of E1. Note that here the exceptions E21 and E22 are supposed to be caught through the exception E1, if it fails then through E. The call exception hierarchy for a situation of method calls in Figures 2(a)(i)–2(a)(iii) is shown in Figure 2(a)(iv).Basic constructs of call exception hierarchy.
(a)
Call exception hierarchy with method calls
(b)
Call exception hierarchy with nested try blocks
(c)
Call exception hierarchy with multiple catch blocks


#### 2.3.2. Nestedtry Blocks

In Java, atry block can be nested within anothertry block and so on. In this case, the call exception hierarchy can be obtained as discussed below. All exceptions at outertry block are at the root. All exceptions at the next innertry blocks are the children of the root. Any exceptions in atry block at the next level are the children of exceptions at its uppertry level and so on. We illustrate such a situation with an example. We consider a two-level nestedtry structure as shown in Figure 2(b). In Figure 2(b)(i) the methodMain is supposed to catch the exception E, which is at the outermosttry block. Now,Main calls method M1() at the innermosttry block. The method M1() may raise two exceptions, namely, E11 and E12. E11 and E12 would be the successor of E1, which is supposed to catch these two exceptions. In turn E1 would be the successor of E. The call exception hierarchy of the program structure of Figures 2(b)(i) and 2(b)(ii) is shown in Figure 2(b)(iii).


#### 2.3.3. Multiplecatch Blocks

There may be multiplecatch blocks corresponding to atry block [19, 20]. In this case, all exceptions, which are supposed to be caught, would be placed as a single node and all exceptions, which are raised by acalled method inside thetry block, would be the children of this node. As an example, let us consider a simpletry with multiplecatch as shown in Figure 2(c). In Figure 2(c)(i), the methodMain contains atry block and thistry block is associated with three parallel catch blocks. These threecatch blocks are corresponding to handle three different exceptions, namely, E1, E2, and E3. Further, the methodMain() calls the method M1(), which is supposed to throw the exceptions E11 and E12 (see Figure 2(c)(ii)). In this case, the exceptions E1, E2, and E3 of thecaller methodMain is at the root and the exceptions E11 and E12 of thecalled method M1() are the children of the exceptions E1, E2, and E3. The call exception hierarchy for this case (Figures 2(c)(i) and 2(c)(ii)) is shown in Figure 2(c)(iii).Related to the call exception hierarchy, we define the following terminologies.Caller Exception. An exception E in aCEH for a program P is called the caller exception if it is a parent of any other exception Ei∈CEH. In other words, all exceptions except at the leaf of call exception hierarchy are thecaller exceptions.Callee Exception. An exception E in aCEH for a program P is called thecallee exception if it is a child of any other exception Ei∈CEH. In other words, all exceptions except the exception(s) at the root of a call exception hierarchy arecallee exceptions.Direct Caller Exception. InCEH, if an exception Ei is the immediate predecessor of another exception Ej, then the exception Ei is called the direct caller exception.Direct Callee Exception. InCEH, if an exception Ei is the immediate successor of another exception Ej, then the exception Ei is called the direct callee exception.


### 2.4. Mutation Testing

Mutation testing [22] is a widely accepted and practical method to evaluate the quality of a test suite to test programs. Mutation testing is a fault-based testing adequacy criterion proposed by DeMillo et al. [23] initially with the namemutation analysis. Given a program P, a set of alternative programs, say ϕ(P), is considered in order to measure the test adequacy of a test suite T. A test set T is adequate to P in relation to ϕ(P) if for each program Q∈ϕ(P), either Q is equivalent to P or Q differs from P on at least one test case t∈T [24]. A mutation testing framework is shown in Figure 3. In the context of mutation analysis some commonly known terminologies are mentioned below.Figure 3
Mutation testing framework.


#### 2.4.1. Mutant

A set of alternative programsϕ(P) is called mutants of P. A mutant program Q∈ϕ(P) is different from P only on some simple syntactic changes. This is also alternatively called fault injection to P.


#### 2.4.2. Mutation Operator

Mutation operator is a set of rules to decide syntactic changes in the programP to obtain mutants ϕ(P).


#### 2.4.3. Mutant Dead

To assess the adequacy of a test setT, each mutant Q∈ϕ(P), as well as the program P, has to be executed against the test cases in T. If the observed output of the mutant Q is the same as that of P for all test cases in T, then Q is considered alive; otherwise it is considered dead or distinguished.


#### 2.4.4. Equivalent Mutant

LetD be the input domain of the test suite T. A mutant Q is called an equivalent mutant if ∀x∈D, P, and Q produce the same results. Note that an equivalent mutant cannot be distinguished and should be discarded from the mutant set as it does not contribute to the adequacy of T.


#### 2.4.5. Mutation Score

Test adequacy is measured by the mutation scoreμ(P,T) as defined in
(1)μP,T=DMTM-EM,
where DM denotes the number of dead mutants of P for a test suite T and TM and EM are the total number of mutants in ϕ(P) and equivalent mutants, respectively. In summary, a low value of the mutation score implies either test suite unable to cover sufficient faults or there is less faults and vice versa.


### 2.5. Java Program Testing Tools

Of late, Java programming has been extensively practiced in industries. A number of Java tools are available to support the Java practitioners. Two commonly used tools are JUnit [16, 25] and Jumble [17, 26]. These two tools are used in this work. We introduce these tools briefly in the following.


#### 2.5.1. JUnit

JUnit [16] is an API framework and is used to automate unit and regression testing. Using JUnit, an entire class, part of a class, that is, a method or some interacting methods and interactions among several objects, can be tested. To test using JUnit, a tester class is required. The tester class contains more than one test case where each test case is written into one test method [25]. The tester class also includes a test executor to run the test cases and methods to set up the state(s) of object(s) before and update the state(s) after the execution of each test case.The advantages with JUnit are that it allows assertions for testing expected results, it has test features for sharing common data, and it helps to isolate and localize errors. The limitation with JUnit is that developers have to write test methods themselves, which requires a substantial amount of coding effort. Further, JUnit is designed to call methods and compare the results they return against expected results. This works for methods that return some results, but many methods instead of returning values have side effects, such as displaying output, modifying states of constituent objects, and modifying attributes’ values. Also, JUnit cannot directly test private methods and not suitable for system or integration testing.


#### 2.5.2. Jumble

Jumble [17] is a mutation testing tool for Java programs, which interoperates with JUnit. Jumble takes a class to be mutation tested and a set of JUnit test classes as parameters. The output is the overall score of how many mutations are successfully caught as well as details about those mutations which are not. Jumble performs its mutations by directly modifying Java byte code using BCEL package [26]. This allows the mutation testing to be done with no compilations.The advantages with Jumble are that it mutates a program at the byte code level; hence, Jumble is faster and it can be used to test the programs even when the source code is not available. Moreover, Jumble uses a heuristic to improve its performance. Using the heuristic, it determines the order in which to run the test cases so that a test fails at a faster rate. It ensures that the longer test cases run only when there is no shorter test case to cover mutations.The major limitation with Jumble is that it only provides mutations operators for conditional, arithmetic, initialization and assignment, expressions, return values, and switch statements [26]. Jumble cannot discriminateboolean,short, andchar data types because all these data types in byte code is represented as 32 bitsinteger. Further, Jumble uses strong mutation [26]. Weak mutation is not possible with Jumble because if JUnit test case uses a value that causes a mutated expression to return a different value, there is no guarantee that the JUnit test case will detect that different value.


## 2.1. Exception Handling Constructs in Java

Java [18] programming language is considered as a model language in this work. In this subsection, we provide an overview of exception handling constructs in Java. Details of exception handling constructs in Java can be found in [19, 20].In Java, exceptions are first-class objects. Each exception is an instance of a class that is derived from the classjava.lang.Throwable, which is defined in the standard Java API [19]. An exception can be raised at any point in the program through athrow statement. Athrow statement is associated with an expression denoting an exception object, which may be a variable (e.g.,throw e), a method call (e.g.,throw m()) or a new instantiation (e.g.,throw new E()), and so forth. Athrow statement can appear anywhere in the program. Java exception handling construct includes three blocks:try,catch andfinally (see Figure 1(a)). Code can be guarded for exceptions within atry block. Exceptions raised through execution of code (implicitly) or athrow statement (explicitly) within atry block may be caught in one or morecatch clause(s) declared immediately following thetry block. Thefinally block is executed independently of what happens in thetry block. It may be noted that atry statement consists of atry block and optionally one or morecatch blocks andfinally block (see Figure 1(a)). The valid instances of atry statement aretry{…}catch (…){…}[catch (…){…}][…] [finally{…}], where the blocks in square brackets indicate optional. Here, atry block contains statement whose execution is monitored for exception occurrences. Acatch block specifies the type of exception it handles andexception handler; that is, it contains a block of code that is executed when an exception of that type is raised in the associatedtry block. If afinally block is associated with atry block, then thefinally block is always executed, irrespective of how control transfers out of thetry block.Exception handling constructs and exceptions in Java [19].
(a)
Exception handling constructs in Java
(b)
Exception class hierarchy in Java


## 2.2. Java Exception Hierarchy

Similar to other Java objects, exceptions are typed, and types are organized into a hierarchy. All exceptions inherit from the class typeThrowable defined injava.lang API. The exception type hierarchy defines three different groups of exceptions:errors,runtime exceptions, andchecked exception (defined inJava.lang.Exception) (also see Figure 1(b)). Note that errors and runtime exceptions are unchecked exceptions. Unchecked exceptions can be thrown at any point in a Java program and, if uncaught, may propagate back to the program entry point, causing the Java Virtual Machine [20] to terminate. The classError is responsible for giving errors in some catastrophic failures. The exceptions of typeRuntimeException are typically created automatically during runtime in response to some execution errors. The exceptions of the typeException are checked exceptions and are used for exceptional conditions that user programs can catch. A list of exceptions that a Java programmer can catch in programs is available in [19]. There are 11 runtime exceptions, 17 errors, and 7 checked exceptions defined inJava.lang API [19] as per the release of Java 6. Programmers can define and extend existing exception type in Java API to customize their needs. Some new exception types can also be defined [20].


## 2.3. Call Exception Hierarchy

In this subsection, we discuss the flow of exceptions within the context of the structure of a given program. This flow of exceptions, we specially term it ascall exception hierarchy (CEH), is useful to understand and evaluate how exceptions are handled within a method, which exceptions might arise during the execution of a method, which exceptions are handled and where, and which exceptions are passed on. Alternatively, a call exception hierarchy specifies which exceptions may propagate to its caller. In essence, the call exception hierarchy is precisely specifying the behavior of the program as far as the exceptions and handling exceptions are concerned [21].We identify three basic control flow constructs to obtain the call exception hierarchy in any program. These are method calls, nesting, and multicatch. Any complex and arbitrary call exception hierarchy can be resolved with the help of the above three basic constructs. We discuss the three basic constructs as follows.


### 2.3.1. Method Calls

A method sayMain can call a method mi, which in turn may call another method say mj and so on. Any method can throw and catch exceptions. The call exception hierarchy, in this case, implies the parent-child relationship among exceptions. An exception E2 is a child of another exception, say E1, if E2 occurs in a method that is called by the method where E1 may occur. The call exception hierarchy with method calls is illustrated with an example in Figure 2(a). In Figure 2(a), the methodMain calls the method M1() and is supposed to catch an exception E. Now, the method M1() calls method M2() and may throw two exceptions, namely, E11 and E12. The method M2() in turns may throw exception E21 and E22. In this case, the exception E should be at the root of the hierarchy because E is supposed to represent all instances of exceptions when the invocation of the method M1 occurs. The exceptions E11 and E12 are E’s children because they are instances, which may propagate to E. Similarly, E1 is a child of E and E21, E22 are the children of E1. Note that here the exceptions E21 and E22 are supposed to be caught through the exception E1, if it fails then through E. The call exception hierarchy for a situation of method calls in Figures 2(a)(i)–2(a)(iii) is shown in Figure 2(a)(iv).Basic constructs of call exception hierarchy.
(a)
Call exception hierarchy with method calls
(b)
Call exception hierarchy with nested try blocks
(c)
Call exception hierarchy with multiple catch blocks


### 2.3.2. Nestedtry Blocks

In Java, atry block can be nested within anothertry block and so on. In this case, the call exception hierarchy can be obtained as discussed below. All exceptions at outertry block are at the root. All exceptions at the next innertry blocks are the children of the root. Any exceptions in atry block at the next level are the children of exceptions at its uppertry level and so on. We illustrate such a situation with an example. We consider a two-level nestedtry structure as shown in Figure 2(b). In Figure 2(b)(i) the methodMain is supposed to catch the exception E, which is at the outermosttry block. Now,Main calls method M1() at the innermosttry block. The method M1() may raise two exceptions, namely, E11 and E12. E11 and E12 would be the successor of E1, which is supposed to catch these two exceptions. In turn E1 would be the successor of E. The call exception hierarchy of the program structure of Figures 2(b)(i) and 2(b)(ii) is shown in Figure 2(b)(iii).


### 2.3.3. Multiplecatch Blocks

There may be multiplecatch blocks corresponding to atry block [19, 20]. In this case, all exceptions, which are supposed to be caught, would be placed as a single node and all exceptions, which are raised by acalled method inside thetry block, would be the children of this node. As an example, let us consider a simpletry with multiplecatch as shown in Figure 2(c). In Figure 2(c)(i), the methodMain contains atry block and thistry block is associated with three parallel catch blocks. These threecatch blocks are corresponding to handle three different exceptions, namely, E1, E2, and E3. Further, the methodMain() calls the method M1(), which is supposed to throw the exceptions E11 and E12 (see Figure 2(c)(ii)). In this case, the exceptions E1, E2, and E3 of thecaller methodMain is at the root and the exceptions E11 and E12 of thecalled method M1() are the children of the exceptions E1, E2, and E3. The call exception hierarchy for this case (Figures 2(c)(i) and 2(c)(ii)) is shown in Figure 2(c)(iii).Related to the call exception hierarchy, we define the following terminologies.Caller Exception. An exception E in aCEH for a program P is called the caller exception if it is a parent of any other exception Ei∈CEH. In other words, all exceptions except at the leaf of call exception hierarchy are thecaller exceptions.Callee Exception. An exception E in aCEH for a program P is called thecallee exception if it is a child of any other exception Ei∈CEH. In other words, all exceptions except the exception(s) at the root of a call exception hierarchy arecallee exceptions.Direct Caller Exception. InCEH, if an exception Ei is the immediate predecessor of another exception Ej, then the exception Ei is called the direct caller exception.Direct Callee Exception. InCEH, if an exception Ei is the immediate successor of another exception Ej, then the exception Ei is called the direct callee exception.


## 2.3.1. Method Calls

A method sayMain can call a method mi, which in turn may call another method say mj and so on. Any method can throw and catch exceptions. The call exception hierarchy, in this case, implies the parent-child relationship among exceptions. An exception E2 is a child of another exception, say E1, if E2 occurs in a method that is called by the method where E1 may occur. The call exception hierarchy with method calls is illustrated with an example in Figure 2(a). In Figure 2(a), the methodMain calls the method M1() and is supposed to catch an exception E. Now, the method M1() calls method M2() and may throw two exceptions, namely, E11 and E12. The method M2() in turns may throw exception E21 and E22. In this case, the exception E should be at the root of the hierarchy because E is supposed to represent all instances of exceptions when the invocation of the method M1 occurs. The exceptions E11 and E12 are E’s children because they are instances, which may propagate to E. Similarly, E1 is a child of E and E21, E22 are the children of E1. Note that here the exceptions E21 and E22 are supposed to be caught through the exception E1, if it fails then through E. The call exception hierarchy for a situation of method calls in Figures 2(a)(i)–2(a)(iii) is shown in Figure 2(a)(iv).Basic constructs of call exception hierarchy.
(a)
Call exception hierarchy with method calls
(b)
Call exception hierarchy with nested try blocks
(c)
Call exception hierarchy with multiple catch blocks


## 2.3.2. Nestedtry Blocks

In Java, atry block can be nested within anothertry block and so on. In this case, the call exception hierarchy can be obtained as discussed below. All exceptions at outertry block are at the root. All exceptions at the next innertry blocks are the children of the root. Any exceptions in atry block at the next level are the children of exceptions at its uppertry level and so on. We illustrate such a situation with an example. We consider a two-level nestedtry structure as shown in Figure 2(b). In Figure 2(b)(i) the methodMain is supposed to catch the exception E, which is at the outermosttry block. Now,Main calls method M1() at the innermosttry block. The method M1() may raise two exceptions, namely, E11 and E12. E11 and E12 would be the successor of E1, which is supposed to catch these two exceptions. In turn E1 would be the successor of E. The call exception hierarchy of the program structure of Figures 2(b)(i) and 2(b)(ii) is shown in Figure 2(b)(iii).


## 2.3.3. Multiplecatch Blocks

There may be multiplecatch blocks corresponding to atry block [19, 20]. In this case, all exceptions, which are supposed to be caught, would be placed as a single node and all exceptions, which are raised by acalled method inside thetry block, would be the children of this node. As an example, let us consider a simpletry with multiplecatch as shown in Figure 2(c). In Figure 2(c)(i), the methodMain contains atry block and thistry block is associated with three parallel catch blocks. These threecatch blocks are corresponding to handle three different exceptions, namely, E1, E2, and E3. Further, the methodMain() calls the method M1(), which is supposed to throw the exceptions E11 and E12 (see Figure 2(c)(ii)). In this case, the exceptions E1, E2, and E3 of thecaller methodMain is at the root and the exceptions E11 and E12 of thecalled method M1() are the children of the exceptions E1, E2, and E3. The call exception hierarchy for this case (Figures 2(c)(i) and 2(c)(ii)) is shown in Figure 2(c)(iii).Related to the call exception hierarchy, we define the following terminologies.Caller Exception. An exception E in aCEH for a program P is called the caller exception if it is a parent of any other exception Ei∈CEH. In other words, all exceptions except at the leaf of call exception hierarchy are thecaller exceptions.Callee Exception. An exception E in aCEH for a program P is called thecallee exception if it is a child of any other exception Ei∈CEH. In other words, all exceptions except the exception(s) at the root of a call exception hierarchy arecallee exceptions.Direct Caller Exception. InCEH, if an exception Ei is the immediate predecessor of another exception Ej, then the exception Ei is called the direct caller exception.Direct Callee Exception. InCEH, if an exception Ei is the immediate successor of another exception Ej, then the exception Ei is called the direct callee exception.


## 2.4. Mutation Testing

Mutation testing [22] is a widely accepted and practical method to evaluate the quality of a test suite to test programs. Mutation testing is a fault-based testing adequacy criterion proposed by DeMillo et al. [23] initially with the namemutation analysis. Given a program P, a set of alternative programs, say ϕ(P), is considered in order to measure the test adequacy of a test suite T. A test set T is adequate to P in relation to ϕ(P) if for each program Q∈ϕ(P), either Q is equivalent to P or Q differs from P on at least one test case t∈T [24]. A mutation testing framework is shown in Figure 3. In the context of mutation analysis some commonly known terminologies are mentioned below.Figure 3
Mutation testing framework.


### 2.4.1. Mutant

A set of alternative programsϕ(P) is called mutants of P. A mutant program Q∈ϕ(P) is different from P only on some simple syntactic changes. This is also alternatively called fault injection to P.


### 2.4.2. Mutation Operator

Mutation operator is a set of rules to decide syntactic changes in the programP to obtain mutants ϕ(P).


### 2.4.3. Mutant Dead

To assess the adequacy of a test setT, each mutant Q∈ϕ(P), as well as the program P, has to be executed against the test cases in T. If the observed output of the mutant Q is the same as that of P for all test cases in T, then Q is considered alive; otherwise it is considered dead or distinguished.


### 2.4.4. Equivalent Mutant

LetD be the input domain of the test suite T. A mutant Q is called an equivalent mutant if ∀x∈D, P, and Q produce the same results. Note that an equivalent mutant cannot be distinguished and should be discarded from the mutant set as it does not contribute to the adequacy of T.


### 2.4.5. Mutation Score

Test adequacy is measured by the mutation scoreμ(P,T) as defined in
(1)μP,T=DMTM-EM,
where DM denotes the number of dead mutants of P for a test suite T and TM and EM are the total number of mutants in ϕ(P) and equivalent mutants, respectively. In summary, a low value of the mutation score implies either test suite unable to cover sufficient faults or there is less faults and vice versa.


## 2.4.1. Mutant

A set of alternative programsϕ(P) is called mutants of P. A mutant program Q∈ϕ(P) is different from P only on some simple syntactic changes. This is also alternatively called fault injection to P.


## 2.4.2. Mutation Operator

Mutation operator is a set of rules to decide syntactic changes in the programP to obtain mutants ϕ(P).


## 2.4.3. Mutant Dead

To assess the adequacy of a test setT, each mutant Q∈ϕ(P), as well as the program P, has to be executed against the test cases in T. If the observed output of the mutant Q is the same as that of P for all test cases in T, then Q is considered alive; otherwise it is considered dead or distinguished.


## 2.4.4. Equivalent Mutant

LetD be the input domain of the test suite T. A mutant Q is called an equivalent mutant if ∀x∈D, P, and Q produce the same results. Note that an equivalent mutant cannot be distinguished and should be discarded from the mutant set as it does not contribute to the adequacy of T.


## 2.4.5. Mutation Score

Test adequacy is measured by the mutation scoreμ(P,T) as defined in
(1)μP,T=DMTM-EM,
where DM denotes the number of dead mutants of P for a test suite T and TM and EM are the total number of mutants in ϕ(P) and equivalent mutants, respectively. In summary, a low value of the mutation score implies either test suite unable to cover sufficient faults or there is less faults and vice versa.


## 2.5. Java Program Testing Tools

Of late, Java programming has been extensively practiced in industries. A number of Java tools are available to support the Java practitioners. Two commonly used tools are JUnit [16, 25] and Jumble [17, 26]. These two tools are used in this work. We introduce these tools briefly in the following.


### 2.5.1. JUnit

JUnit [16] is an API framework and is used to automate unit and regression testing. Using JUnit, an entire class, part of a class, that is, a method or some interacting methods and interactions among several objects, can be tested. To test using JUnit, a tester class is required. The tester class contains more than one test case where each test case is written into one test method [25]. The tester class also includes a test executor to run the test cases and methods to set up the state(s) of object(s) before and update the state(s) after the execution of each test case.The advantages with JUnit are that it allows assertions for testing expected results, it has test features for sharing common data, and it helps to isolate and localize errors. The limitation with JUnit is that developers have to write test methods themselves, which requires a substantial amount of coding effort. Further, JUnit is designed to call methods and compare the results they return against expected results. This works for methods that return some results, but many methods instead of returning values have side effects, such as displaying output, modifying states of constituent objects, and modifying attributes’ values. Also, JUnit cannot directly test private methods and not suitable for system or integration testing.


### 2.5.2. Jumble

Jumble [17] is a mutation testing tool for Java programs, which interoperates with JUnit. Jumble takes a class to be mutation tested and a set of JUnit test classes as parameters. The output is the overall score of how many mutations are successfully caught as well as details about those mutations which are not. Jumble performs its mutations by directly modifying Java byte code using BCEL package [26]. This allows the mutation testing to be done with no compilations.The advantages with Jumble are that it mutates a program at the byte code level; hence, Jumble is faster and it can be used to test the programs even when the source code is not available. Moreover, Jumble uses a heuristic to improve its performance. Using the heuristic, it determines the order in which to run the test cases so that a test fails at a faster rate. It ensures that the longer test cases run only when there is no shorter test case to cover mutations.The major limitation with Jumble is that it only provides mutations operators for conditional, arithmetic, initialization and assignment, expressions, return values, and switch statements [26]. Jumble cannot discriminateboolean,short, andchar data types because all these data types in byte code is represented as 32 bitsinteger. Further, Jumble uses strong mutation [26]. Weak mutation is not possible with Jumble because if JUnit test case uses a value that causes a mutated expression to return a different value, there is no guarantee that the JUnit test case will detect that different value.


## 2.5.1. JUnit

JUnit [16] is an API framework and is used to automate unit and regression testing. Using JUnit, an entire class, part of a class, that is, a method or some interacting methods and interactions among several objects, can be tested. To test using JUnit, a tester class is required. The tester class contains more than one test case where each test case is written into one test method [25]. The tester class also includes a test executor to run the test cases and methods to set up the state(s) of object(s) before and update the state(s) after the execution of each test case.The advantages with JUnit are that it allows assertions for testing expected results, it has test features for sharing common data, and it helps to isolate and localize errors. The limitation with JUnit is that developers have to write test methods themselves, which requires a substantial amount of coding effort. Further, JUnit is designed to call methods and compare the results they return against expected results. This works for methods that return some results, but many methods instead of returning values have side effects, such as displaying output, modifying states of constituent objects, and modifying attributes’ values. Also, JUnit cannot directly test private methods and not suitable for system or integration testing.


## 2.5.2. Jumble

Jumble [17] is a mutation testing tool for Java programs, which interoperates with JUnit. Jumble takes a class to be mutation tested and a set of JUnit test classes as parameters. The output is the overall score of how many mutations are successfully caught as well as details about those mutations which are not. Jumble performs its mutations by directly modifying Java byte code using BCEL package [26]. This allows the mutation testing to be done with no compilations.The advantages with Jumble are that it mutates a program at the byte code level; hence, Jumble is faster and it can be used to test the programs even when the source code is not available. Moreover, Jumble uses a heuristic to improve its performance. Using the heuristic, it determines the order in which to run the test cases so that a test fails at a faster rate. It ensures that the longer test cases run only when there is no shorter test case to cover mutations.The major limitation with Jumble is that it only provides mutations operators for conditional, arithmetic, initialization and assignment, expressions, return values, and switch statements [26]. Jumble cannot discriminateboolean,short, andchar data types because all these data types in byte code is represented as 32 bitsinteger. Further, Jumble uses strong mutation [26]. Weak mutation is not possible with Jumble because if JUnit test case uses a value that causes a mutated expression to return a different value, there is no guarantee that the JUnit test case will detect that different value.


## 3. Mutation Operators

We consider eight mutation operators to create mutant programs [15]. These mutation operators are listed in Table 1. There are two types of mutation operators: related tocatch block andtry block. These mutation operators are discussed in the following subsections. First, we discuss the proposed mutation operators to change incatch blocks. We then discuss the mutation operators related to alteration intry blocks. We refer toCEH as the call exception hierarchy of a Java program P in our discussions.Table 1
Exception handling related mutation operators.
Operator
Description
B
D
Catch block deletion
B
R
Catch block replacement
B
I
Catch block insertion
R
E
Catch and rethrow exception
T
D
Deletion in try block
T
R
Replacement in try block
T
I
Insertion in try block
T
A
Alter try block


### 3.1.BD: Deletion ofcatch Block

The mutation operatorBD is proposed to delete acatch block, sayCB, following atry block, sayTB. Deleting acatch block <S2> using the mutation operatorBD can be formally stated asBD⇒TB:try…CB:catch(E){…}→TB:try{…}
.Figure4 explains the operation of the mutation operatorBD. If there are more than onecatch block, then it can delete in an arbitrary way one or morecatch block(s). The objective of this mutation operator is to alter the exception propagation path. When an exception, say E, is thrown inTB, the mutant program withBD should not handle the exception E appropriately because of the absence ofCB. As a consequence, the exception raised atTB should be back propagated to the caller exception, if any in theCEH. The exception E then may be handled at an upper level if the exception type of E matches with some other exception at upper level. It may be noted that the original program and a mutant version may cause same exception meeting some reachability at thetry blockTB. In that case, deleting the wholecatch blockCB would not produce any distinctions between the two programs for a test case. Hence, this operator may produce equivalent mutants.Figure 4
Mutation operatorBD.


### 3.2.BR: Replacement ofcatch Block

The mutation operatorBR is proposed to cover faults such that a program does not handle exception correctly. LetCB be thecatch block to be altered. The mutation operatorBR attempts to deviate from the exception propagation path. The mutation operatorBR creates mutant program(s) depending on the level of the occurrence of thecatch blockCB. Also, see Figure 5, which states the three different levels ofCBs and their possible mutants. Three rules are to be followed.Figure 5
Mutation operatorBR.Rule 1.
If theCB is at theroot node in theCEH, thenBR replaces the exception parameter type ofCB with that ofDCE, whereDCE denotes thedirect callee exception of the exception ofCB.Rule 2.
If theCB is at anyleaf node in theCEH, thenBR replaces the exception parameter type ofCB with that ofDPE, whereDPE denotes thedirect caller exception of the exception ofCB.Rule 3.
If theCB is at an intermediate level, that is, neither at the root nor at leaf of theCEH, thenCBR replicates thecatch blockCB into two blocksCB1 andCB2, such thatBR replaces the exception parameter type ofCB with that ofDCE andDPE asCB1 andCB2, respectively. Here,DPE andDCE denote thedirect caller anddirect callee exceptions of the exception type ofCB inCEH, respectively.The mutation operatorBR can be formally stated as follows.Rule 1.
IfCB is at the root ofCEHBR⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E′){…}
whereE′ is theDCE of E.Rule 2.
IfCB is any leaf node inCEHBR⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E′){…}
whereE′ is theDPE of E.Rule 3.
IfCB is any node other than root and leaf node inCEHBR⇒TB:try…CB:catchE…→[TB:]try{…}[CB1:]catch(E′){…}[CB2:]catch(E′′){…}
whereE′ and E′′ are theDCE andDPE of E, respectively.


### 3.3.BI: Insertion incatch Block

The purpose of the mutation operatorcatch block insertionBI is to catch all possible exceptions that might arise at atry blockTB. In other words, ifCB is thecatch block associated with thetry blockTB, thenBI tries to create mutant programs, which hide all exceptions in thetry blockTB. The mutation operatorBI inserts catch blocks CB1,CB2,…,CBn for all the exception parameter types E1,E2,…,En where E1,E2,…,En are the callee exceptions in theCEH for the exception at thetry blockTB.It can be formally stated asBI⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CBi]catch(Ei){…}whereCBi is a catch block with exception Ei for each Ei∈CEH.The operation ofBI mutation operator is depicted in Figure 6. Since the mutation operatorBI hides some exceptions to propagate upward, it may produce equivalent mutants.Figure 6
Mutation operatorBI.


### 3.4.RE: Catch and Rethrow Exception

A situation that may arise in a program is that exception raised in atry block may not be propagated down the exception handling stack. To cover such a fault, the mutation operator calledcatch and rethrow exception and denoted byRE is proposed. The mutation operatorRE for acatch block, sayCB, associated with atry block, sayTB, catches the exception typeE ofCB and rethrows the exceptionE inside theCB (see Figure 7, the operation ofRE).Figure 7
Mutation operatorRE.Rethrowing an exception using the mutation operatorRE can be formally stated asRE⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E){…;throwEi;…}for allEi∈CEH.LetE be the exception type at thecatch blockCB. Also, let E1,E2,…,En be the callee exceptions of the exceptionE according to call exception hierarchyCEH. The mutation operatorRE creates the mutant programs by inserting a throw statement to throw the exception Ei for each Ei∈E,E1,E2,…,En. Note the difference betweenBI andRE. InBI, acatch block is inserted with an exception, whereas, inRE, it inserts a throw statement in thecatch block to throw exception of callee exception type of the exception atcatch block including itself.


### 3.5.TD: Deletion intry Blocks

The mutation operatorTD consists of two types of operators: method-level and class-level mutation operators. Concepts of these operators are taken from [27]. These operators are listed in Table 2.Table 2
Mutation operatorTD.
Operator
Description
Level
AOD
Arithmetic operator deletion
Method-level
COD
Conditional operator deletion
LOD
Logical operator deletion
IHD
Information hiding deletion
Class-level
IOD
Overriding method deletion
ISD
Super keyword deletion
OMD
Overloading method deletion
JTD
This keyword deletion
JSD
Static modifier deletion
JID
Member variable initialization deletionThe first three mutation operators in Table2 are the method-level mutation operators and they modify Java expressions by deleting primitive operators in Java programming language [19]. The rest of the mutation operators in Table 2 are the class-level mutation operators dealing with object-oriented features such as inheritance, polymorphisms, and dynamic binding. These operators are to modify declarations, modifiers, initializations, and so forth. A detailed description of these mutant operators can be found in [28].The mutant operatorTD can be mathematically stated as follows:TD⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after deleting one or more operator(s) or declaration(s).


### 3.6.TR: Replacement intry Blocks

Like the mutation operatorTD, the mutation operatorTR consists of method-level and class-level mutation operators [27, 28], which are listed in Table 3. As shown in Table 3, the first six mutation operators are to modify expressions by replacing some compatible Java primitive operators. The remaining mutation operators in Table 3 are to replace modifiers, assignments, initialization, and so forth [28].Table 3
Mutation operatorTR.
Operator
Description
Level
AOR
Arithmetic operator replacement
Method-level
ROR
Relation operator replacement
COR
Conditional operator replacement
LOR
Logical operator replacement
ASR
Assignment operator replacement
SOR
Shift operator replacement
IOP
Overridden method calling position change
Class-level
IOR
Overridden method rename
PCC
Cast type change
OMR
Overloading method content change
EOA
Reference and assignment replacement
EOC
Reference and content replacement
EAM
Accessor method change
EMM
Modifier method changeThe mutant operatorTR can be formally stated as follows:TR⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after replacing one or more operator(s) or declaration(s).


### 3.7.TI: Insertion intry Blocks

Different mutation operators belonging to the proposed mutation operatorTI are shown in Table 4. In Table 4, the mutation operators in first three rows are to modify expressions by inserting some compatible Java primitive operators. The rest of the operators in Table 4 are related to insertions of modifiers, assignments, initializations, and so forth [28].Table 4
Mutation operatorTI.
Operator
Description
Level
AOI
Arithmetic operator insertion
Method-level
COI
Conditional operator insertion
LOI
Logical operator insertion
IHI
Hiding variable information
Class-level
ISI
Super keyword insertion
PCI
Type cast operation insertion
JTI
This keyword insertion
JSI
Static modifier insertionThe mutant operatorTD can be mathematically stated as follows:TI⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after inserting one or more operator(s) or declaration(s).


### 3.8.TA: Altertry Block

The mutation operators discussed in the previous subsections are proposed to modify some statements withintry blocks. In the mutation operatorTA, we propose to include some statements, which are outsidetry blocks, or move some statements outsidetry blocks. Figure 8(ii) shows the working of the mutation operatorTA. In Figure 8(i), P1 is a Java program and there is atry block such that a variable, say in, is initialized in thetry block and it is referenced later outside thetry block. Suppose that in is initialized with a null pointer. In that case, it may throw null pointer exception [29]. We mutate the program P1 to P1′ by including the statement of reference within thetry block with the objective that whether thecatch block corresponding totry block is able to catch the exception type which might raise due to the reference of in. Here, the variable in is initialized and referenced within thetry block and exception raised say due to null pointer assignment in statement marked as (A) may be caught in thecatch block. Alternatively, let us consider the case of Java program P, where a variable x is initialized and referenced in atry block. We apply the mutation operatorTA to P to have a mutant program P′ such that the reference statement is moved from thetry block and place it outside thetry-catch block. The objective of this type of mutation is to check whether the exception raised may be propagated and catch in any caller exception or not. In the mutant program, it is now placed outside thetry block and which is now beyond the scope of thecatch block of thetry block.Figure 8
Mutation operatorTA.The mutant operatorTA can be mathematically stated as follows:TI⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S obtained after either keeping both declaration and reference of an object within the sametry block if they were not together or declaration in thetry block but its reference outside thetry block, when they were originally together within atry block.


## 3.1.BD: Deletion ofcatch Block

The mutation operatorBD is proposed to delete acatch block, sayCB, following atry block, sayTB. Deleting acatch block <S2> using the mutation operatorBD can be formally stated asBD⇒TB:try…CB:catch(E){…}→TB:try{…}
.Figure4 explains the operation of the mutation operatorBD. If there are more than onecatch block, then it can delete in an arbitrary way one or morecatch block(s). The objective of this mutation operator is to alter the exception propagation path. When an exception, say E, is thrown inTB, the mutant program withBD should not handle the exception E appropriately because of the absence ofCB. As a consequence, the exception raised atTB should be back propagated to the caller exception, if any in theCEH. The exception E then may be handled at an upper level if the exception type of E matches with some other exception at upper level. It may be noted that the original program and a mutant version may cause same exception meeting some reachability at thetry blockTB. In that case, deleting the wholecatch blockCB would not produce any distinctions between the two programs for a test case. Hence, this operator may produce equivalent mutants.Figure 4
Mutation operatorBD.


## 3.2.BR: Replacement ofcatch Block

The mutation operatorBR is proposed to cover faults such that a program does not handle exception correctly. LetCB be thecatch block to be altered. The mutation operatorBR attempts to deviate from the exception propagation path. The mutation operatorBR creates mutant program(s) depending on the level of the occurrence of thecatch blockCB. Also, see Figure 5, which states the three different levels ofCBs and their possible mutants. Three rules are to be followed.Figure 5
Mutation operatorBR.Rule 1.
If theCB is at theroot node in theCEH, thenBR replaces the exception parameter type ofCB with that ofDCE, whereDCE denotes thedirect callee exception of the exception ofCB.Rule 2.
If theCB is at anyleaf node in theCEH, thenBR replaces the exception parameter type ofCB with that ofDPE, whereDPE denotes thedirect caller exception of the exception ofCB.Rule 3.
If theCB is at an intermediate level, that is, neither at the root nor at leaf of theCEH, thenCBR replicates thecatch blockCB into two blocksCB1 andCB2, such thatBR replaces the exception parameter type ofCB with that ofDCE andDPE asCB1 andCB2, respectively. Here,DPE andDCE denote thedirect caller anddirect callee exceptions of the exception type ofCB inCEH, respectively.The mutation operatorBR can be formally stated as follows.Rule 1.
IfCB is at the root ofCEHBR⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E′){…}
whereE′ is theDCE of E.Rule 2.
IfCB is any leaf node inCEHBR⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E′){…}
whereE′ is theDPE of E.Rule 3.
IfCB is any node other than root and leaf node inCEHBR⇒TB:try…CB:catchE…→[TB:]try{…}[CB1:]catch(E′){…}[CB2:]catch(E′′){…}
whereE′ and E′′ are theDCE andDPE of E, respectively.


## 3.3.BI: Insertion incatch Block

The purpose of the mutation operatorcatch block insertionBI is to catch all possible exceptions that might arise at atry blockTB. In other words, ifCB is thecatch block associated with thetry blockTB, thenBI tries to create mutant programs, which hide all exceptions in thetry blockTB. The mutation operatorBI inserts catch blocks CB1,CB2,…,CBn for all the exception parameter types E1,E2,…,En where E1,E2,…,En are the callee exceptions in theCEH for the exception at thetry blockTB.It can be formally stated asBI⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CBi]catch(Ei){…}whereCBi is a catch block with exception Ei for each Ei∈CEH.The operation ofBI mutation operator is depicted in Figure 6. Since the mutation operatorBI hides some exceptions to propagate upward, it may produce equivalent mutants.Figure 6
Mutation operatorBI.


## 3.4.RE: Catch and Rethrow Exception

A situation that may arise in a program is that exception raised in atry block may not be propagated down the exception handling stack. To cover such a fault, the mutation operator calledcatch and rethrow exception and denoted byRE is proposed. The mutation operatorRE for acatch block, sayCB, associated with atry block, sayTB, catches the exception typeE ofCB and rethrows the exceptionE inside theCB (see Figure 7, the operation ofRE).Figure 7
Mutation operatorRE.Rethrowing an exception using the mutation operatorRE can be formally stated asRE⇒[TB:]try{…}[CB:]catch(E){…}→[TB:]try{…}[CB:]catch(E){…;throwEi;…}for allEi∈CEH.LetE be the exception type at thecatch blockCB. Also, let E1,E2,…,En be the callee exceptions of the exceptionE according to call exception hierarchyCEH. The mutation operatorRE creates the mutant programs by inserting a throw statement to throw the exception Ei for each Ei∈E,E1,E2,…,En. Note the difference betweenBI andRE. InBI, acatch block is inserted with an exception, whereas, inRE, it inserts a throw statement in thecatch block to throw exception of callee exception type of the exception atcatch block including itself.


## 3.5.TD: Deletion intry Blocks

The mutation operatorTD consists of two types of operators: method-level and class-level mutation operators. Concepts of these operators are taken from [27]. These operators are listed in Table 2.Table 2
Mutation operatorTD.
Operator
Description
Level
AOD
Arithmetic operator deletion
Method-level
COD
Conditional operator deletion
LOD
Logical operator deletion
IHD
Information hiding deletion
Class-level
IOD
Overriding method deletion
ISD
Super keyword deletion
OMD
Overloading method deletion
JTD
This keyword deletion
JSD
Static modifier deletion
JID
Member variable initialization deletionThe first three mutation operators in Table2 are the method-level mutation operators and they modify Java expressions by deleting primitive operators in Java programming language [19]. The rest of the mutation operators in Table 2 are the class-level mutation operators dealing with object-oriented features such as inheritance, polymorphisms, and dynamic binding. These operators are to modify declarations, modifiers, initializations, and so forth. A detailed description of these mutant operators can be found in [28].The mutant operatorTD can be mathematically stated as follows:TD⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after deleting one or more operator(s) or declaration(s).


## 3.6.TR: Replacement intry Blocks

Like the mutation operatorTD, the mutation operatorTR consists of method-level and class-level mutation operators [27, 28], which are listed in Table 3. As shown in Table 3, the first six mutation operators are to modify expressions by replacing some compatible Java primitive operators. The remaining mutation operators in Table 3 are to replace modifiers, assignments, initialization, and so forth [28].Table 3
Mutation operatorTR.
Operator
Description
Level
AOR
Arithmetic operator replacement
Method-level
ROR
Relation operator replacement
COR
Conditional operator replacement
LOR
Logical operator replacement
ASR
Assignment operator replacement
SOR
Shift operator replacement
IOP
Overridden method calling position change
Class-level
IOR
Overridden method rename
PCC
Cast type change
OMR
Overloading method content change
EOA
Reference and assignment replacement
EOC
Reference and content replacement
EAM
Accessor method change
EMM
Modifier method changeThe mutant operatorTR can be formally stated as follows:TR⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after replacing one or more operator(s) or declaration(s).


## 3.7.TI: Insertion intry Blocks

Different mutation operators belonging to the proposed mutation operatorTI are shown in Table 4. In Table 4, the mutation operators in first three rows are to modify expressions by inserting some compatible Java primitive operators. The rest of the operators in Table 4 are related to insertions of modifiers, assignments, initializations, and so forth [28].Table 4
Mutation operatorTI.
Operator
Description
Level
AOI
Arithmetic operator insertion
Method-level
COI
Conditional operator insertion
LOI
Logical operator insertion
IHI
Hiding variable information
Class-level
ISI
Super keyword insertion
PCI
Type cast operation insertion
JTI
This keyword insertion
JSI
Static modifier insertionThe mutant operatorTD can be mathematically stated as follows:TI⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S after inserting one or more operator(s) or declaration(s).


## 3.8.TA: Altertry Block

The mutation operators discussed in the previous subsections are proposed to modify some statements withintry blocks. In the mutation operatorTA, we propose to include some statements, which are outsidetry blocks, or move some statements outsidetry blocks. Figure 8(ii) shows the working of the mutation operatorTA. In Figure 8(i), P1 is a Java program and there is atry block such that a variable, say in, is initialized in thetry block and it is referenced later outside thetry block. Suppose that in is initialized with a null pointer. In that case, it may throw null pointer exception [29]. We mutate the program P1 to P1′ by including the statement of reference within thetry block with the objective that whether thecatch block corresponding totry block is able to catch the exception type which might raise due to the reference of in. Here, the variable in is initialized and referenced within thetry block and exception raised say due to null pointer assignment in statement marked as (A) may be caught in thecatch block. Alternatively, let us consider the case of Java program P, where a variable x is initialized and referenced in atry block. We apply the mutation operatorTA to P to have a mutant program P′ such that the reference statement is moved from thetry block and place it outside thetry-catch block. The objective of this type of mutation is to check whether the exception raised may be propagated and catch in any caller exception or not. In the mutant program, it is now placed outside thetry block and which is now beyond the scope of thecatch block of thetry block.Figure 8
Mutation operatorTA.The mutant operatorTA can be mathematically stated as follows:TI⇒[TB:]try{…;S;…}[CB:]catch(E){…}→[TB:]try{…;S′;…}[CB:]catch(E){…}whereS′ is an altered statement of S obtained after either keeping both declaration and reference of an object within the sametry block if they were not together or declaration in thetry block but its reference outside thetry block, when they were originally together within atry block.


## 4. Mutation Analysis

In the last section, we have discussed the proposed mutation operators to test the effectiveness of test cases for testing exception handling constructs in Java. In this section, we discuss our analysis to evaluate the effectiveness of the proposed mutation operators. To do this, we have considered a number of Java programs. We then apply mutation operators to create a number of mutant programs for each Java program. A number of test cases which are used in our analysis is then discussed. All these steps are discussed in the following subsections.


### 4.1. Subject Programs

In order to verify the effectiveness of our approach, we have carried out a number of experiments. In our experiments, we have considered five different types of subject programs from different application domains. The Java programs considered are:(i)
library information systems (LIS): automating the regular routines in library management,(ii)
cell phone system (CPS): designing hand-set services for mobile communications,(iii)
trading house automation system (TAS): automating business processes of trading houses,(iv)
conference management systems (CMS): automating various activities in conference organization,(v)
vending automation system (VAS): automating vending processes.The designs ofLIS andCPS are followed from [30] and [31], respectively. The applications, namely,TAS andCMS, are designed in UML 2.0 by the students of postgraduate course (software engineering) in Computer Science and Engineering Department, Indian Institute of Technology Kharagpur [32, 33]. The UML models of the application VAS are taken from [34]. We follow the designs from the above-mentioned sources and implemented them in Java. The implementation characteristics are summarized in Table 5.Table 5
Characteristics of subject programs.
Subject program
Size (LOC)
Number of classes
Exception handling constructs
Try blocks
Catch blocks
Outside exceptions
L
I
S
3356
16/04
26
33
18
T
A
S
2448
16/06
30
38
29
C
P
S
2257
21/07
24
46
21
C
M
S
2146
25/09
19
32
27
V
A
S
1864
10/04
11
25
31The Java programs we have considered are neither very small nor very large, but of moderate sizes. In column 3 of Table5, the numerator indicates the number of entity and boundary classes and the denominator indicates the number of modal classes also called controller classes (classes which exhibit distinct behavior in different states). The numbers of totaltry blocks,catch blocks are shown in columns 4 and 5 of Table 5, respectively. The number of exceptions thrown in Java methods, which are not within the scope oftry block, is shown in column 6 of Table 5.


### 4.2. Mutant Programs

In our mutation analysis, we have considered fault-based mutation analysis technique. For this, we have generated the mutant programs of the considered programs (in Table5). The mutant programs are generated by seeding faults randomly to the programs based on a fault model. The fault model we have followed in our work is stated below.(1)
Catch block misses exception: the exception thrown in atry block does not match with the exception parameter type of thecatch block associated with thetry block.(2)
Catch block handles exceptions incorrectly: the body of acatch block, that is, the exception handling routine, is not proper to handle the exceptions.(3)
Catch block rethrows exceptions: catch block handles an exception and rethrows the same or another exception(s).(4)
Catch block catches exceptions: exception thrown matches with the exception parameter type in acatch block and does not perform any routine but avoids abnormal exit.(5)
Try block throws exceptions implicitly: due to faults in expression, keyword, declaration, initialization, and assignment, some exceptions may be generated implicitly.(6)
Try block misses or throws explicit exceptions: some exceptions are missed and thrown outsidetry block; some exceptions are thrown deliberately insidetry block.Based on the above fault model, we have applied the mutation operators as discussed in Section3, and the number of mutant programs created for each programs is shown in Table 6.Table 6
Mutant programs.
Mutation operator
Subject programs
L
I
S
T
A
S
C
P
S
C
M
S
V
A
S
B
R
16
14
19
22
14
B
D
17
19
20
26
18
B
I
12
18
21
22
15
R
E
8
11
16
10
13
T
R
34
27
29
26
21
T
D
23
26
21
22
20
T
I
30
26
25
24
22
T
A
9
11
12
10
8
Jumble
98
105
102
118
91We have also generated a number of mutant programs with Jumble [17, 26]. The mutant programs with Jumble for each sample program is shown in the last row of Table 6. According to the proposed mutation operators we create 149 mutant programs forLIS (see column 2, Table 6), 152 mutant programs forTAS (column 3, Table 6), 163 mutant programs forCPS (column 4, Table 6), 162 mutant programs forCMS (column 5, Table 6), and 131 mutant programs forVAS (column 6, Table 6).


### 4.3. Generating Test Cases

To test the subject programs and their mutant versions, we have considered several test cases for each subject program. Generation of test cases is not the focus of this work. We have studied the existing test case generation approaches and followed an appropriate approach as discussed in the following.Structural testing techniques [35] develop test cases to cover various structural elements of a program such as control-flow, data-flow, branch, and path. Control-flow based structural testing criteria [36] use the flow of control in a program to generate test cases. Data-flow-based testing criteria [37–40] use the data-flow relationships to select the test cases. In branch testing [36, 41], test cases are developed by considering inputs that cause certain branches in the program under test to be executed. Similarly, in path testing [42, 43], test cases are generated to execute certain paths in programs. All the existing structural testing techniques, however, not necessarily cover new structural element such as exception handling constructs, all statements that raise exceptions, and those that catch exceptions. Please note that these criteria may be similar to the traditional coverage criteria [36] that require the coverage of statements or branches. Nevertheless, the criteria such as branch testing [41] have weak fault detection capabilities [44]. As a consequence, all-throw, all-catch coverage criteria are not necessarily strong coverage criteria. In fact, this coverage criterion does not cover the various exceptional control flow paths [4]. Sinha and Harrold [9] proposed a family of exception testing criteria to adequately test the behavior of exception handling constructs. Based on the proposed criteria [9], Sinha and Harrold [4] proposed a testing mechanism dealing with exception handling constructs. In [4], a technique has been proposed to construct control-flow graph representation of programs with explicit exception occurrences, that is, exceptions that are raised explicitly throughthrow statements and exception handling constructs. This control-flow representation is useful to trace various exceptional control flow paths and cover different types of exception that can be raised at a statement or the complex control and data interactions, both intra- and intermethods, that may have in the presence of exception handling constructs. In our work, we propose the test case generation scheme following the control flow representation of programs proposed in [4, 9].From the control-flow representation, we can specify the paths to be tested; however, test data to exercise a path is a challenging task, particularly, for complex and large number of paths. The existing approaches of test data generation can be divided into three classes: random, static, and dynamic. Random test data generation is easy to automate but problematic [45]. First, it produces a statistically insignificant sample of the possible paths through the program under test. Second, it may be expensive to generate the expected output data for the large amount of input data produced. Finally, given that exceptions occur only rarely, the input domain which causes an exception is likely to be small. Random test data generations may never hit on this small area of the input domain.Static approaches to test data generation generally use symbolic execution. Many test data generation approaches presented in the literature use symbolic execution to obtain structural test data [46–50]. Symbolic execution works by traversing a control flow graph of the program under test and building up symbolic representations of the internal variables in terms of the input variables for the desired path. Branches within the code introduce constraints on the variables. Solutions to these constraints represent the desired test data. A number of problems exist with this approach. Using symbolic execution it is difficult to analyze recursion, array, and indices which depend on input data and some loop structures. Also, the problem of solving arbitrary constructs is known to be undecidable.Dynamic test-data generation involves execution of the program under test and a directed search for test data that meets the desired criterion [51]. Local search techniques only work effectively for linear continuous functions. Consequently, these techniques are likely to become stuck at a local optimum [52]. DeMillo and Offutt [53] and Offutt et al. [54] proposed approaches for test data generation for mutation testing. The technique proposed in [53] is based on mutation analysis to create test data approximating the test adequacy. The technique is to automatically generate constraints and solve them to create test cases for unit testing. Another technique proposed in [54] is an extension of previous research [53] in constraint-based testing. This later method takes an initial set of values for each input and then refine the values through the control-flow graph of the program and resolve the path constraint dynamically. In this work, to generate the test data for test cases we follow the approach proposed in [54].


## 4.1. Subject Programs

In order to verify the effectiveness of our approach, we have carried out a number of experiments. In our experiments, we have considered five different types of subject programs from different application domains. The Java programs considered are:(i)
library information systems (LIS): automating the regular routines in library management,(ii)
cell phone system (CPS): designing hand-set services for mobile communications,(iii)
trading house automation system (TAS): automating business processes of trading houses,(iv)
conference management systems (CMS): automating various activities in conference organization,(v)
vending automation system (VAS): automating vending processes.The designs ofLIS andCPS are followed from [30] and [31], respectively. The applications, namely,TAS andCMS, are designed in UML 2.0 by the students of postgraduate course (software engineering) in Computer Science and Engineering Department, Indian Institute of Technology Kharagpur [32, 33]. The UML models of the application VAS are taken from [34]. We follow the designs from the above-mentioned sources and implemented them in Java. The implementation characteristics are summarized in Table 5.Table 5
Characteristics of subject programs.
Subject program
Size (LOC)
Number of classes
Exception handling constructs
Try blocks
Catch blocks
Outside exceptions
L
I
S
3356
16/04
26
33
18
T
A
S
2448
16/06
30
38
29
C
P
S
2257
21/07
24
46
21
C
M
S
2146
25/09
19
32
27
V
A
S
1864
10/04
11
25
31The Java programs we have considered are neither very small nor very large, but of moderate sizes. In column 3 of Table5, the numerator indicates the number of entity and boundary classes and the denominator indicates the number of modal classes also called controller classes (classes which exhibit distinct behavior in different states). The numbers of totaltry blocks,catch blocks are shown in columns 4 and 5 of Table 5, respectively. The number of exceptions thrown in Java methods, which are not within the scope oftry block, is shown in column 6 of Table 5.


## 4.2. Mutant Programs

In our mutation analysis, we have considered fault-based mutation analysis technique. For this, we have generated the mutant programs of the considered programs (in Table5). The mutant programs are generated by seeding faults randomly to the programs based on a fault model. The fault model we have followed in our work is stated below.(1)
Catch block misses exception: the exception thrown in atry block does not match with the exception parameter type of thecatch block associated with thetry block.(2)
Catch block handles exceptions incorrectly: the body of acatch block, that is, the exception handling routine, is not proper to handle the exceptions.(3)
Catch block rethrows exceptions: catch block handles an exception and rethrows the same or another exception(s).(4)
Catch block catches exceptions: exception thrown matches with the exception parameter type in acatch block and does not perform any routine but avoids abnormal exit.(5)
Try block throws exceptions implicitly: due to faults in expression, keyword, declaration, initialization, and assignment, some exceptions may be generated implicitly.(6)
Try block misses or throws explicit exceptions: some exceptions are missed and thrown outsidetry block; some exceptions are thrown deliberately insidetry block.Based on the above fault model, we have applied the mutation operators as discussed in Section3, and the number of mutant programs created for each programs is shown in Table 6.Table 6
Mutant programs.
Mutation operator
Subject programs
L
I
S
T
A
S
C
P
S
C
M
S
V
A
S
B
R
16
14
19
22
14
B
D
17
19
20
26
18
B
I
12
18
21
22
15
R
E
8
11
16
10
13
T
R
34
27
29
26
21
T
D
23
26
21
22
20
T
I
30
26
25
24
22
T
A
9
11
12
10
8
Jumble
98
105
102
118
91We have also generated a number of mutant programs with Jumble [17, 26]. The mutant programs with Jumble for each sample program is shown in the last row of Table 6. According to the proposed mutation operators we create 149 mutant programs forLIS (see column 2, Table 6), 152 mutant programs forTAS (column 3, Table 6), 163 mutant programs forCPS (column 4, Table 6), 162 mutant programs forCMS (column 5, Table 6), and 131 mutant programs forVAS (column 6, Table 6).


## 4.3. Generating Test Cases

To test the subject programs and their mutant versions, we have considered several test cases for each subject program. Generation of test cases is not the focus of this work. We have studied the existing test case generation approaches and followed an appropriate approach as discussed in the following.Structural testing techniques [35] develop test cases to cover various structural elements of a program such as control-flow, data-flow, branch, and path. Control-flow based structural testing criteria [36] use the flow of control in a program to generate test cases. Data-flow-based testing criteria [37–40] use the data-flow relationships to select the test cases. In branch testing [36, 41], test cases are developed by considering inputs that cause certain branches in the program under test to be executed. Similarly, in path testing [42, 43], test cases are generated to execute certain paths in programs. All the existing structural testing techniques, however, not necessarily cover new structural element such as exception handling constructs, all statements that raise exceptions, and those that catch exceptions. Please note that these criteria may be similar to the traditional coverage criteria [36] that require the coverage of statements or branches. Nevertheless, the criteria such as branch testing [41] have weak fault detection capabilities [44]. As a consequence, all-throw, all-catch coverage criteria are not necessarily strong coverage criteria. In fact, this coverage criterion does not cover the various exceptional control flow paths [4]. Sinha and Harrold [9] proposed a family of exception testing criteria to adequately test the behavior of exception handling constructs. Based on the proposed criteria [9], Sinha and Harrold [4] proposed a testing mechanism dealing with exception handling constructs. In [4], a technique has been proposed to construct control-flow graph representation of programs with explicit exception occurrences, that is, exceptions that are raised explicitly throughthrow statements and exception handling constructs. This control-flow representation is useful to trace various exceptional control flow paths and cover different types of exception that can be raised at a statement or the complex control and data interactions, both intra- and intermethods, that may have in the presence of exception handling constructs. In our work, we propose the test case generation scheme following the control flow representation of programs proposed in [4, 9].From the control-flow representation, we can specify the paths to be tested; however, test data to exercise a path is a challenging task, particularly, for complex and large number of paths. The existing approaches of test data generation can be divided into three classes: random, static, and dynamic. Random test data generation is easy to automate but problematic [45]. First, it produces a statistically insignificant sample of the possible paths through the program under test. Second, it may be expensive to generate the expected output data for the large amount of input data produced. Finally, given that exceptions occur only rarely, the input domain which causes an exception is likely to be small. Random test data generations may never hit on this small area of the input domain.Static approaches to test data generation generally use symbolic execution. Many test data generation approaches presented in the literature use symbolic execution to obtain structural test data [46–50]. Symbolic execution works by traversing a control flow graph of the program under test and building up symbolic representations of the internal variables in terms of the input variables for the desired path. Branches within the code introduce constraints on the variables. Solutions to these constraints represent the desired test data. A number of problems exist with this approach. Using symbolic execution it is difficult to analyze recursion, array, and indices which depend on input data and some loop structures. Also, the problem of solving arbitrary constructs is known to be undecidable.Dynamic test-data generation involves execution of the program under test and a directed search for test data that meets the desired criterion [51]. Local search techniques only work effectively for linear continuous functions. Consequently, these techniques are likely to become stuck at a local optimum [52]. DeMillo and Offutt [53] and Offutt et al. [54] proposed approaches for test data generation for mutation testing. The technique proposed in [53] is based on mutation analysis to create test data approximating the test adequacy. The technique is to automatically generate constraints and solve them to create test cases for unit testing. Another technique proposed in [54] is an extension of previous research [53] in constraint-based testing. This later method takes an initial set of values for each input and then refine the values through the control-flow graph of the program and resolve the path constraint dynamically. In this work, to generate the test data for test cases we follow the approach proposed in [54].


## 5. Results and Discussion

Test suites for each program considered in our work were generated as discussed in Section4. Mutant programs for each program is also developed using the mutation operators discussed in Section 3. In this section, we discuss our experiments and result is obtained. Let T be the test suite for a program P and ϕ(P) its set of mutant programs. For each test case t∈T, we execute the program P and all mutant programs Q∈ϕ(P). If the observed output of the mutant Q is different than that of P, then we increase the count of the number of mutants killed. This way, we calculate the total numbers of mutants killed by a given test suite T. Further, we also calculate the equivalent mutants, such that two mutant programs Q1∈ϕ(P) and Q2∈ϕ(P) are not distinguishable as far as the output of the two is concerned. We determine whether a mutant Q∈ϕ(P) is an equivalent mutant or not. To do this, we look through the semantic exception hierarchy (as discussed in Section 2.2). If the exception parameter type in any originalcatch block is a super class of all the derived types (according to Java exception hierarchy) in the mutantcatch block, then we judge the mutant Q to be an equivalent mutant. This is because the base type of exception can also accommodate all its subclass types. We also check if some mutants due to the mutation operatorsBD,BI,TD,TR, andTI are equivalent or not.Our experimental results of mutation analysis with five considered subject programs are presented in Table7. The number of mutants killed, equivalent mutants detected, and mutation score for each program and its number of test cases in test suite are shown in Table 7. It is observed that the number of mutants killed by Jumble with the mutant programs created by Jumble is on average 21. On the other hand, the number of mutants killed with mutant programs created by our proposed mutant operators is on average 87. It is also observed that mutant killed by Jumble is mainly related to logical, arithmetic, and conditional expression, whereas mutant killed with our approach is largely related to exception handling constructs. The performance of our mutation analysis is attributed in Figure 9. Figure 9 compares the mutant programs created, mutant killed, mutation points with exception handling constructs, and mutation score according to Jumble and our approach.Table 7
Results of mutation analysis.
Program under test
Number of test cases
Mutant killed
Equivalent mutants
Mutation score
Jumble
Our approach
Jumble
Our approach
Jumble
Our approach
L
I
S
35
18
116
32
16
27
87
T
A
S
48
14
129
28
12
18
92
C
P
S
56
23
112
26
15
30
76
C
M
S
44
29
117
29
23
33
84
V
A
S
32
20
108
23
16
29
94
Average
21
116
28
16
27
87Mutation analysis: Jumble versus our approach.
(a)
Mutant programs
(b)
Mutants killed
(c)
Mutation points
(d)
Mutation scoresIt may be noted that JUnit considers one module at a time. In contrast, we have followed the system testing approach. This is because the unit testing approach is not so effective to test an object-oriented program as multiple objects are in collaboration to solve a problem [55]. Moreover, our hierarchy of call exception is not necessarily limited to a single class but rather distributed across several classes. As a consequence, we need to have access to all the relevant modules so that from the callee class we can determine the actual location from where an exception has been thrown and caught. With modular approach it cannot be determined. Hence, in our approach, we have considered the whole program, that is, a nonmodular analysis technique. We admit that modular approach is faster but the nonmodular analysis at the use case level is quite practical and usually followed in object-oriented software engineering development environment [56].Time Complexity. The running time complexities of both Jumble and our approach are O(n), where n is the number of exceptional handling constructs in a program.


## 6. Related Work and Comparison

Gallagher and Narasimhan [8] proposed an approach for testing exception handling constructs. They proposed structural analysis to select the effective test cases from a given set of test cases to test exception handling constructs in Java programs. In their structural analysis, the whole structure of a program under test is analyzed to retrieve all possible execution paths. They define a fitness function to measure the performances of test cases. All test cases are executed and then evaluated one by one and the high quality test cases are selected based on their fitness measures.Ji et al. [15] proposed four mutation operators for Java exception handling constructs. They aimed to develop effective mutant programs based on those mutation operators. The major difference with the work of Ji et al. [15] and our work is that they proposed the mutation operators to test whether the Java exceptional handling constructs will be able to catch the exceptions or not whereas objective of our work is to test the adequacy of test suites through mutation analysis on Java exceptions. Further, in Ji et al. [15] work, they considered exceptions, which are raised implicitly, that is, calling Java library routines. On the other hand, in addition to implicit raise of exceptions, we consider the generation of exceptions explicitly using arithmetic and logical control statements. That is why we consider many more conventional mutation operators other than the mutation operators as proposed by Ji et al. [15]. Moreover, to ensure that there is a sufficient number of exceptions raised, we have given extra effort to develop test cases accordingly, which is grossly ignored in the work of Ji et al. [15]. More significantly, Ji et al. [15] adopted a trivial experiment with JUnit to test the effectiveness of their proposed mutation operators, that is, whether their proposed mutation operators really capable of causing mutant programs or not. It may be noted that JUnit tests a program at a unit level (i.e., one class at a time), which in fact can catch exceptions raised by the methods in that class only. But, in general, an exception can be raised from method(s) located not necessarily in the same class it is actually caught. Indeed, it is a difficult task to catch the exceptions unless we have a full tracking of all paths of exceptions from the point of cause of exception to the caller routine (and hence main class). This requires entire program to be taken into account rather than only a particular unit. Furthermore, in our approach, emphasize has been given to locate the cause of exception at any point on a path of execution.Sinha and Harrold [9] described a class of test adequacy criteria that can be used to test the behavior of exception handling constructs. They present a subsumption hierarchy of the criteria and point out the relationship of the criteria to those found in traditional subsumption hierarchies. They also proposed techniques for generating the testing requirements for the criteria using some control flow representations.Buhr [21] proposed some exception handling mechanisms in the context of developing rich language features for developing robust programs. They also discussed the exception propagation mechanism and proposed propagation models such as static, dynamic, and propagation.Sinha and Harrold [4] extended their work reported in [9] to analyze and test programs with exception handling constructs. They proposed techniques to construct control-flow representations for Java programs with explicit exceptions occurrences. They used these representation to perform some analysis such as static and dynamic slicing, structural and regression testing, and dynamic execution profiling.Tracey et al. [6] proposed an optimization technique for automatically generating test data to test exceptions. They considered Ada as the model language in their approach. The proposed approach is based on the application of a dynamic global optimization based search for the required test data.Adamopoulos et al. [57] reported on the cause of equivalent mutants. Offutt and Pan [58] proposed a technique to detect equivalent mutants and infeasible paths.Andrews et al. [14] made a thorough investigation of a fundamental assumption that has been underlying much of experimental software testing research that whether faults generated by hand or from mutation operators is representative of real faults. They observed that the use of mutation operators is able to yield trustworthy results; that is, generated mutants are similar to real faults. Mutants appear, however, to be different from hand seeded faults that seem to be harder to detect than real faults.Ma et al. [27] introduced a number of mutation operators to create mutant programs in the context of Java programming language. They propose a tool for entire mutation process and testing [28]. The mutant operators proposed in [27] only modify expressions by replacing, deleting, and inserting primitive features of Java programming language at method-level and object-oriented specific features such as inheritance, polymorphisms, and dynamic binding at class-level. However, Ma et al. did not consider any mutation operators to handle exceptions related faults.Work of Gallagher and Narasimhan [8] and Sinha and Harrold [4] followed static analysis and their approaches not necessarily locate all exception points, particularly those exceptions raised implicitly. For example, an exception can be raised implicitly from calling Java library routine. As a consequence, it is very difficult to draw control flow representations or path identification to cover all problems. Hence, test case selection according to these approach may not be adequate. The proposed method, on the other hand, does not determine exception paths explicitly. We attempt to change exception propagation path according to the characteristic of Java exception handling constructs.Jumble provides mutation operators for arithmetic expression, conditional predicate, initialization, assignment, return values, and control constructs such as if-then, if-then-else, switch, and loop. It is to be noted that the mutation operators according to Jumble are not sufficient enough for exception handling constructs.


## 7. Conclusions

Exception handling is an important mechanism to develop robust and reliable programs. It is a difficult task to test whether all exception points are covered and all exception handler routines are arranged appropriately or not. It is also not possible to identify all exception paths precisely because there are some exceptions which can be raised implicitly. This paper proposes a number of mutation operators to create mutant programs. With these mutant programs, we verify the effectiveness of a given test suite, particularly to verify the exception handling constructs. In other words, mutation operators used in this paper can generate effective mutant programs which can raise and handle different type of exceptions. The test suite which can kill more mutant programs is with a higher mutation score. Effectiveness of the proposed mutation operators is comparable to that of Jumble. Jumble also reports the quantity of mutants killed as the score of a test suite. Jumble provides mutations operators, which are mainly limited to arithmetic expressions, conditions, return values, and control statements such as decision and loop. The main limitation in Jumble is that there is no explicit mutation operators in Jumble that can take care exception handling constructs. Our experiments with five different programs substantiate that the proposed mutation operators are 69% (on average, mutation score in Jumble and our approach is 27 and 87, resp.) more effective to build mutants and then kill the mutant programs than the mutation analysis technique in Jumble. Nevertheless, this result is based on experiments with moderate sized Java programs from different application domains and can be taken as indicative. To have conclusive decision we should go for more experiments with more subjects.Using the proposed mutation operators and mutation analysis method, our work can be further extended to guide automatic synthesis of test cases for testing exception handling constructs. Further, the mutation operators are proposed considering Java as the model language. We can extend the approach for other advanced programming languages.



---

*Source: 197983-2014-12-25.xml*