Automated Testing of Eclipse and NetBeans Refactoring Tools

This position paper presents our experience in automated testing of Eclipse and NetBeans refactoring tools. Test inputs for these tools are Java programs. We have developed ASTGen, a framework for automated generation of abstract syntax trees (ASTs) that represent Java programs. ASTGen allows developers to write imperative generators whose executions produce ASTs. ASTGen offers a library of generic, reusable, and composable generators that make it relatively easy to build more complex generators. We have developed about a dozen of complex generators and applied them to test at least six refactorings in each tool. So far, we have found 28 unique, new bugs and reported them, 13 in Eclipse Bugzilla and 15 in NetBeans Issuezilla. This is ongoing work, and the numbers are increasing. We advocate the importance of automated testing—not only automated execution of manually written tests (using JUnit or XTest) but also automated generation of test inputs. We have developed several oracles that programmatically check whether a refactoring tool correctly made some program transformations (or gave warning that a specific refactoring should not apply to the given input program). We hope that this paper motivates developers of refactoring tools to incorporate such generation and oracles into their tools. While most refactoring tools are already quite reliable, we believe that the use of such generation would further increase reliability, to the benefit of all users of refactoring tools. Moreover, we argue that such generation can be useful for testing other related tools that take (Java) programs as inputs. To encourage collaboration and enable others to try out ASTGen, we have made our ASTGen code and all experimental results publicly available at the ASTGen web page, http://mir.cs. uiuc.edu/astgen 1. WHY AUTOMATED GENERATION? Testing involves several activities, including generation of test inputs (and expected outputs), execution of test inputs, and checking of obtained outputs. For a refactoring tool, each input consists of a program and a refactoring to apply, and each output is either a refactored program or a warning if the specific transformation might change the program’s semantics. ∗This paper is based on the work [1] to be presented at the 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2007) in Dubrovnik, Croatia. It is often said that manual testing is tedious and errorprone. Indeed, developers of refactoring tools automate a large portion of testing. For instance, we have counted 2,673 JUnit tests for the major refactorings in Eclipse version 3.2. JUnit automatically executes these tests and checks the obtained outputs. However, JUnit does not automatically generate test inputs, and to the best of our knowledge, Eclipse developers manually wrote their JUnit tests. Automated generation of test inputs has one significant benefit: it makes it easier to generate a large number of test inputs, which hopefully results in a more thorough testing and enables finding bugs before they are encountered in production runs. However, automated generation of test inputs, especially for refactoring tools, poses several challenges. We discuss these challenges and our solution. 1.1 Generation of Input Programs How does one automatically generate valid Java programs to give as inputs to a refactoring tool? There is no obvious answer. While simpler test inputs, say one integer or a sequence of integers, can be generated even randomly, it is unclear how one could randomly generate sequence of characters (or abstract syntax trees) that satisfy the syntactic and semantic constraints for a valid Java program. Moreover, even if one could generate programs randomly, how would one ensure that these programs have properties relevant to the refactoring under test? We have developed the ASTGen framework to generate a large number of relevant Java programs. ASTGen is not fully automatic. It requires developer to write a class (which we call generator) that can produce test inputs relevant to a specific refactoring. ASTGen provides a library for generation of simple AST nodes. This library makes it easy to build more complex combinations of AST nodes. We have written several generators that produce Java programs for testing refactoring engines. More details are in the conference paper [1]. We point out that the generators do not always produce programs that compile. (The column “CI” in Figure 1 shows how many of the generated inputs compile and are thus valid inputs for a refactoring.) 1.2 Execution of Refactorings How does one automatically run a refactoring tool on the automatically generated input programs? This is seemingly an easy task: just develop a piece of code that (efficiently) runs a specific refactoring on each of the generated programs. However, we have encountered a number of problems while developing this piece of code, in both Eclipse and Generation Oracles Bug Refactoring Generator TGI Time CI WS DNC C/I Diff Reports [min:sec] Ecl NB Ecl NB Ecl NB Rename(Method) MethodReference 9540 89:12 9540 3816 0 0 0 0 5724 0 0 Rename(Field) FieldReference 3960 28:2