Extensible code auto-fix framework based on XML query languages

A method is provided to automatically correct an original source code. An abstract syntax tree (AST) is created from the original source code where the AST includes AST nodes. AST node filter queries are evaluated on the AST to filter out AST nodes with defect patterns. Automatic fixes are applied to the filtered AST nodes to transform the AST. A modified source code is created by deserializing the transformed AST.

BACKGROUND

Coding standard violations, non-conformance to best practices, and defect patterns are abundant in existing source code. These source code issues lead to unmaintainable code and potential bugs in later stages of software life cycle when they are more expensive to fix. It is important to detect and correct these source code problems early in the development cycle when they are less expensive to fix.

DETAILED DESCRIPTION

Static analysis techniques such as peer code review and automatic static analysis are available to address coding standard violations, non-conformance to best practices in source code, and defect patterns. However, they are not effective in practice due to (a) involvement of significant human effort, (b) difficulty in validating code against a lengthy coding standard, and (c) prioritization of logic validation in favor of enforcing style and best practices. Automatic static analysis integrated with tool-assisted code review process has been proposed to automate style and design related checks. Even with such a solution, a developer still has to analyze the reported issues, find potential fixes, and re-submit the code for review. Thus what are needed are method and apparatus to pre-process the source code to detect and fix automatically a large subset of source code problems so that there are less style or design issues for a reviewer to detect and a developer to fix.

In examples of the present disclosure, a software tool is provided to automatically detect and correct source code problems. The tool may be standalone or integrated with an integrated development environment (IDE) or a code review tool. In examples of the present disclosure, the tool uses automatic static analysis tools to detect source code problems and fixes these issues with built-in automatic fixes. The tool is configurable as new static analysis tools, custom defect patterns, and new automatic fixes can be added to the tool. The automatic fixes may include automatic refactoring that improve code readability and reduce code complexity in order to improve the maintainability of code.

In examples of the present disclosure, source code problems are auto-detected and auto-corrected using abstract syntax trees (ASTs) and extensible markup language (XML) query languages. A source code may be represented as an AST, which may be modeled as a document object model (DOM) tree. AST node filter queries, such as XPath and XQuery expressions, may be evaluated on the AST DOM tree by an XML query engine to filter out AST nodes with source code problems, and automatic fixes are applied to fix the problems by transforming the AST. AST update queries, such as XQuery UPDATE expressions, may be evaluated on the AST DOM tree by the XML query engine to filter out AST nodes with source code problems and fix those problems by transforming the AST. Static analyzers may be used to identify source code problems in the source code and automatic fixes are applied to fix those problems by transforming the AST.

FIG. 1is a block diagram illustrating the architecture of a software tool100to automatically detect and correct problems in original source code102in examples of the present disclosure. Tool100may be implemented with computer readable instructions to be executed by a processor. Tool100includes modules grouped into the following stages: stage 1—problem identification; stage 2—AST generation, stage 3—document object model (DOM) tree generation and node filtering; stage 4—auto-correction; and stage 5—user interaction.

In stage 1, a problem finder module104invokes static analyzers106to analyze original source code102and identify source code problems, such as coding standard violations, best practice violations, and defect patterns. Static analyzers106may include FindBugs, Checkstyle, and PMD. A configuration module108provides the configurations for static analyzers106, such as rules or checks to run and customized errors or warning messages. As output formats of static analyzers106vary, problem finder104invokes output parsers110to convert the static analyzer outputs to a common format. Problem finder104sends the source code problems to a first auto-fix module124in stage 4.

In stage 2, an AST parser selector112selects an AST parser114from a set of AST parsers114based on the programming language of original source code102. AST parsers114may include ASTParser in Eclipse JDT core and ANTLR parsers. AST parser selector112uses the selected AST parser114to generate an AST based on original source code102, and outputs the AST to a DOM tree generator116in stage 3 and first auto-fix module124in stage 4.

In stage 3, DOM tree generator116creates a DOM tree adapter that wraps around the AST. The DOM tree adapter provides a DOM-compatible interface to the underlying AST so updates on the DOM tree adapter are translated to updates on the underlying AST. An AST node filter118uses an extensible markup language (XML) query engine120to evaluate AST node filter queries122on the DOM tree adapter to filter out AST nodes that match user defined defect patterns. An AST node filter query122may be an XPath or an XQuery expression that is to detect AST nodes having a user defined defect pattern. AST node filter118creates a query filter fix context for each of the query results. A query filter fix context is a fix context that includes information to be used by an automatic fix to correct a source code problem identified by an AST node filter query122.

In stage 4, first auto-fix module124creates a static analyzer problem fix context for each of the source code problems received from problem finder104in stage 1. Similar to a query filter fix context, a static analyzer problem fix context is a fix context that includes information to be used by an automatic fix to correct a source code problem identified by a static analyzer106.

First auto-fix module124then determines an automatic fix for each of the fix contexts, and applies the automatic fix to the AST received from the selected AST parser114. For each fix context, first auto-fix module124selects an automatic fix126from a database127based on the AST node filter query ID or the source code problem type listed in the fix contexts. A second auto-fix module128uses XML query engine120to evaluate AST update queries130on the AST DOM tree adapter by identifying and transforming user defined defect patterns in the AST DOM tree adapter received from DOM tree generator116. An AST update query128may be an XQuery UPDATE expression that can be applied without a fix context. First auto-fix module124and second auto-fix module128outputs the transformed AST to an AST deserializer132in stage 5.

In stage 5, AST deserializer132generates a modified source code134based on the transformed AST. A visual client136presents the identified problems, original source code102, and modified source code134to user. Visual client136uses a difference generator138to provide a side-by-side comparison of original source code102and modified source code134, and provide options to inspect the source code problems and the corresponding fixes, and to selectively apply a subset of fixes to original source code102. Modified source code134may be shown in redlined form using underlining and strikethroughs.

FIG. 2is a pseudo code200for implementing software tool100(FIG. 1) in examples of the present disclosure.FIG. 3is a flowchart of a method300representing pseudo code200(FIG. 2) in examples of the present disclosure. Method300may begin in block302.

In block302, problem finder104(FIG. 1) of tool100uses static analyzers106(FIG. 1) finds problems in source code102(FIG. 1). Block302corresponds to line 1 of pseudo code200. Block302may be followed by block304.

In block304, AST parser selector112(FIG. 1) of tool100selects an AST parser114(FIG. 1) based on the source code language and generates an AST. Block304corresponds to lines 3 and 4 of pseudo code200. Block304may be followed by block306.

In block306, DOM tree generator116(FIG. 1) of tool100generates a DOM tree adapter that wraps around the AST. Block306corresponds to line 6 of pseudo code200. Block306may be followed by block308.

In block308, AST node filter118(FIG. 1) of tool100evaluates AST node filter queries122(FIG. 1) on the DOM tree adapter and creates a query filter fix context for each of the query results. Block308corresponds to lines 7 to 11 of pseudo code200. Block308may be followed by block310.

In block310, first auto-fix module124(FIG. 1) of tool100creates a static analyzer problem fix context for each of the problems identified by static analyzers106in block302. Block310corresponds to lines 13 to 15 of pseudo code200. Block310may be followed by block312.

In block312, first auto-fix module124of tool100determines an automatic fix126from database127(FIG. 1) for each of the fix contexts created in blocks308and310, and applies the fix to the AST. First auto-fix module124selects the appropriate automatic fix126based on the AST node filter query ID or the source code problem type listed in each fix context. Block312corresponds to lines 16 to 18 of pseudo code200. Block312may be followed by block314.

In block314, second auto-fix module128(FIG. 1) of tool100transforms the AST by evaluating AST update queries130(FIG. 1) on the DOM tree adapter. Block314corresponds to lines 19 and 20 of pseudo code200. Block314may be followed by block316.

In block316, AST deserializer132(FIG. 1) of tool100deserializes the AST to generate modified source code134(FIG. 1). Block316corresponds to line 22 of pseudo code200. Block316may be followed by block318.

In block318, visual client136(FIG. 1) of tool100presents the identified source code problems, original source code102, and modified source code134to user. As described above, a side-by-side comparison is presented so user may inspect the source code problems and the corresponding fixes, and to selectively apply a subset of fixes to original source code102. Block318corresponds to line 23 of pseudo code200.

Referring toFIG. 1, software tool100is extendable in the following manners in examples of the present disclosure.

New static analyzers106may be added to stage 1 to detect new source code problems.

For a source code problem identified by a static analyzer106in stage 1 that does not have a built-in automatic fix, a new automatic fix126may be created and added to database127. When automatic fixes126are written in Java in some examples of the present disclosure, an interface called “IFix” is provided. The interface IFix has an apply method to accept static analyzer problem fix contexts for source code problems identified by static analyzers106. For example, a static analyzer problem fix context includes the position of a source code problem in source code102, the source code problem type identified by static analyzer106, the root of the AST, and an instance of “ASTRewrite” to edit the AST. Automatic fix126implements the interface IFix and applies a fix to the AST. First auto-fix module124selects the appropriate automatic fix126from database127based on the source code problem type listed in the static analyzer problem fix contexts.

Automatic fix126may follow a visitor design pattern to visit AST nodes of an AST node type associated with the source code problem, check if each AST node covers the source code problem, and apply changes to each AST node that covers the source code problem.FIG. 4shows a snippet of an automatic fix400for the Checkstyle's “RedundantModifier” check that follow the visitor design pattern in examples of the present disclosure. The RedundantModifier check detects redundant modifiers such as public, static, final modifiers for a variable declaration in an interface. As described later, a designer tool800(FIG. 8) may be used to identify the AST node type associated with a specific source code problem.

AST node filter queries122and their automatic fixes126may be added to detect and fix custom defect patterns. AST node filter queries122may be written as XPath or XQuery expressions.FIG. 5shows an AST node filter query500written as an XPath expression in examples of the present disclosure. AST node filter query500may be used to detect a Java coding standard violation that requires floating point constants to be written with a digit before the decimal point. AST node filter queries122like AST node filter query500may be designed using designer tool800(FIG. 8).

AST node filter118uses XML query engine120to evaluate an AST node filter query122on a DOM tree adapter. AST node filter118stores AST nodes matching AST node filter query122or AST nodes with AST node properties matching AST node filter query122in a query filter fix context. The query filter fix context may include the query string and a list of the AST nodes that match the query string.

When automatic fixes126are written in Java in some examples of the present disclosure, an interface called “IQueryFilterFix” is provided. The interface IQueryFilterFix has an apply method to accept query filter fix contexts for user defined defect patterns identified by AST node filter queries122. An automatic fix126implements the interface IQueryFilterFix and applies a fix to the AST. First auto-fix module124selects the appropriate automatic fix126from database127based on the AST node query filter ID in the query filter fix context.

FIG. 6shows an automatic fix600for AST node filter query500(FIG. 5) in examples of the present disclosure. Automatic fix600modifies the TOKEN property of the NumberLiteral AST node.

When the defect pattern is too complicated to be expressed in XPath or XQuery expressions, an AST node filter query122may be written as “/” to match the root of the DOM tree adapter and the corresponding automatic fix can traverse the entire DOM tree adapter to find the desired defect pattern and fix it.

An AST update query, which may be written as an XQuery UPDATE expression, may be added when a fix is possible by deleting an AST node or by replacing the value of an AST node property in examples of the present disclosure.FIG. 7shows an AST update query700in examples of the present disclosure. AST update query700provides a fix for Checkstyle's “ConstantName” check, which detects any constant identifier with characters other than upper-case letters, digits, and underscores.

FIG. 8shows designer tool800in examples of the present disclosure. Designer tool800provides an interface that may be used to design AST node filter queries122(FIG. 1), automatic fixes126(FIG. 1), and AST update queries130(FIG. 1). In the main window of designer tool800, an AST view pane802displays the AST of a snippet of source code102(FIG. 1) entered in a source code pane804.FIG. 9shows a sample AST900in examples of the present disclosure. The highlighted AST node902corresponds to the code fragment904“f=0.5f” in the source code. Each of the AST nodes can have zero or more attributes (names prefixed with @) and zero or more child elements (nodes with name in upper-case letters). Each of the child elements can have zero or more AST nodes as its children where the child count is in parentheses after the node name. The “*BINDING” attributes are special attributes that does not have a structural significance and is used for cross-referencing between AST nodes. For example, any “SimpleName” node corresponding to the float variable “f” (say the node representing f in the ExpressionStatement f++) will have the same NAMEBINDING value as that of its VariableDeclarationFragment (node representing its declaration). The listing of AST update query700(FIG. 7) uses this concept to find the references of the constant identifier.

Designer tool800may be used to design an automatic fix126for source code102problem identified by a static analyzer106in examples of the present disclosure. A user manually identifies a problem in source code102. The user checks the output of static analyzer106for the source code problem and notes the source code problem type. The user uses designer tool800to generate an AST of source code102and identifies an AST node to modify to fix the source code problem, such as AST node902to modify to fix the floating point without a digit before the decimal point inFIG. 9. The user writes an automatic fix126for the AST node type of the identified AST node and the source code problem type. This automatic fix126takes as input a static analyzer problem fix context.

Designer tool800may be used to design an AST node filter query122and its automatic fix126in examples of the present disclosure. A user manually identifies a problem in source code102. The user uses designer tool800to generate an AST of source code102and identifies an AST node to modify to fix the source code problem. As shown inFIG. 10in examples of the present disclosure, the user writes an AST node filter query122as an XPath expression in an area1002of XPath view pane806(FIGS. 8 and 10), runs AST node query filter122to detect a user defined defect pattern in source code102, and reviews the result in an area1004of XPath pane806. The user may modify AST node query filter122until the query results are satisfactory. The user then writes an automatic fix126for AST node filter query122for the AST node query filter ID. This automatic fix126takes as input a query filter fix contexts. User may also design AST node filter queries using XQuery.

Designer tool800may be used to design an AST update query130in examples of the present disclosure. A user manually identifies a problem in source code102. The user uses designer tool800to generate an AST of source code102and identifies an AST node to modify to fix the source code problem. As shown inFIG. 11in examples of the present disclosure, the user writes an AST update query130as an XQuery UPDATE expression in an area1102of XQuery view pane808(FIGS. 8 and 11), runs AST update query130to transform a user defined defect pattern in source code102, and reviews the result in an area1104of XQuery pane808. The user may modify AST update query until the query results are satisfactory.

FIG. 12is a flowchart of a method1200to add a new fix for a source code problem to tool100(FIG. 1) in examples of the present disclosure. The new fix may be an automatic fix126(FIG. 1), an AST node filter query122(FIG. 1) and a corresponding automatic fix126, or a new AST update query130(FIG. 1). Method1200may begin in block1202.

In block1202, a relevant AST node that includes the source code problem is determined. As described above, a user may use designer tool800(FIG. 8) to generate an AST of a source code and identifies an AST node to modify to fix the source code problem. The source code problem may be automatically detected by a static analyzer106(FIG. 1) or manually detected from source code102(FIG. 1). Block1202may be followed by block1204.

In block1204, an automatic fix is written to fix the source code problem identified by a static analyzer106. Alternatively an AST node filter query122is written to identify the source code problem and an automatic fix126is written to fix the source code problem. Alternatively an AST update query130is written to fix the source code problem. Block1204may be followed by block1206.

In block1206, the new fix is added to tool100. For example, the new fix is added to a configuration file that is read when tool100is executed to analyze source code.

From the foregoing, it will be appreciated that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. For example, automatic fixes126may be written in another programming language such as C or C++, and designer tool800may test automatic fixes126written in another languages such as Java or C++. Tool100may be used as a stand-alone tool or integrated with an IDE or a code review tool. For example, tool100may be a pre-processing module of a code review tool that automatically corrects a subset of source code problems before sending a review request to code reviewers with a list of the remaining source code problems. In another example, tool100may be an IDE plugin. Accordingly, the various embodiments disclosed herein are not intended to be limiting, with the true scope and spirit being indicated by the following claims.