Abstract:
A computer-implemented method for evaluating software code includes measuring a first coverage of a test applied to the software code and then making a modification in a first section of the software code. A second coverage of the test applied to the software code is measured after making the modification. A difference is identified between the first coverage and the second coverage in a second section of the software code, which is separate from the first section and was not modified since the first coverage was measured, and an indication of the difference is output.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates generally to computer systems and software, and specifically to detecting bugs in software code. 
       BACKGROUND OF THE INVENTION 
       [0002]    Regression testing is a technique for discovering bugs that may have been introduced into software code as an unintended result of changes made to the code. This sort of testing is typically initiated after a programmer has attempted to fix a recognized bug or otherwise changed or added code to an existing program. For this purpose, programmers often develop a suite of tests, in the form of inputs, commands and execution parameters, for example, that are applied to exercise various parts of the code after a change has been made. 
         [0003]    Software code coverage analysis is commonly used to track areas of a program that are exercised by a set of tests (and thus to find areas that are not exercised). A code coverage analyzer is a software tool that determines which lines of code were executed in a given test run. It can be used to give a quantitative measure of code coverage, as well as to assist in creating additional test cases to increase coverage and identify redundant test cases that do not increase coverage. For example, Bullseye Testing Technology (Redmond, Wash.) offers the BullseyeCoverage code coverage analyzer for C++ and C, which can be used to evaluate various coverage measures, including line coverage, statement coverage, branch coverage, condition coverage and function coverage. Rosenblum et al. describe a method for choosing an efficient regression test suite based on coverage analysis in “Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies,”  IEEE Transactions on Software Engineering  23:3 (March, 1997), pages 146-156. 
         [0004]    Another use of coverage analysis in test suite generation is described in U.S. Patent Application Publication 2005/0044533. A current software build is compared to a reference software build, which is typically a known, previous build. The comparison identifies those areas in the current software build that have changed with regard to the reference software build. The identified areas are used by a coverage analysis process to determine a focused test suite to test the modified areas of the current build. 
         [0005]    U.S. Patent Application Publication 2005/0223361 describes another method for non-redundant software testing, based on identifying changes in the execution paths of a software program. Test cases capable of traversing the changed or new execution paths are identified and executed to test the modified code. 
         [0006]    In contrast to regression testing, software change impact analysis provides a forecast of the potential effects of changes before the changes are implemented. Impact analysis, in other words, estimates what will be affected in software and related documentation if a proposed software change is made. For example, U.S. Patent Application Publications 2006/0117310 describes a method for analyzing the impact of an update to a software system. The impact analysis identifies resources that are affected by the update, such as configuration information and executable code in dynamic link libraries. 
       SUMMARY OF THE INVENTION 
       [0007]    Embodiments of the present invention provide methods, apparatus and software products for evaluating software code. A processor measures a first coverage of a test applied to the software code before a programmer makes a modification in a first section of the code, and measures a second coverage after the modification has been made. The processor identifies differences between the first coverage and the second coverage in other, unmodified sections of the software code and outputs an indication of the differences. The programmer may use this indication in debugging the code 
         [0008]    The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which: 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  is a schematic, pictorial illustration of a system for debugging software code, in accordance with an embodiment of the present invention; 
           [0010]      FIG. 2  is a block diagram showing components of a software program that are modified and tested in accordance with an embodiment of the present invention; and 
           [0011]      FIG. 3  is a flow chart that schematically illustrates a method for testing and debugging software code, in accordance with an embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
       [0012]    When programmers make modifications to an existing software program in order to fix a recognized bug or to add new functionality, these modifications often give rise to unexpected new bugs. These new bugs can be manifested not only in the section of the software code that was modified, but also in the behavior of other, unmodified sections of the code, typically due to interaction between the sections that the programmer did not take into account. Current methods for detecting these sorts of bugs, such as impact analysis and regression testing, are time-consuming and not completely effective. 
         [0013]    Embodiments of the present invention that are described hereinbelow provide an efficient tool that can be used to detect bugs arising from modifications in a program that might otherwise escape the programmer&#39;s attention. The tool is based on a novel combination of regression testing and coverage analysis, stemming from the realization that changes in test coverage may be indicative of unintended changes in the control flow of the program. The tool detects changes in coverage that have occurred in unmodified sections of the program in order to warn and otherwise assist the programmer in identifying and fixing these bugs. 
         [0014]      FIG. 1  is a schematic, pictorial illustration of a system  20  for debugging software code, in accordance with an embodiment of the present invention. System  20  comprises a code processor  22 , which is operated by a programmer to analyze and debug software code, which is typically stored in a memory  24 . The programmer interacts with processor  22  via a user interface, which typically comprises an input device  26 , such as a keyboard and/or mouse, and an output device  28 , such as a display monitor and/or printer. 
         [0015]    Processor  22  applies selected tests from a regression test suite to the software code before and after the programmer makes modifications in the code. The processor measures the coverage of the tests before and after a given modification, and generates an output that is indicative of changes that have occurred in the coverage (particularly in sections of the code that were not modified.) The output may be delivered to the programmer via output device  28  and/or recorded in memory  24 . Typically, the programmer responds to this output by reviewing the code to find the cause of the coverage change, and then debugging the code if necessary. Alternatively or additionally, processor  22  may automatically suggest or implement a code correction or may perform other analysis or invoke other debugging tools. 
         [0016]    Typically, processor  22  comprises a general-purpose computer, which is programmed in software to carry out the functions described herein. The software may be downloaded to the computer in electronic form, via a network, for example, or it may alternatively be provided on tangible media, such as optical, magnetic, or electronic memory. Processor  22  may comprise a single computer, as illustrated in  FIG. 1 , or it may comprise a group of two or more computers, with the various functions divided up among them. 
         [0017]    Reference is now made to  FIGS. 2 and 3 , which schematically illustrate a method for testing and debugging software code in a software program  30 , in accordance with an embodiment of the present invention.  FIG. 2  is a block diagram that schematically shows code sections  32 ,  34 ,  36 ,  38  of program  30  and tests  46 ,  48 , . . . , in a regression test suite  44 , which are applied in order to test the program  30 . Sections 32, 34, 36, 38 are identifiably-separate components of program  30 . For instance, each section may comprise a different file or class or a particular group of files or classes. 
         [0018]      FIG. 3  is a flow chart that shows steps in the method of testing and debugging program  30  following a modification in the program, as described hereinbelow. Although this method is illustrated, for the sake of clarity, with reference to the particular program structure shown in  FIG. 2 , the principles of this method may be applied,  mutatis mutandis , to substantially any sort of program organization for which a regression test suite can be defined, as will be apparent to those skilled in the art. For instance, the coverage analysis that is performed as part of this method may relate to functional coverage, i.e., sections  32 ,  34 ,  36 ,  38  may be functional blocks of the program code, rather than structural elements such as files or classes. Functional coverage techniques that may be used in this context are described, for example, by Piziali, in  Functional Verification Coverage and Analysis  (Kluwer Academic Publishers, Boston, 2004). 
         [0019]    Test suite  44  is defined and developed for regression testing of program  30 , at a test definition step  50 . Methods for designing test suites are known in the art and are beyond the scope of the present invention, for which any suitable method of test design may be used. Tests  46 ,  48 , . . . , in suite  44  are generally chosen and designed so that they collectively give good coverage to the various sections of program  30 , using techniques described above in the Background of the Invention, for example. Typically, a certain set of the tests in test suite  44  is defined for regression testing of each section of program  30 , wherein the tests defined for each section give good coverage of that section, but not necessarily of other sections of the program. 
         [0020]    Processor  22  may keep a record of which tests cover each particular code section or group of sections, so that the appropriate regression tests may be run after any given code modification. Thus, in the example shown in  FIG. 2 , the processor has determined by suitable coverage analysis that tests  46 , labeled T 1  and T 2 , cover sections  32  and  34 , whereas test  48 , labeled T 3 , covers other sections of the program. These coverage measurements may be made, for instance, using the BullseyeCoverage tool mentioned in the Background of the Invention or other coverage analyzers that are known in the art. Some compiler packages (such as the Intel® Fortran compiler) offer test prioritization capability, which can be used to select and prioritize the tests that are most relevant to the subset of the software code that has been changed. 
         [0021]    Continuing with this example, it is assumed that a programmer plans to make a modification in a code block  40  in section  32 , which is a file named NNN.C. Before making the modification, processor  22  makes a baseline measurement of coverage of the associated tests  46  (T 1  and T 2 ), at a first coverage measurement step  52 . The baseline measurement may have been made previously and stored in memory  24 , for instance, in which case it is not necessary to repeat the measurement before making the code modification. Alternatively, this measurement may be invoked, either by the programmer or automatically by processor  22 , before the actual modification is made or before the modified code is compiled. After the baseline measurement has been stored, the software modification in block  40  is carried out, at a code modification step  54 . The code is then compiled and prepared for testing. 
         [0022]    Processor  22  runs tests  46  on program  30 , and checks the post-modification coverage of the tests, at a regression testing step  56 . In view of the modification to the code in block  40 , some changes in coverage may be expected in section  32 , and possibly in related sections  34 , as well. For the most part, however, coverage changes are not expected in sections  36  and  38 , which do not relate directly to the functions exercised by tests  46 . 
         [0023]    Processor  22  detects changes in coverage of unmodified sections of the code, at a change detection step  58 . If no such changes are detected, the processor exits from the method of  FIG. 3  and informs the programmer that there have been no significant coverage changes. In the example shown in  FIG. 2 , however, the processor determines that the coverage by T 1  of a line  42  of code in file XYZ.C has changed. The change may be detected, for instance, because line  42  previously was not covered by T 1  and has now been covered or, alternatively, because line  42  was covered previously but is no longer covered. Alternatively or additionally, coverage changes of this sort may be detected and analyzed with granularity either coarser or finer than a single line of code. 
         [0024]    As noted earlier, the unexpected change in the coverage of line  42  in block  36  may be indicative of a bug that the modification in block  40  has introduced into program  30 , due to an unintended effect of the modification. Processor  22  therefore outputs to the programmer an indication of the coverage change via output device  28 , at a reporting step  60 . The report may simply comprise a listing of any and all changes that have occurred between the pre- and post-modification coverage measurements and the test or tests in which the changes occurred. (If two or more tests caused the same change, the programmer or processor  22  may be able to infer the cause of the change on the basis of the common features of these tests.) 
         [0025]    Alternatively, processor  22  may filter the changes at step  60  before reporting them to the programmer. For example, the programmer or the processor may indicate groupings of code sections (such as sections  32  and  34 ), so that the processor reports only the coverage changes that occur outside the group to which the modified section belongs. As another example, the programmer may indicate to the processor that coverage changes in certain code sections, such as “bookkeeping” blocks in one of sections  38 , are not of interest. Alternatively or additionally, upon reviewing the list of coverage changes, the programmer may manually mark (using a mouse on the display monitor, for instance) the changes that are not of interest. Based on the programmer&#39;s selections, the processor may then derive rules to be applied in filtering out certain coverage changes that may occur in subsequent iterations through step  60 . For example, if the programmer consistently marks off coverage change indications with respect to a particular file, routine or program variable, the processor may determine that it should not warn the user of future coverage changes relating to that file, routine or variable. 
         [0026]    The programmer, assisted by processor  22 , checks the coverage change indications that were generated at step  60  in order to determine whether they are indicative of bugs that have been created in program  30 , at a bug evaluation step  62 . If the programmer and/or processor determines at step  62  that no new bug has been created, the processor exits from the method of  FIG. 3 . 
         [0027]    Otherwise, if a bug has been found to exist, the programmer debugs the code, using the tools, hints and warnings provided by the processor, at a debugging step  64 . For example, processor  22  may automatically insert a breakpoint in the vicinity of line  42  and open a debugging window at the point in test  46  at which the unexpected coverage change occurs, thus enabling the programmer to step through program  30  in the vicinity of this line. After debugging, the processor may repeat the regression testing and coverage checking, beginning from step  56 , as described above. 
         [0028]    It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.